Forecasting Using First-Order Difference of Time Series and Bagging of Competitive Associative Nets

Transcription

1 Forecasng Usng Frs-Order Dfference of Tme Seres and Baggng of Compeve Assocave Nes Shuch Kurog, Ryohe Koyama, Shnya Tanaka, and Toshhsa Sanuk Absrac Ths arcle descrbes our mehod used for he 2007 Forecasng Compeon for Neural Neworks and Compuaonal Inellgence. We have employed he frs-order dfference of me seres for dealng wh he seasonaly of he monhly daa. Snce he dfferencng removes he rend of me seres, we have developed a mehod o esmae he rend. Moreover, we have used he baggng of compeve assocave ne called CAN2 as a learnng predcor, where he CAN2 s for learnng an effcen pecewse lnear approxmaon of a nonlnear funcon, and he baggng for reducng he varance of he predcon. I. INTRODUCTION Ths arcle descrbes he mehod whch we have used for he 2007 Forecasng Compeon for Neural Neworks and Compuaonal Inellgence. A he compeon, he compeors should forecas 11 or 111 me seres as accuraely as possble by means of usng her mehods. One of he dffcules of hs problem s ha he 11 and 111 me seres are monhly daa whch nvolve varous properes of me seres, such ha some are saonary bu some have posve rend, no all bu some nvolve seasonaly wh he perod 12, some seem o have oulers, and so on. Thus, we would lke o employ or develop a general mehod whch can overcome he above dffcules. One of he echnques whch we have employed s o use he frs-order dfference of me seres for dealng wh he seasonaly. Here, snce he dfferencng removes he rend of he me seres, we have also developed a mehod o esmae he rend of me seres. On he oher hand, o cope wh he compeon problem, our frs decson was o use our compeve assocave ne called CAN2. Here, he CAN2 has been nroduced for ulzng compeve and assocave schemes [1], [2] and learnng an effcen pecewse lnear approxmaon of a nonlnear funcon. Ths approach has been shown effecve n several areas such as funcon approxmaon, conrol, ranfall esmaon and me seres predcon [3]- [8]. Here, noe ha he dfferences of he CAN2 o oher smlar mehods are as follows. The mehod of local lnear models [9] uses lnear models obaned from K-neares neghbors of npu vecors whle he CAN2 ulzes lnear models assocave memores opmzed by he learnng mehod nvolvng compeve and assocave schemes. The CAN2 may be vewed as a mxure-of-expers model ha ulzes lnear models as expers and compeve scheme as gang. Alhough he MARS mulvarae adapve regresson Shuch Kurog, RyoheKoyama, Shnya Tanaka, and Toshhsa Sanuk are wh he Deparmen of Conrol Engneerng, Kyushu Insue of Technology, Tobaa, Kakyushu , Japan phone: ; fax: ; emal: kuro@cnl.kyuech.ac.jp. splnes model [10] as a mxure-of-expers model execues connuous pecewse lnear approxmaon, he CAN2 execues dsconnuous one nendng for opmzng each lnear model n he correspondng Vorono regon. Furher, we employ he baggng scheme [11] for reducng he varance of he predcon by he CAN2. In he followng secons, we descrbe our mehod n deal, some expermenal resuls, and hen he concluson. II. METHOD FOR THE FORECASTING COMPETITION Here, we show our mehod employed for he forecasng compeon afer formalzng he compeon problem. A. The Problem o be Solved A he compeon, a se of real valued monhly me seres y R s provded, where T gven I[s gven, gven ] denoes a pon n me or a monh, and ndcaes he ndex of a me seres n he ndex se I A I[1, 111] of he daase A or he complee daase or I B I[101, 111] of he daase B or he reduced daase. Here, I[s, ] {s, s + 1, s + 2,, } ndcaes a se of negers from s o. For each me seres y, a compeor should predc forecas y for he fxed me horzon of h = 18 monhs, or for T forecas I[ gven +1, gven +h]. The forecass ŷ for I[ + 1, + h] are evaluaed based on he SMAPE Symmerc Mean Absolue Percen Error gven by SMAP E = 1 h +h = +1 B. Forecasng Usng Orgnal Tme Seres ŷ y ŷ + y /2 Here, we show he forecasng usng he orgnal me seres on whch he forecasng usng he frs-order dfference shown below s based. Frsly, we suppose ha he me seres y sasfes y = f x + ɛ, 2 where ɛ represens nose, and f x s a nonlnear funcon of a vecor x x 1, x 2,, x k T. Here, he jh elemen x j of x s a daa pon of a gven me seres wh a delay, y l τ = y lj τ j, whose ndex l j and delay τ j are seleced from all l and τ

2 so ha y lj τ j has he jh larges correlaon: R l, τ = y y y l τ y l τ y y 2, 2 y l τ y l τ where y and y l τ ndcae he mean of y and y l τ, respecvely. Here, he range of for he mean s se so ha boh y and y l τ can have provded values, and we se R l, τ = 0 f he number of y and y l τ for he mean s lle han h = 18. In order o forecas unknown values of he me seres, we use he baggng of he CAN2 see below for deals as a predcor. Furher, for esmang he performance of he predcor and une he parameer values of he predcor, we run valdaon ess whch use y T ran = I[s gven, ] for ranng he predcor, and y T pred = I[ + 1, + h] for valdang he predcon, where s a pon n me whch sasfes gven h for valdaon. Of course, we can make he fnal forecas by usng = gven. The predcor, a frs, learns y T ran o make he mul-sep predcon gven by 3 ŷ = f x, θ, 4 where θ represens he parameer values of he predcor, and he elemen of he vecor x x 1, x 2,, x k T s gven by { τ ylj j for τ j T ran x j ŷ lj τ j l j, for τ j T pred l j. where he elemens x j are deermned successvely for = + 1, + 2,. Thus, he learnng, forecasng, and valdaon can be done by he above procedure. C. Forecasng Usng Frs-Order Dfference The forecasng usng he frs-order dfference s descrbed as follows see Secon II-E.1 for dealed reasons of hs mehod. Frs, we make he frs-order dfference y = y y 1, and use a predcor o learn y for } = I[s gven + 1,, ], and ge he mul- pred for T = T ran \{s gven sep predcon y = f d x, θ d 5 I[ + 1, + h], where he elemens of he vecor x x 1, x 2,, x k d T are generaed by means of replacng x j and y lj τ j n Eq.5 by x j and y lj τ j, respecvely. The dmenson k d s of he vecor x, and f d x, θ d s a nonlnear funcon of x acheved by he predcor wh he parameer values represened by θ d. Then, he predcon of y for T pred s consruced by ŷ d y + j= +1 y j. 6 Here, y j s he predcon y j obaned by he predcor ha has learned y for I[s gven + 1, ], and we somemes neglec for smplcy. As shown n Secon II-E.1, he rend of he predcon ŷ d s no so relable, so we frs employ an averagng mehod. Namely, we use ŷ da 1 a 1 a l=0 j= +1 ŷ d l, 7 where a 1 s he number of averagng. Furher, we employ a mehod o modfy he rend of he predcon as follows: frs, we make a mh order polynomal approxmaon ỹ m = m j=0 a j j of y for T gven and a ral value of y T = y + h for m = 0, 1, 2,, where represens he mh order polynomal rend over T gven { +h}. Nex, we make a frs-order approxmaon = b 0 +b 1 of ỹ m for I[ h+1, ], where represens he frs-order shor-erm rend of y. We can selec he bes y T o mnmze he loss funcon Lỹ m,1, y, h + 1, by means of a lne search of y T for each m = 0, 1, 2,. Furher, we can deermne m from a vew of ỹ m and y see he expermenal resul shown below. Wh he seleced y T = yt and m = m, we oban he frs-order shor-erm rend ỹ m,1 = b 0 + b 1 for he forecas perod T forecas. On he oher hand, from he predcon ŷ d, we can oban he frs-order rend approxmaon as ŷ d,1 = c 0 +c 1. Then, we can modfy he rend of he predcon ŷ d as ỹ m ỹ m,1 ỹ m,1 ŷ da,m = ŷ da + b 0 c 0 + b 1 c 1. 8 D. CAN2 and he Baggng 1 Assumpons on he gven daase: Le D n {x, y I n } be a gven ranng daase, where I n {1, 2,, n} denoes he ndex se of he daase, and x x 1, x 2,, x k T and y denoe an npu vecor and he arge scalar value, respecvely. Noe ha x and y, respecvely, correspond o x and y, or x and y, nroduced n he prevous secon. Here, we suppose he relaonshp gven by y r + ɛ = fx + ɛ, 9 where r fx s a nonlnear funcon of x, and ɛ represens zero-mean nose wh he varance σ 2. 2 CAN2: A CAN2 has N uns see Fg. 1. The jh un has a wegh vecor w j w j1,, w jk T R k 1 and an assocave marx or a row vecor M j M j0, M j1,, M jk R 1 k+1 for j I N {1, 2,, N}. The CAN2 approxmaes he above funcon fx by ŷ fx ỹ c M c x, 10 where x 1, x T T R k+1 1 denoes he exended npu vecor o he CAN2, and ỹ c = M c x s he oupu value of he ch un of he CAN2. The ndex c ndcaes

3 Inpu x 0 =1 x 1. Compeve Cells Assocave Cells x k w 11 Fg. 1. w 1k 1s un M 10 M 11 M 1k Oupu y~ 1 ~y c w N1 w Nk Nh un M N0 M N1 M Nk Schemac dagram of he CAN2 he un who has he wegh vecor w c closes o he npu vecor x, or c argmn x w j. 11 j I N The above funcon approxmaon parons he npu space V = R k no he Vorono or Drchle regons V j {x j = argmn x w }, 12 I N for j I N, and performs pecewse lnear approxmaon of he funcon fx. Noe ha we have developed an effcen bach learnng mehod see [6] for deals, and we use n he presen compeon. The mehod consss of eraons of 1 compeve learnng based on a graden mehod, 2 assocave learnng employng recursve leas squares, and 3 renalzaon of uns based on an asympoc opmaly creron see [4] for overcomng local mnma problems of he graden mehod. We have used he same parameer values as for he funcon approxmaon problems shown n [6], excep he number of uns nvolved n he CAN2 whch s uned so ha he predcon acheves smaller SMAPE for he valdaon perods see Secon II-B and Secon III. 3 Baggng: Le Dj αn be he jh boosrap sample se mulse, or bag nvolvng αn elemens, where he elemens n Dj αn are resampled randomly wh replacemen from he gven ranng daase D n, and α > 0. Here, we would lke o menon ha an elemen n D n s no n Dj αn wh he probably 1 1/n αn whch approxmaely s exp α when n s large. Thus, he number of ndvdual or dfferen elemens n Dj αn approxmaely s n eff α n1 exp α. For example, n eff n whch s used n he convenonal baggng mehods [11], [12], and n eff n, whch we have employed n he presen mehod because of s emprcal good performance n several predcon problems see e.g. [8]. y~ N The baggng boosrap aggregaon for esmang he arge value r = fx s done by he mean gven by ŷ b 1 ŷ j b, 13 j I b where ŷ j ŷ j x denoes he predcon by he jh predcor CAN2 whch has learned D αn j. E. Analyss of he Mehod Here, we show some analyss of he presen mehod for examnng how he mehod works. 1 Forecasng usng he frs-order dfference: An advanage of usng he frs-order dfference s supposed o be based on ha he range of he npu vecor x s smaller han ha of he orgnal npu vecor x. Thus, even f here are few ranng daa smlar o he daa o be predced n he orgnal npu space, much more ranng daa are avalable va he frs-order dfference. For example, suppose a me seres y = y 1 +y 2 conssng of y 1 = y and y 2 = y 2 1 y wh y 1 1 = y 2 1 = y 2 2 = 1. Then, y 1 = = 1, 2, s an ncreasng seres, and y 2 = y = 1, 1, 0, 1, 1, 0, for = 1, 2, 3, 4, 5, 6, s a perodc seres wh he perod 6. Thus, y s an ncreasng seres wh flucuaon. Then, he funcon y = f x of he embeddng vecor x = y 1, y 2,, y k T s hard o be learned because x s dfferen for every = 1, 2,, where he dmenson k s assumed o be suffcenly large for f x o be a funcon e.g. k 6. However, y = 1 y 2 2 s a perodc seres wh he perod 6, and hen he funcon y = f d x of x = y 1, y 2,, y k d s easy o be learned because here are only 6 dfferen paerns of npu-oupu pars x, y o be learned and hey appear many mes n a ranng daase wh a suffcenly large number of members, where k d 2 s necessary for f d x o be a funcon. A dsadvanage of usng he frs-order dfference s ha he rend of he predcon s unsable and unrelable, whch we can see from he followng example: suppose ha a me seres y = y 1 + a + ɛ for = 1, 2, s gven, where y 0 = 0, a 0 and ɛ represens a nose. Then, he me seres whou nose s represened by r = a, and he frs-order dfference whou nose s wren by y = 1 ba + b y 1 for a ceran b 0 b 1. If a predcor learns o predc he deal frs-order dfference whou nose y = 1 ba+b y 1, he predcon of y for = + 1, + 2, s gven by y = a + ɛ ɛ 1b. Thus, he reconsruced predcon s gven by ŷ d = a + ɛ + ɛ ɛ 1 j=1 bj. So, he absolue value of he predcon error, ŷd r = ɛ + ɛ ɛ 1 j=1 bj ncreases wh he ncrease of me for ɛ ɛ 1 and b 0, and he sgn of he error depends on he nose ɛ and ɛ 1. Thus, n order o overcome hs nsably of he rend, we have developed he mehod descrbed n Secon II-C.

4 2 Coeffcens of he baggng: Insead of he baggng predcon gven by Eq.13, a more general aggregaon of predcons s gven by ỹ b j I b b j ŷ j, 14 where we suppose ha b j 0 and j I b b j = 1. Snce he predcon ŷ j nvolves he varaon caused by he nose ɛ n he ranng daa and he varaon of he boosrap resamplng daase Dj αn for j I b, he mean µ, he varaon δ j and he varance ρ 2 of he predcon ŷj = µ +δ j are gven by µ E ɛ,dj αn ŷ j, δ j ŷ j µ, ρ 2 E ɛ,dj αn δ j 2, where E ɛ,dj αn s he mean wh respec o he varaon of ɛ and he varaon of Dj αn y = r + ɛ, he varaon δ j. Snce he predcor learns of he predcon s supposed o have a posve correlaon wh ɛ, or δ j ɛ > E ɛ,d αn j Then, he expecaon of he squared predcon error ẽ b 2 = ỹ b y 2 s gven by E ɛ,dj αn ẽ b 2 = [ b 2 l µ r 2 + E ɛ,dj αn δ j ɛ 2], l I b 1 [ µ r 2 + E ɛ,dj b αn δ j ɛ 2], 16 where he nequaly s derved by he arhmec-geomerc mean nequaly, and he equaly holds when b j s consan. Thus, he mean of he square error of he aggregaon akes he mnmum when b j = 1/b j I b. Therefore, he baggng predcon gven by Eq.13 s supposed o be he mos effecve predcons among he aggregaon gven by Eq Bas and varance decomposon of predcon error: The generalzaon error or he predcon error for he populaon daa s gven by L gen I pop e b I pop ndcaes he ndex se of he populaon, e b ndcaes he predcon error. Le us suppose ŷ b where µ b E ɛ,d αn j ŷb 2, where = ŷ b y = µ b +δ b, = µ s he predcon mean, δ b = 1/b j I b δj he varaon from he mean. Then, he expecaon of L gen s gven by E ɛ,dj αn L gen = β b 2 + ρ2 b + σ2, 17 I pop ŷb E ɛ,dj αn where β b y = µ b r denoes he bas erm, and ρ 2 /b represens he varance erm. Thus, he varance erm of he baggng, ρ 2 /b, can be reduced by he ncrease of b. Here, we would lke o noe ha n order o reduce he bas erm β b 2, we have developed a bagboosng mehod [13] for he CAN2, bu we had few mprovemens. We hnk s because he amoun of he bas s no so large. Snce we had suffered from a huge compuaonal cos wh lle mprovemen of he performance, we abandoned he use of he bagboosng mehod. In addon, he above analyss s o see ha he varance of he predcon s reduced by he baggng. Here, he reducon of he varance s mporan for selecng opmal parameer values, such as he npu elemens x j = y lj τ j see Secon II-B, and he number N of he uns n a CAN2 see Secon II-D.2. Namely, we wll selec such parameer values so ha hey can acheve smalles SMAPE for a couple of valdaon perods, and he SMAPE chages largely from perod o perod f he varance s bg. III. EXPERIMENTAL RESULTS Here, we show some expermenal resuls for he reduced daase conssng of 11 me seres. We have run a valdaon es for y 101 or he 101h me seres. The predcons usng he frs-order dfference for a valdaon perod T pred 101 = I[157, 174] are shown n Fg. 2, where he nal me = 1 represens January 1978, whch s he earles me of all provded me seres. From a, we can see ha boh he predcons usng he frs-order dfference, y and y , predc he frs-order dfference y 101 of he gven me seres y 101 very well. However, from b, we can see ha he drecly reconsruced me seres ŷ d and ŷ d are no so good. Namely, we can see ha he error of hem s bgger han ha of y and y However, from Fg. 2c, we can see ha he modfed predcons ŷ d2,m 101 for m = 0, 1, 2 seem o have acheved a good performance. Here, as shown n Fg. 2d, e and f, we have used he mh order polynomal approxmaon ỹ m 101 of he orgnal daa and a expeced fnal value y T = y whch s uned by a lne search o be y T = yt ; see Secon II-C for obanng ŷd2,m 101. As shown a he commen n Fg. 2c, he SMAPE as a performance ndex of ŷ d2,m 101 s smalles for m = 1 n hs predcon perod I[157, 174]. However, we can see ha every approxmaon seems reasonable from Fg. 2d, e and f. Therefore, as one of he soluons, we decde o use m = 0 whch has he medum rend. Here, we would lke o noe ha he uned fnal value yt = y deermne he rend of he forecasng perod I[175, 192] for he submsson. Acually yt = 5116, 5192, 4996 for m = 0, 1, 2 n Fg. 2d, e, f, respecvely, and yt for m = 0 akes he medum value. Furher, he uned fnal value yt for m = 3, 4, 5 are 4801, 5349, 6402, respecvely, whch also show ha m = 0 s reasonable. However, a larger m bgger han 2 ofen provded unreasonable rend. Thus, we usually compare he predcons only for m = 0, 1, 2. The predcons for he forecasng perod T forecas 101 = I[175, 192] for he submsson are shown n Fg. 3. Here, he arge values y 101 are unknown for T forecas 101. Alhough boh ŷ101 d 174 and ŷ101 d 173 ncrease o above a = 192, we have examned ha ŷ101 d 172 decreases o below as he ncrease of me. Thus, we hnk ha he modfed predcons ŷ d2,m 101 for m = 0, 1, 2 are reasonable. We can also confrm ha ŷ d2,m 101 for m = 0 acheves he medum rend. Furher, he dfference of ŷ d2,m 101 for m = 0, 1, 2 s no so large, so he selecon

5 y y y 101 y y d2,2 101 y d2,0 101 y 101 a y d2,1 101 y y d y 101 y d y b y 101 ~ y 0,1 101 y101 d2,0 ~ y c d m = 0 y ~ 1, y d2,1 ~ y y 101 y 101 ~ y 2,1 101 y101 d2,2 ~101 y 2 e m = 1 f m = 2 Fg. 2. Example of predcons usng he frs-order dfference for a valdaon perod. a Predcons y c and y c of he frs-order dfference y 101. b Reconsruced predcons ŷ101 d 156 and ŷ101 d 155. c The modfed predcons ŷ d2,m 101 for m = 0, 1, 2, where he SMAPE s 0.463, and 0.493, respecvely. d,e and f show he mh order polynomal approxmaon ỹ m 101 of he rend of y 101, he frs order shor-erm rend ỹ m,1 101, and he modfed predcons ŷ d2,m 101 for m = 0, 1, 2. of m may no be so mporan for hs me seres, whle seems mporan for oher some me seres. IV. CONCLUSION We have descrbed he mehod whch we have used for he me seres forecasng compeon. The mehod uses he frs-order dfference for dealng wh he seasonaly of he me seres. We have developed a mehod o esmae he rend of me seres snce he dfferencng neglecs he rend of he me seres. We have shown he CAN2 for learnng effcen pecewse lnear approxmaon of nonlnear funcon, and he baggng of he CAN2 for reducng he varance of he predcon by he sngle CAN2. ACKNOWLEDGMENT We would lke o noe ha our works on he CAN2 are parally suppored by he Gran-n-Ad for Scenfc Research B of he Japanese Mnsry of Educaon, Scence, Spors and Culure. REFERENCES [1] T. Kohonen, Assocave Memory, Sprnger Verlag, [2] D. E. Rumelhar and D. Zpser, A feaure dscovery by compeve learnng, ed. D. E. Rumelhar, J. L. McClelland and he PDP Research Group: Parallel Dsrbued Processng, The MIT Press, Cambrdge, vol. 1, pp , [3] S. Kurog, M. Tou and S. Terada, Ranfall esmaon usng compeve assocave ne, Proc. of 2001 IEICE General Conference n Japanese, vol. SD-1, pp , 2001.

6 y y y y a y d2,1 101 y d2,0 101 y d2, Fg y d y y d b y 101 ~ y y101 d2,0 ~101 y 0,1 c d m = 0 ~ y y101 d2,1 y 101 y ~101 1,1 ~ y y101 d2,2 ~ y 101 y 2,1 101 e m = 1 f m = 2 Example of predcons usng he frs-order dfference for he forecasng perod T forecas 101 = I[175, 192]. [4] S. Kurog, Asympoc opmaly of compeve assocave nes for her learnng n funcon approxmaon, Proc. of ICONIP2002, vol. 1, pp , [5] S.Kurog, Asympoc opmaly of compeve assocave nes and s applcaon o ncremenal learnng of nonlnear funcons, IEICE D-II n Japanese, vol. J86-D-II-2, pp , [6] S. Kurog, T. Ueno and M. Sawa, A bach learnng mehod for compeve assocave ne and s applcaon o funcon approxmaon, Proc. of SCI2004, vol. V, pp , [7] S. Kurog, T. Ueno and M. Sawa, Bach learnng compeve assocave ne and s applcaon o me seres predcon, Proc. of IJCNN 2004, CD-ROM, July [8] S. Kurog, S. Tanaka and R. Koyama, Combnng he predcons of a me-seres and he frs-order dfference usng baggng of compeve assocave nes, Proc. of ESTSP 2007, pp , 2007 [9] J.D.Farmer and J.J.Sdorowch, Predcng chaoc me seres, Phys. Rev.Le., vol. 59, pp , [10] J.H.Fredman, Mulvarae adapve regresson splnes, Ann Sa, vol. 19, pp. 1 50, [11] L. Breman, Baggng predcors, Machne Learnng, vol. 24,pp , [12] J.G. Carney and Padrág Cunnngham, The NeuralBAG algorhm: Opmzng generalzaon performance n bagged neural neworks, Proceedngs of ESANN 1999, pp , [13] M.Delng, Bagboosng for umor classfcaon wh gene expresson daa, Bonformacs, vol. 2018,pp , 2004.