Neural Networks with Wavelet Based Denoising Layers for Time Series Prediction

Transcription

1 Neural Netwrks with Wavelet Based Denising Layers fr Time Series Predictin UROS LOTRIC 1 AND ANDREJ DOBNIKAR University f Lublana, Faculty f Cmputer and Infrmatin Science, Slvenia, {urs.ltric, andre.dbnikar}@fri.uni-l.si Abstract T avid preprcessing f nisy data, tw special denising layers based n wavelet multireslutin analysis are integrated int the layered neural netwrks. A gradient based learning algrithm is develped which uses the same cst functin fr setting bth the neural netwrk weights and the free parameters f denising layers. The prpsed layers, integrated int feedfrward and recurrent neural netwrks, are validated n time series predictin prblems: the Feigenbaum sequence, the rubber hardness time series and the yearly average sunspt number. It is shwn that the intrduced denising layers imprve the predictin accuracy in bth cases. Keywrds: feedfrward and recurrent neural netwrks, wavelet multireslutin analysis, denising, gradient based threshld adaptatin, time series predictin 1 Crrespnding authr: Urs Ltric, University f Lublana, Faculty f Cmputer and Infrmatin Science, Trzaska 25, 1000 Lublana, Slvenia, urs.ltric@fri.uni-l.si, phne: , fax:

2 1. Intrductin Mdels f dynamical systems in which the structure and parameters are extracted directly frm real data cannt avid the prblem f nise. Therefre, nise reductin has becme an imprtant issue in mdeling with neural netwrks [1]. Usually, nise reductin is treated as a separate prblem and preprcessing methds such as filtering [1, 2] based n statistical criteria are used. Hwever, attempts have been made t achieve nise reductin by integrating varius perfrmance measures int the cst functin withut influencing the input data [3, 4]. Anther apprach prpses t include methds fr input data reductin in the neural netwrk learning prcess [5]. In this paper tw special denising layers that enable neural netwrks t deal with nise withut separate data preprcessing are intrduced. These layers apply nvel filtering methds riginating in the wavelet multireslutin analysis [6]. Such methds have already been reprted in neural netwrk predictin prblems [7], hwever, nly as a preprcessing technique and nt integrated int the mdel itself. At the same time, mdels which integrate wavelets features in neural netwrks can be fund in literature [8, 9]. The design f the prpsed denising layers enables fr their integratin int the marity f layered neural netwrk. By this integratin, the same cst functin is used as the mdel perfrmance criterin and as the nise level estimatin criterin. The parameters f denising layers are set in the prcess f learning and any gradient based learning algrithm can be applied. In cntrast t methds using cst functins with additinal terms [3, 4] the prpsed denising layers can be added t any neural netwrk withut rewriting its learning prcess equatins. 2. Wavelet Based Denising In wavelet multireslutin analysis a data sample is treated n different scales. It is decmpsed int cefficients f apprximatin n a carser scale and cefficients f remaining details n finer scales [10]. One-dimensinal wavelet multireslutin analysis is based n the scaling functin φ () t and the 2

3 crrespnding mther wavelet ψ () t fulfilling certain technical cnditins. By dilatin and translatin f bth functins n abscissa the basis functins φk t k / 2, = 2 φ(2 ) and ψ k t k / 2, = 2 ψ (2 ), k, Z, are derived. An arbitrary cntinuus data sample can be written as a superpsitin f these basis functins where the sum with cefficients a J, k p() t = a φ () t + d ψ () t, Jk, Jk, k, k, k Z J k Z data sample n the scale J and the sum with cefficients the details f the data sample n scale., k Z, represents the apprximatin f the d k,, k Z, represents (1) In further discussin, nly discrete data samples p = ( p1, p 2, p, ) T N are cnsidered. The discrete wavelet transfrm and its inverse are linear peratins, and as such, they can be written in matrix frm as respectively, where the C -dimensinal vectr 11, 1, C1 J, 1 J, CJ J, 1 J, CJ D R c= W p and p= W c, (2) T c = ( d,, d,, d,, d, a,, a ) represents the discrete wavelet transfrm f the data sample. The decmpsitin matrix recnstructin matrix cnvlutin schemes [11]. D W and the R W can be cnstructed by applying the pyramidal As the wavelet cefficients f details having small abslute values are likely t cntribute t nise, Dnh and Jhnstne [6] prpsed denising by threshlding. Denising is a three-step prcess, cmpsed f discrete wavelet transfrmatin f a data sample t the space f wavelet cefficients, scale dependent threshlding f the wavelet cefficients f details, and cnstructin f a denised data sample frm threshlded wavelet cefficients in terms f inverse discrete wavelet transfrm. Fllwing the wrk f Dnh and Jhnstne [6], the generalized sft threshlding functin 3

4 1 2 2 Td ( k τ ) d k ( d k τ ) s ( d k τ ) s,, =, +, +, + + (3) 2 was intrduced [12] with parameter s 0 determining the level f smthness (Figure 1). Figure 1 The functin remves the wavelet cefficients f details d k, smaller than threshld τ and reduces the abslute values f larger wavelet cefficients f details. With the threshld τ any level f denising can be chsen, frm n denising by setting τ = 0 up t cmplete remval f infrmatin by setting = d, with d, max being the value f the abslutely largest wavelet τ, max cefficient f details n the -th scale [12]. 3. Denising Layers In neural netwrks terminlgy the wavelet denising prcess can be described by tw layers, the threshlding layer and the recnstructin layer, presented in Figure 2. Figure 2 The threshlding layer has C cmputatinal units r threshlding neurns, split int J + 1 grups, differing in threshlds τ. The number f scales J equals t the largest integer smaller than r equal t lg2 N. While the threshlds f the first J grups, prcessing the wavelet cefficients f details, are allwed t change, the threshld f the last grup, representing the apprximatin n scale J, is fixed t zer. The threshlding neurn is essentially the standard neurn [13] with nnlinear sigmid activatin functin replaced by generalized sft threshlding functin. When the input data sample p ( q) at sme discrete time q 4

5 is presented t the denising layers, the utput f the k -th threshlding neurn belnging t the grup is given by equatins N D k k k Wk l l l= 1 c% ( q) = T( c ( q), τ ), c ( q) =, p ( q), (4) where the weighted sum ck ( q ) represents ne f the wavelet cefficients ( a k, r d,,, k Z ) and the threshlding neurn utput c% k( q) represents the k matching threshlded wavelet cefficient. Threshlded wavelet cefficients are further passed t the recnstructin layer having N neurns with linear activatin functins, which generate the denised sample p% ( q) with elements C R p% k( q) = Wkl, c% l( q), k = 1,, N. (5) l= Gradient Based Learning Algrithm fr Threshlds The gal f learning is t adapt the mdel s free parameters in rder t minimize a cst functin. When gradient based algrithms are used, the gradients f the cst functin r its additive parts with respect t the mdel s free parameters need t be expressed analytically [14]. Oppsed t standard neurns with adaptable weights, the weights f threshlding neurns are fixed as given by the decmpsitin matrix D W. Similarly, the weights f linear neurns in recnstructin layer are given by the recnstructin matrix R W. Hence, the nly adaptable parameters f the denising layers are the threshlds, determining the shape f threshlding neurns activatin functins. Althugh the threshlding functin with parameter s = 0 exhibits nice statistical prperties [6], it can lead t a situatin where gradient based adaptatin f threshlds is unintentinally stpped [12]. Therefre, the parameter s = 001. d, > 0, fund empirically, is used in threshlding neurn activatin 2 max functins. Mrever, t cmply with the gradient based algrithms, uncnstrained threshlds τ, = 1, J,, which are mapped t the bunded threshlds τ by the sigmid functin (1 τ = d ) max / +, are intrduced [12]., e τ 5

6 Let us suppse that the cst functin depends n sme functin ε, which n the ther hand depends n the denised data sample, ε= ε ( p% ( q ), ). T calculate gradients f the functin ε with respect t the threshlds, an apprach similar t the ne f Trentin [15] has been used. Having in mind that each element f the denised data sample is btained frm all wavelet cefficients, ne can write ε ε p% ( q) τ =. τ τ τ N i i= 1 p% i( q) The first factr in the sum is btained frm the mdel t which the denised data sample is passed. Cnsidering the Equatin (5) and the mapping f threshlds, the secnd and the third factr can be expressed as with p% ( q) τ Tc ( ( q), τ ) τ τ τ C i R l = il τ τ d W, /, max l= 1 (1 ) (6) (7) Tc ( l( q), τ ) 1 cl( q) τ cl( q) + τ = +. τ ( c( q) τ ) + s ( c( q) + τ ) + s (8) 4. Neural Netwrk Mdel with Denising Layers The denising layers are usually placed between the neural netwrk mdel inputs and its first nnlinear layer. T demnstrate their universality, the integratin f the denising layers int the feedfrward tw-layered perceptrn and the twlayered perceptrn with glbal recurrent cnnectins is presented. Only equatins f the recurrent mdel are given, since they can be easily reduced t the feedfrward versin. The riginal mdels and their mdificatins are shwn in Figure 3. Figure 3 6

7 4.1. Tw-Layered Perceptrn with Recurrent Cnnectins The tw-layered perceptrn with glbal recurrent cnnectins starts by merging the inputs x ( q) = ( x1 ( q ),, x ( q)) T T t the last knwn mdel utputs y ( q 1). N Obtained vectr z( q) = ( y ( q 1) T, x ( q) T ) T is passed further t the N h nnlinear neurns in the hidden layer with utputs N + N h h h h h k ϕ ( k( )) k( ) ωk, l l( ) 1 h l= 0 y = s q, s q = z q, k =,, N, (9) where h h ϕ ( s) = tanh( s) is the nnlinear activatin functin, ω kl, are the adaptive weights and the bias z0( q ) = 1. The utputs f the hidden layer are further fed t the N neurns in the utput layer, giving the mdel utputs N h y = ϕ ( s ( q)), s ( q) = ω, y ( q), k = 1,, N, (10) h k k k k l l l= 0 with the nnlinear activatin ϕ ( s) = tanh( s), the adaptive weights ω kl, and the h bias y0 ( q ) = 1. Neural netwrk mdels are cmmnly trained n a set f knwn input utput pairs, { p( q), t ( q)}, with t ( q) being the vectr f target values. The cst functin usually depends n the errrs f each input utput pair, e = t y, k = 1, O,, k k k O N, and therefre its gradients can be expressed with the gradients f the mdel utputs. Accrding t the derivatin f the Real Time Recurrent Learning [16], tw dynamical systems are btained fr the tw-layered perceptrn with glbal recurrent cnnectins, determining the gradients f utputs, Nh N k h h h m i, k k, l l il l, m i, l= 1 m= 1 (11) ρ ( q) = ϕ ( s ( q)) ω ϕ ( s ( q)){ δ z ( q) + ω ρ ( q 1)} Nh N k h h h h m i, k ik k, l l l, m i, l= 1 m= 1 (12) π ( q) = ϕ ( s ( q)){ δ y ( q) + ω ϕ ( s ( q)) ω π ( q 1)}, 7

8 with ρ k ( q) = y ( q) / ω h and π k ( q) = y ( q) / ω. In Equatins (11) and (12), i, k i, i, k i, ϕ h () s and ϕ ( s) dente the derivatives f the crrespnding activatin functins and δ i is the Krnecker symbl. At the beginning f the learning prcess k k ρ, (0) = 0 and π, (0) = 0 is set. i i By setting the number f utputs N t zer in Equatins (9), (11) and (12), the equatins gverning the tw-layered perceptrn withut recurrent cnnectins are btained Integratin f Denising Layers The tw-layered perceptrn with glbal recurrent cnnectins and integrated denising layers is shwn in Figure 3b. The input sample p ( q) is first passed t the threshlding layer and then further t the recnstructin layer in rder t btain the denised sample p% ( q). The prcess is gverned by Equatins (4) and (5), respectively. The inputs f the tw-layered perceptrn with glbal recurrent cnnectins are then substituted with the denised data sample, x( q) = p% ( q). Applying the Equatins (9) and (10) the mdel utputs are finally cmputed. As pinted ut in sectin 1, the gradients f a cst functin can be btained frm the gradients f the mdel utputs. The gradients f the mdel utputs with respect t the weights f the neurns in the hidden layer and t the weights f the neurns in the utput layer are given by Equatins (11) and (12). The gradients f utputs with respect t the threshlds are btained frm Equatin (6) by setting the mdel utput y ( q ) as the functin ε. In this case, the first factr in Equatin (6) becmes k yk ( q) = ϕ ω ϕ ω. p% ( q) i Nh h h h ( sk( q)) k, l ( sl ( q)) l, N + i l= 1 Again, the equatins f the feedfrward tw-layered perceptrn with the denising layers are btained by setting N t zer. (13) 8

9 5. Results The prpsed denising layers were integrated int the tw-layered perceptrn (TLP) and the tw-layered perceptrn with glbal recurrent cnnectins (RTLP). The perfrmance f the mdels enhanced with the denising layers, the DTLP mdel and the DRTLP mdel, respectively, was tested n three ne-step-ahead predictin prblems: the lgistic map, the hardness f rubber cmpund and the yearly average sunspt number. The time series mdel usually cnnects a value f a time series n mdel utput with N previus values presented n mdel inputs. T establish such cnnectin, the mdel is trained n a set f knwn input utput pairs. Frm each time series, first 85% f input utput pairs were included in the training set, used t set free parameters f the mdels, while the remaining 15% f input utput pairs frmed the testing set, used fr mdel cmparisn. The ptimal values f tplgical parameters, which cannt be included in the gradient based algrithms, were fund by inspecting the free parameter space. The mdel cnfiguratins were allwed t have frm 4 t 20 inputs ( N ) with the number f free parameters nt exceeding 35% f the number f all input utput pairs. The wavelets frm symlet family [10], denising layers. M S, 2M N 1 +, were used in the cnstructin f the As the perfrmance measures, the rt mean squared errr, nrmalized t standard deviatin f a time series, (NRMSE) and the mean abslute percentage errr (MAPE) were cnsidered. On each mdel cnfiguratin, the Levenberg- Marquardt gradient algrithm [14] was used t adapt weights and threshlds. The learning was repeated 20 times and nly the cnfiguratins with the smallest NRMSE were used in further analysis Lgistic Map The lgistic map is knwn as a simple mdel which yields chas [17]. It is given by the recursive relatin xt = 4 xt 1(1 xt 1). In the fllwing experiments, 250 values were used, beginning with x 1 =

10 Detailed cmparisn f the applied mdels, each having 4 inputs, 10 neurns in the hidden layer and 1 neurn in the utput layer, is given in Table 1. Table 1 The number f neurns in threshlding and recnstructin layers is established by the number f inputs N and the wavelet rder M S, in these mdels the numbers are 9 and 4, respectively. All mdels succeeded t determine the relatinship 5 between the cnsecutive values. Small values f threshlds, i.e., τ = d 1 1max, 4 and τ = d f the DRTLP mdel, indicate that the mdels with 2 2, max denising layers are capable f distinguishing between chatic data and nise, which is nt the case when denising methds based n statistical criteria are applied [12]. Slightly higher errrs bserved at the mdels with the denising layers riginate in the mapping f the uncnstrained threshlds τ t the bunded threshlds τ, nt allwing the latter t becme zer in finite number f learning steps Hardness f Rubber Cmpund The quality f rubber prducts, such as bicycle and mtrcycle tubes, greatly depends n the quality f its cmpnents, particularly f a rubber cmpund. Thus, in the rubber industry the quality f a rubber cmpund after each mixing is permanently mnitred. An imprtant indicatr fr the quality f a rubber cmpund is its hardness. Variatin f hardness in 199 successive mixings tgether with sme denised data samples, btained by the denising layer f the DTLP and the DRTLP mdels, is shwn in Figure 4. Quite substantial denising perfrmed by bth, the DTLP mdel and the DRTLP mdel, reveals the nisy nature f the time series. Figure 4 10

11 The predictin errrs given in Table 2 shw that the denising layers imprve predictin: the DTLP mdel has an advantage ver the TLP mdel and the DRTLP mdel ver the RTLP mdel. Within the testing set, the NRMSE measure is decreased by mre than 6% and the MAPE measure by at least 3%. Larger predictin errrs n the training set may be explained by higher variatins f the training set data. Table 2 The success f the DTLP and the DRTLP mdels can be attributed t the fact that the denising layers have simplified the relatins between input and utput data samples by remving sme nise frm the input data samples. Mutual cmparisn f the mdels TLP and RTLP shws that recurrent cnnectins d nt necessarily imprve predictin. This is mst prbably due t the learning algrithms which are far mre cmplex fr recurrent neural netwrk mdels Sunspt number The number f sunspts, dark areas f cncentrated magnetic field n the Sun, is highly crrelated with the Sun activity. The yearly average f the sunspt number, cllected between the years 1700 and 2001, was used in the analysis [18]. Figure 5 shws the variatins f the yearly average f the sunspt number and sme denised data samples btained by the mdels DTLP and DRTLP. Figure 5 Denised data samples differ substantially. The DTLP mdel fund almst n nise in the time series, i.e., the details were remved mainly n the first scale, = 087. d, while the details n the ther scales were practically left intact. τ 1, max The DRTLP mdel, n the ther hand, perfrmed denising n all three scales, 11

12 i.e., τ 1 = 011d. 1max,, τ 2 = 018d. 2, max and τ 3 = 035d. 3, max, which reflects in reductin f amplitudes in denised data samples. Table 3 Detailed cmparisn f the mdels given in Table 3 shws that the predictin imprvement f the DTLP mdel cmpared t the TLP mdel is similar t the predictin imprvement f the DRTLP mdel ver the RTLP mdel, regardless f the cmpletely different cncepts f denising. In bth cases, the NRMSE and MAPE perfrmance measures are reduced frm 5% and even up t 25% n training sets. Furthermre, the recurrent cnnectins f the mdels RTLP and DRTLP had reduced the predictin errrs n the training set fr at least 7% in cmparisn t the mdels TLP and DTLP, respectively. Very large MAPE errrs n the training set, cmpared t the MAPE errrs n the test set, are due t the nature f the time series, having many small values in the training set. Namely, the MAPE perfrmance measure extremely penalizes errrs in predictin f small values. 6. Cnclusin The denising based n wavelet multireslutin analysis can be frmulated as a generalizatin f neural netwrk layers. Oppsed t standard neurns, in which free parameters determine the weighted sum, the free parameters f threshlding neurns determine the shape f threshlding functins. A gradient based algrithm fr setting the free parameters f threshlding neurns was develped. This algrithm enables fr the same cst functin t be used fr setting bth the weights f standard neurns and the threshlds f threshlding neurns. In this way, the denising layers can be cmpletely integrated int many types f layered neural netwrk and denising is nt treated as a separate prcess. Althugh the learning algrithm is designed fr gradient based methds, it can easily be generalized t varius nn-gradient algrithms, including evlutinary. 12

13 The presented examples f applicatin n ne-step-ahead predictin prblems revealed that the denising layers imprve predictin n nisy data sets. The utilizatin f the cst functin which minimizes predictin errr ensures that denising is perfrmed in such a way that the ptimal predictin is achieved. Unlike statistical methds, the neural netwrks with denising layers can distinguish between nise and chas and practically n denising is perfrmed in the latter case. Acknwledgements The prect has been funded in part by the Slvenian Ministry f Educatin, Science and Sprt under Grant N. Z References 1. Masters T (1995) Neural, Nvel & Hybrid Algrithms fr Time Series Predictin. Jhn Wiley & Sns, Trnt. 2. Weigend A S, Gershenfeld N A (1994) Time Series Predictin: Frecasting the Future and Understanding the Past. Addisn Wesley, Reading. 3. Frsee F D, Hagan M T (1997) Gauss-Newtn Apprximatin t Bayesian Regularizatin. In Prceedings f the 1997 IEEE Internatinal Jint Cnference n Neural Netwrks, Hustn, pp Drucker H, Cun Y L (1992) Imprving Generalizatin Perfrmance Using Duble Backprpagatin. IEEE Trans. Neural Netw. 3 (6): Ster B (2003) Latched recurrent neural netwrk. Electrtechnical Review 70 (1-2): 46-51, 6. Dnh D L (1995) De-Nising by Sft-Threshlding. IEEE Trans. Inf. Thery 41: Prchazka A, Mudrva M, Strek M. (1998) Wavelet Use fr Nise Reectin and Signal Mdelling. In Prchazka A et al. (eds). Signal Analysis and Predictin, Birkhauser, Bstn, pp Zhang Q, Benveniste A (1992) Wavelet Netwrks. IEEE Trans. Neural Netw. 3: Quipeng L, Xialing Y, Quanke F (2003) Fault Diagnsis Using Wavelet Neural Netwrks, Neural Prcessing Letters 18: Daubechies I (1992) Ten Lectures n Wavelets, SIAM, Philadelphia. 11. Ltric U, Dbnikar A (2003) Matrix Frmulatin f Multilayered Perceptrn with Denising Unit. Electrtechnical Review 70, in press, Ltric U (2000) Using Wavelet Analysis and Neural Netwrks fr Time Series Predictin, Ph. D. Thesis. University f Lublana, Faculty f Cmputer and Infrmatin Science, Lublana. 13. Haykin S (1999) Neural Netwrks: A Cmprehensive Fundatin, 2nd editin. Prentice-Hall, New Jersey. 14. Press W H, Teuklsky S A, Vetterling W T, Flannery B P (1992) Numerical Recipes in C, 2nd editin, Cambridge University, Cambridge. 13

14 15. Trentin E (2001) Netwrks with Trainable Amplitude f Activatin Functins. Neural Netw. 14: Williams R J, Zipser D (1989) A Learning Algrithm fr Cntinually Running Fully Recurrent Neural Netwrks. Neural Cmputatin 1: Schuster H G (1984) Deterministic Chas. An Intrductin,. Physik, Weinheim. 18. Sunspt Index Data Center, Ryal Observatry f Belgium (2001) Yearly Definitive Sunspt Number, 14

15 Tables: Table 1. Mdel cmparisn in predictin f the lgistic map. Mdel Structure NRMSE MAPE [Free parameters] Training Testing Training Testing TLP [61] DTLP 2 4-(9-4)-10-1, S [64] RTLP [71] DRTLP 4-(9-4)-10-1, 2 S [74] Table 2. Mdel cmparisn in predictin f rubber hardness. Mdel Structure NRMSE MAPE [Free parameters] Training Testing Training Testing TLP [21] DTLP 2 19-(53-19)-1-1, S [27] RTLP [15] DRTLP 18-(76-18)-1-4, 8 S [36] Table 3. Mdel cmparisn in predictin f the yearly average sunspt number. Mdel Structure NRMSE MAPE [Free parameters] Training Testing Training Testing TLP [34] DTLP 5 9-(36-9)-8-1, S [93] RTLP [55] DRTLP 11-(37-11)-3-9, 5 S [103]

16 Figure legends: Figure 1. Influence f the parameter s n the shape f the generalized sft threshlding functin. Figure 2. Wavelet based denising layers. Figure 3. a) The tw-layered perceptrn with glbal recurrent cnnectins N - N - N and b) its enhancement with the wavelet based denising layers N -(C - h N )- N - N. When the recurrent cnnectins (dashed) are mitted, the h feedfrward tw-layered perceptrn (a) and the tw-layered perceptrn with the denising layers (b) are btained. Figure 4. Variatin f a rubber hardness with examples f the denised data samples p% ( q), btained by the denising layers f the mdels DTLP and DRTLP. Figure 5. Variatin f the yearly average sunspt number with examples f the denised data samples p% ( q), btained by the denising layers f the mdels DTLP and DRTLP. 16

17 Figures Figure 1 T ( dk, ) t s>0 s=0 -t t d k Figure 2 recnstructin layer N-dimensinal denised data sample p( ~ q) threshlding layer grup 1 1 grup grup J grup J+1 0 N-dimensinal data sample p( q) Figure 3 a) y () q b) y () q 1 N 1 N D -1 D -1 nnlinear layers y h() q 1 N h D -1 D -1 nnlinear layers y h() q 1 N h 1 N x( q) y ( q-1) denising layers p( ~ q) 1 N 1 C 1 N p( q) 17

18 Figure 4 Figure 5 18