A Novel Learning Method for Elman Neural Network Using Local Search

Transcription

1 Neura Information Processing Letters and Reviews Vo. 11, No. 8, August 2007 LETTER A Nove Learning Method for Eman Neura Networ Using Loca Search Facuty of Engineering, Toyama University, Gofuu 3190 Toyama city, , Japan E-mai: ztang@eng.u-toyama.ac.jp, daaosha@hotmai.com (Submitted on May15, 2007) Abstract Eman Neura Networ (ENN) have been efficient identification too in many areas since they have dynamic memories. However, the oca minima probem usuay occurs in the process of the earning because of the empoyed bac propagation agorithm. In this paper, we propose a nove earning method for ENN by introducing adaptive earning parameter into the traditiona oca search agorithm. The proposed earning networ requires ess memory and it is abe to overcome the disadvantages of the gradient descent. Meanwhie it is aso abe to acceerates the speed of the convergence and avoid the oca minima probem in a certain extent. We appy the new method to the Booean Series Prediction Questions to demonstrate its vaidity. Simuation resuts show that the proposed agorithm has a better abiity to find the goba minimum than bac propagation agorithm within reasonabe time. Keywords Eman Neura Networ (ENN), Loca Search (LS) Method, Adaptive Learning Parameter, Booean Series Prediction Question (BSPQ) 1. Introduction Eman Neura Networ (ENN) is a type of partia recurrent neura networ, which consists of two-ayer bac propagation networs with an additiona feedbac connection from the output of the hidden ayer to its input ayer[1]. The advantage of this feedbac path is that it aows ENN to recognize and generate tempora patterns and spatia patterns. This means that after training, interreations between the current input and interna states are processed to produce the output and to represent the reevant past information in the interna states [2-3]. As a resut, the ENN has been widey used in various fieds from a tempora version of the Excusive-OR function to the discovery of syntactic or semantic categories in natura anguage data. However, since ENN often uses bac propagation (BP) to dea with the various signas, it has proved to be suffering from a sub-optima soution probem [4-5]. At the same time, for the ENN, it is ess abe to find the most appropriate weights for hidden neurons and often get into the sub-optima areas because the error gradient is approximated [6].Furthermore, The efficiency of the ENN is imited to ow order system due to the insufficient memory capacity [7]. Therefore, severa approaches have been suggested in the iterature to increase the performance of the ENN with simpe modifications [8-10]. Aso these improved modifications attempt to add other feedbac connections factors to the mode that wi increase the capacity of the memory in order to overcome the tendency to sin into oca minima. However, gradient descent (bac propagaton) used by ENN has the disadvantages of being trapped in oca minima resuting in sub optima soutions and cacuations are not as straightforward since it requires functiona derivatives. So the gradient descent method for ENN sti hods various disadvantages which is difficuty to overcome. In this wor we pan to expore the potentia of using an adaptive oca search as earning in order to overcome to the disadvantages of gradient descent. LS method can be more robust to oca minima than derivative-based methods. It can be appied to probems where the derivative information is not reiabe or even to probems where the function is not differentiabe, nor even continuous[11]. So here we are proposing a new agorithm method for the ENN to overcome its weaness (the compexity of the derivative). 181

2 A Nove Learning Method for ENN Using Loca Search Figure 1. The Structure of the ENN Figure 2. Interna Process Anaysis of ENN 2. Eman Neura Networ Structure Figure 1 shows the structure of a simpe ENN. In Figure 1, after the hidden units are cacuated, their vaues are used to compute the output of the networ and are aso a are stored as "extra inputs" (caed context unit) to be used when the next time the networ is operated. Thus, the recurrent contexts provide a weighted sum of the previous vaues of the hidden units as input to the hidden units. As shown in the Figure 1, the activations are copied from hidden ayer to context ayer on a one for one basis, with fixed weight of 1.0 (w=1.0). The forward connection weight is trained between hidden units and context units as we as other weights. If sef-connections are introduced to the context unit when the vaues of the sef-connections weights (a) are fixed between 0.0 and 1.0 (usuay 0.5) before the training process, it is an improved ENN as proposed by Pham and Liu [4]. When weights (a) are 0, the networ is the origina ENN. Figure 2 is the interna earning process of ENN by the error bac-propagation agorithm. From Figure 2 we can see that training such a networ is not straightforward since the output of the networ depends on the inputs and aso a previous inputs to the networ. So, it shoud trace the previous vaues according to the recurrent connections(figure 3). So, the cacuation of the functiona derivatives is not straightforward and it eads to ow efficiency to dea with various signa probems. Figure 3 shows that a ong ENN where by a bac propagation is used to cacuate the derivatives of the error (at each output unit) by unroing the networ to the beginning. At the next time step t+1 input is represented. The context units contain vaues which are exacty the hidden unit vaues at time t (and the time t-1, t-2 ) and these context units provide the networ with memory [5]. Therefore, the ENN networ is converted into a dynamica networ that is efficient in the use of tempora information of the input sequence, both for cassification as we as for prediction [12-13]. However, the efficiency of the ENN is imited to ow order system due to the insufficient cacuation of the derivatives in some degree. The earning commony used in the ENN is bacpropagation agorithm since it adjusts the networ parameters (weights and threshods) to minimize the error measure function (Eq.(1)) using a gradient descent technique. Therefore it needs to compute the differentias of the error measure, activation function and their anaog mutipications. Figure 3. Unro the ENN through Time 182

3 Neura Information Processing Letters and Reviews Vo. 11, No. 8, August 2007 E = T E P t= 0 where p indexes over a the patterns for the training set in the time interva [0, T]. In our paper, the time eement is updated by the next input of the pattern from training set. So we can get the fowing equation. E p is defined by T E = E P = E t= 0 P p= 1 = 2 p ( t 2 j pj o pj ) p 1 E (3) where t pj is the target vaue (desired output) of the j-th component of the output for pattern p and o pj is the j-th unit of the actua output pattern produced by the presentation of input pattern p at the time, and j indexes a the output units. 3. Agorithm Loca Search is one of the many optimization methods avaiabe for soving computationa probems. Based on the most fundamenta understanding of how a LS moves from candidate soution to a neighbouring soution interativey, we have appied this fundamenta methodoogy on the muti-ayer ENN. We define a vector V whose eements incude a adaptabe parameters (weights between a ayers and threshods of each neuron) as V = ( w 11, w12,... w ij..., θ1, θ 2,..., θi,...) (4) The error function E can be expressed as E = E(V) (5) Then, we can iterativey adjust V to minimize the error function E(V). First, the search starts at an initia point and moves aong one of d directions, where d denotes the number of eements of the vector V. Then we can define the -th direction vector at iteration as n T e = ( 0,...,0,1,0, ) (6) The sequence of iterations V 0, V 1,...V... in R n can be described as foows. For the -th iteration ( 0), a + positive change V in a direction e with a positive step as when R 1 ( V is the j-th eement in V ), resuts in a change E + as V T V + = e, (7) E + = E( V + e ) E( V ). (8) Simiary, a negative change V in a direction e with a positive step as when R 1, resuts in a change V E in E as V = e (9) ( E = E V e ) E( V ) (10) where is a positive constant and usuay =1 for a. (When =0, the initia iterate V 0 and the step size 0 is given.) Then the foowing earning rue appies. + + V + e if E < 0 and E < E + V + 1 = V e if E < 0 and E < E (11) V otherwise Here, we adapt the constant as a variabe in order to acceerate the convergence by setting different vaues of in different intervas of the E vaue (β = 1.0). In the simuation, we set the different as (1) (2) 183

4 A Nove Learning Method for ENN Using Loca Search β if E > 1 = 0. 1* β if 0 < E < 1. (12) 0. 01* β otherwise As can be seen, our agorithm performs the oca search iterativey and minimizes the error measure function aong with the set of decent directions directy. Sufficient search directions are incuded to guarantee that if the current iteration is not a stationary point, Then, it can find the nearest minima efficienty. Furthermore, the earning is performed by simpy changing the weight and threshod vectors by a sma positive or negative constant, and accepting the change if it produces a smaer error measure. Moreover, the agorithm is extremey simpe to specify and impement in hardware appications for the foowing reasons: As a direct search method, no bac propagation pass is needed and ony a forward path is required. This means, in terms of anaog impementations, that no bidirectiona circuits and hardware for the bac propagation are needed. No compex anaog mutipiers and other anaog computations are needed to generate the derivative. 4. Simuations In order to test the effectiveness of the proposed adaptive oca search method for ENN, we compare its performance with those of the origina ENN agorithm (proposed by Eman [1]) and improved ENN agorithm (proposed by Pham and Liu [4]) on a series of BSPQ probems incuding "11", "111" and "00" probems. Because the training procedures of our agorithm and the bac propagation ENN agorithm are different, we use the CUP time instead of the iteration steps. Three agorithms started from randomy generated weights and threshods. The error criterion MSE was set to 0.1 and At the same time,every method wi run 100 trias. Two aspects of training agorithm performance-"success rate" and "training time" were assessed for each agorithm. A the simuations were performed on a Pentium4 2.8GHz, 1G PC. We used the modified bac propagation agorithm with momentum 0.9 for the origina ENN and improved ENN. For a the trias, 200 patterns were provided to satisfy the equiibrium of the training set and 20 random pattern were acted as test set. 4.1 "11" Questions Booean Series Prediction Questions is one of the probems about time sequence prediction. First et us see the definition of the BSPQ. Suppose that we want to train a networ with an P input and T targets as defined beow [14]. P= and T = Here T is defined to be 0, except when two 1's occur in P in which case T is 1 and we caed this probem as "11" probem (one ind of the BSPQ ). Aso when "00"or "111" (two 0's or three 1's) occurs, it is named as the "00" or "111" probem. Firsty, we dea with the "11" question and anayze the effect of the memory in the context ayer of the networ. Tabe 1 compares the training simuation resuts of the three methods, we can see that our proposed method coud amost 100% succeeds to get the convergent criterion. Of course, the origina ENN was abe to predict the requested test set, but the training success rate was sow. Furthermore, it succeeded ony 75% when E was set to 0.1. And the improved ENN has increased the abiity of the dynamic memorization of the networ because of the sef-connection weights gene (a=0.5). Athough improved ENN coud acceerate the convergent of the earning process (time was ess than origina ENN), it coud not essentiay avoid the oca minima probems because of the characteristics of the gradient descent technique. Tabe 1. Experiment resuts for the 11 question with 5 neurons in the hidden ayer Success rate Average CPU time Methods (100 trias) (second) (1-5-1 networ) MSE=0.1 MSE=0.01 MSE=0.1 MSE=0.01 Origina ENN 75% 72% improved ENN 83% 82% propose ENN 100% 98%

5 Neura Information Processing Letters and Reviews Vo. 11, No. 8, August 2007 Figure 4. Training error curves of three ENN agorithm Figure 5. Expected Output T 1 and prediction resut with Proposed ENN Figure 4 shows the earning characteristics for the three methods with the same initiaization weight for the networ (1-5-1), when E was set to From the simuation resuts we can see that the proposed ENN agorithm ony needed about 3.5 seconds to be successfu, but the origina ENN and the improved ENN agorithm do not have goa vaue and gets trapped into oca minima point A and B, respectivey. The improved ENN have acceerated the earning process but it coud not escape the oca minima probem. However, with our proposed ENN agorithm it coud get the convergent point by avoiding the oca minima. We define the prediction test set P 1 randomy as stated in the 20's figures beow. P 1 = And we used the we trained networ to do the fina prediction about sequence P 1 to test the prediction capabiity of it. For the prediction set P 1, we can get its corresponding target resuts (for 11 question) with beow T 1. T 1 = Figure 5 is the simuation prediction resut of P 1 for the "11" question with our proposed ENN agorithm. In Figure 5, the two ines represented the T 1 ine and the prediction resuts ine respectivey. From Figure 5 we can see that the toerance for every pattern was ess than So the networ has enough abiity to do the prediction of the given tas as desired. For the same probem, as we graduay increased the quantity of the neuron of the hidden ayer, the origina ENN and improved ENN were abe to attain the convergence point with the finite iterative steps, however with our proposed ENN it amost got 100% successfu rate for error criterion by enhancing the search'capacity of the networ. The resuts are shown in Tabe 2. In order to better testify the effectiveness of the proposed method, we continued to increase the number of neuron in the hidden ayer. From the Tabe 3, we can see, the proposed method coud get amost 100% success rate than the other two agorithms by using adaptive LS method. However, the simuation time required by the porposed method is much onger than the rest methods at times. 4.2 "111" and "00" Questions As we change rues of the input sequence, we can continue to testify the vaidity of our proposed ENN agorithm. The specific parameter set of the networ were same as the "11" questions. Tabe 4 is the specific comparison resuts from the "111" question for the three agorithms. From Tabe 4 we can see that, for the networ (1-7-1), the success rate of the proposed ENN was ower (ony 93% when the error criterion was set to 0.1) because of the insufficient capacity with ess hidden unit nodes. As the number of unit nodes in the hidden was increased, our proposed ENN agorithm coud amost attain 100% succeess rate than the other two agorithms athough the time was onger. The proposed ENN has superior advantages than other two agorithms which use gradient ascent technique. 185

6 A Nove Learning Method for ENN Using Loca Search Tabe 2. Experiment resuts for the 11 question with 7 neurons in the hidden ayer Methods(1-7-1 Success rate(100 trias) Average CPU time (second) networ) MSE =0.1 MSE =0.01 MSE =0.1 MSE =0.01 Origina ENN 86% 82% Improved ENN 95% 93% Proposed ENN 100% 97% Tabe 3. Experiment resuts for the 11 question with 10 neurons in the hidden ayer Methods( Success rate(100 trias) Average CPU time (second) networ) MSE =0.1 MSE =0.01 MSE =0.1 MSE =0.01 Origina ENN 95% 89% Improved ENN 95% 91% Proposed ENN 100% 98% Tabe 4. Experiment resuts for the 111 question with different neuron units in the hidden ayer Structure of the Networ networ networ networ Items Success rate(100 trias) Average CPU time (second) Methods MSE =0.1 MSE =0.01 MSE =0.1 MSE=0.01 Origina ENN 61% 53% Improved ENN 80% 75% Proposed ENN 93% 90% Origina ENN 79% 73% Improved ENN 85% 79% Proposed ENN 100% 96% Origina ENN 85% 75% Improved ENN 96% 91% Proposed ENN 100% 98% Tabe 5. Experiment resuts for the 00 question with different neuron units in the hidden ayer Structure of the Networ networ networ networ Items Methods Success rate(100 trias) Average CPU time (second) MSE=0.1 MSE=0.01 MSE=0.1 MSE=0.01 Origina ENN 85% 76% Improved ENN 91% 89% Proposed ENN 97% 97% Origina ENN 89% 86% Improved ENN 95% 92% Proposed ENN 99% 97% Origina ENN 90% 87% Improved ENN 97% 95% Proposed ENN 100% 98% Tabe 5 is the specific comparison resuts from the "00" question for the three agorithms. From Tabe 5 we can see that, as the compexity of the probem was increased, the training time aso increased for three agorithms, but our proposed agorithm was much more effective on the compicated BSPQ probem for the requested error criterion. Athough the BSPQ probem is ony a simpe earning tas for ENN, the other probems such as the detection of the wave ampitude can aso be repeated with the proposed agorithgm to test its effectiveness. We fee there is a posibiity for the proposed agorightmn to be appied as we as in the Jordan networ or other partiay modified recurrent neura networs since the structure of the ENN and the above mentioned networs are amost simiar. 186

7 Neura Information Processing Letters and Reviews Vo. 11, No. 8, August Concusions In this paper, we proposed a adaptive oca search method for Eeman Neura Networ. The approach was shown to be of high convergence to goba minimum than the other agorithms. The proposed agorithm was appied to the BSPQ probems of "11", "111"and "00" probems. The simuation resuts showed that the oca search agorithm is an effective new method which can get the requested goba minimum and attained high success rate. Besides that by using oca search to repace the bacpropagation method cacuations are easier since no derivatives are required and oca search is straightforward. However one of the drawbac of the proposed agorithm is that the processing time can sometimes tae onger than the reguar methods. Athough the proposed agorithm is ony been experimented with ENN, we somehow fee that there are posibiity for it to be impemented in Jordan networ or other partiay modified recurrent neura networs due to the simiarities of the structure. This area of research woud be one of our foregoing efforts in expanding the effectiveness of oca search agorithm in networs simiar with ENN. References [1] J. L. Eeman, "Finding Structure in Time", Cognitive Science,14, ,1990. [2] C.W. Omin, C.L. Gies, Extraction of rues from dicrete-time recurrent neura networs, Neura Networs 9(1),41-52,1996. [3] P. Stage, B. Sendhoff, Organisation of past states in recurrent neura networs: impicit embedding, in: M. Mohammadian (Ed.), Computationa Inteigence for Modeing, Contro & Automation, IOS Press, Amsterda.pp.21-27,1999. [4] D.T. Pham and X. Liu, "Identification of inear and noninear dynamic systems using recurrent neura networs", Artificia Inteigence in Engineering, Vo.8,pp.90-97,1993. [5] G. Cybeno, "Approximation by superposition of a sigmoid function", Mathematics of Contro, Signas, and Systems, 2, , [6] A. Smith,"Branch prediction with Neura Networs: Hidden ayers and Recurrent Connections", Department of Computer Science University of Caifornia, San Diego La Joa, CA 92307, [7] A. Kaini and S. Sagirogu, "Eeman Networ with Embedded Memory for System Identification", Journa of Informaiton Science and Engineering 22, ,2006. [8] D.P. Kwo, P. Wang, and K. Zhou,"Process identification using a modified Eeman neura networ", Internationa Symposium on Speech, Image Processing and Neura Networs, 1994,pp, [9] X.Z.Gao, X.M. Gao, and S.J. Ovasa, "A modified Eeman neura networ mode with appication to dynamica systems identification", in Proceedings of the IEEE Internationa Conference on System, Man and Cybernetics, Vo.2, pp ,1996. [10] W. Chagra, R. B. Abdennour, F. Bouani, M. Ksouri, and G. Favier, "A comparative study on the channe modeing using feedforward and recurrent neura networ structures," in Proceedings of the IEEE Internationa Conference on System, Man and Cybernetics, Vo.4, pp ,1998. [11] M. Momma, K.P. Bennett, "A Pattern Search Method for Mode Seection of Support Vector Regression", 2003 [12] S. Lawrence, C.L. Gies, S. Fong, Natura anguage grammatica inference with recurrent neura networ, IEEE Trans. Knowedge Data Eng. 12(1), , [13] J.T. Conner, D. Martin, L.E. Atas, recurrent neura networs and robust time series prediction, IEEE Trans. Neura Networs 5(2) , [14] 187

8 A Nove Learning Method for ENN Using Loca Search Zhiqiang Zhang received his B.S degree and M.S. degree from Shandong University, Jinan, Shandong, China, in 2002, 2005 and in Computer Science and Management Science, respectivey. He is currenty woring for his Ph.D. degree at Toyama University, Japan. His main research interests are neura networs and optimizations. Zheng Tang received a B.S. degree from Zhejiang University, Zhejiang, China in 1982 and an M.S. degree and a D.E. degree from Tsinghua University. From 1988 to 1989 he was an Instructor in the Institute of Microeectronics an Tsinghua University. From 1990 to 1999, he was an Associate Professor in the Department of Eectrica and Eectronic Engineering, Miyazai University, Miyazai, Japan. In 2000, he joined Toyama University, Toyama, Japan, where he is currenty a Professor in the Department of Inteectua Information Systems. His current research interests incude inteectua information technoogy, neura networs, and optimizations. Catherine Vairappan received the B.E. degree in Eectronics Engineering from Mutimedia University in Maaysia and M.S. in Information Systems from Northern Kentucy University, USA in 2000 and 2004 respectivey. Currenty a candidate for Ph.D degree in Information System Engineering. Her research interests are in the area of earning agorithms in neura networ and optimization probems. 188