Multgradent for Neural Netorks for Equalzers 1 Chulhee ee, Jnook Go and Heeyoung Km Department of Electrcal and Electronc Engneerng Yonse Unversty 134 Shnchon-Dong, Seodaemun-Ku, Seoul 1-749, Korea ABSTRACT Recently, a ne tranng algorthm, multgradent, has been publshed for neural netorks and t s reported that the multgradent outperforms the backpropagaton hen neural netorks are used as a classfer When neural netorks are used as an equalzer n communcatons, they can be veed as a classfer In ths paper, e apply the multgradent algorthm to tran the neural netorks that are used as equalzers Experments sho that the neural netorks traned usng the multgradent notceably outperforms the neural netorks traned by the backpropagaton Keyords: Equalzer, multgradent, neural netorks, tranng algorthm, pattern classfcaton 1 Introducton Neural netorks have been successfully appled n pattern recognton, sgnal processng, and communcatons In partcular, there has been a great nterest n usng neural netorks to mplement equalzers hch can be veed as classfcaton problems hose dstrbuton functons are unknon [1, ] Many researchers reported that neural netorks could be a promsng soluton to equalzaton problems and proposed varous mplementatons When neural netorks are used as an equalzer, one of the most frequently used tranng algorthms s the backpropagaton algorthm Recently, a ne tranng algorthm, hch s called multgradent, has been proposed [3] The multgradent s a specalzed tranng algorthm hen neural netorks are used as a classfer It has been reported that the multgradent outperforms the backpropagaton algorthm n pattern classfcaton [3] Snce neural netorks are used as a classfer hen they are used as equalzers, the multgradent algorthm can be used for such neural netorks In ths paper, e apply the multgradent algorthm to neural netorks that are used as equalzers and evaluate the performance Channel Equalzaton Problem If nput sgnal x( s transmtted through a lnear dspersve channel of fnte mpulse response th the coeffcents a k, the receved sgnal y( can be modeled by y( = ak x( n k) + e( k = here e( s the addtve hte Gaussan nose specfed by the follong statstcs: E e( =, E e( e( m) = σ e δ ( n m [ ] [ ] ) e here σ s nose varance The nput sgnal x( s chosen ndependently from {-1, 1} th equal probablty and equalzaton s to estmate the orgnal nput sgnal x( from the receved sgnal y( n the presence of nose and nterference Equalzers have been mportant n dgtal communcaton systems to guarantee a relable data transmsson and numerous equalzaton algorthms have been proposed Among varous equalzaton methods, lnear equalzaton has been dely used due to ther speed and smplcty The lnear equalzer s frequently mplemented usng the least mean square (MS) algorthm as follos: W = W + cλy n+1 n n 1 The Korea Scence and Engneerng oundaton partly supported the publcaton of ths paper through BERC at Yonse Unversty 1 SYSTEMICS, CYBERNETICS AND INORMATICS VOUME 1 - NUMBER 3
here T Y n = [ y( n ), y( n 1), y(, y( n + 1), y( n + )], λ s the learnng rate, c s 1 f sgnal 1 s transmtted and 1 f sgnal 1 s transmtted The lnear equalzer can perfectly reconstruct the orgnal nput sgnal f the receved sgnal s lnearly separable Hoever, the decson boundary for equalzaton s hghly nonlnear n many cases and neural netorks hch can form an arbtrary nonlnear decson boundary can be better adopted for equalzaton 3 Multgradent [3] A typcal neural netork has the nput layer, a number of hdden layers, and the output layer g 1 shos an example of 3-layer feedforard neural netorks for a pattern-class problem The decson rule s to choose the class correspondng to the output neuron th the largest output [4] In g 1, X n = (x 1, x,, x M ) T represents the nput vector, Y = (y 1, y ) T the output vector, and B = (b 1, b ) T the bas vector We may nclude the bas term n the nput layer as follos: X = (x 1, x,, x M,1) T = (x 1, x,, x M, x M +1 ) T here x M +1 = b 1 = 1 Assumng that there are K neurons n the hdden layer, the eght matrces W1 and W for the pattern class neural netork can be represented by h h h h 1,1 1, 1, M 1, M + 1 h h h h =,1,, M, M + 1 W 1 : : : : h h h h K,1 K, K, M K, M + 1 1,1 1, 1, K 1, K + 1 W =,1,, K, K + 1 here h j, s the eght beteen nput neuron and hdden neuron j and k, j s the eght beteen hdden neuron j and output neuron k In order to tran the neural netork, e need to fnd matrces and W that produce a desrable W1 sequence of output vectors for a gven sequence of nput vectors et W be the vector contanng all eghts In other ords, W = ( h 1,1, h h 1,,, K, M +1, 1,1, 1,,,,K +1 ) T = ( 1,, 3,, ) T here =((M+1)K+(K+1)) and K s the number of hdden neurons Then, e may ve W as a pont n the dmensonal space In the above example, there are ((M+1)K+(K+1)) eghts to adjust et be the vector contanng all the elements of W W 1 and W : W = ( h 1,1, h h 1,,, K, M +1, 1,1, 1,,,,K +1 = ( 1,, 3,, ) T here =((M+1)K+(K+1)) Then, W can be veed as a pont n the dmensonal space In ths paradgm, the learnng process can be veed as fndng a soluton pont n the dmensonal space x 1 x x M W1 bas Xn X Z Z Y Y g 1 An example of 3-layer feedforard neural netorks ( pattern classes) In multlayer feedforard neural netorks, the output vector Y can be represented as a functon of X and W : Y = y 1 = 1(X,W ) (X, W) y W ) T y 1 y SYSTEMICS, CYBERNETICS AND INORMATICS VOUME 1 - NUMBER 3 11
assumng a pattern-class classfcaton problem Durng learnng phase, f X belongs to class ω 1, e move W n such a drecton that y 1 ncreases and y decreases We can fnd the drecton by takng the gradents of y 1 and y th respect to W : y = y 1 + y 1 + + y here { } s a bass of the -dmensonal space Thus, f e update W n the drecton of αy 1 βy, here α, β >, y 1 ll ncrease and y ll decrease In general, e update the eght vector W as follos: W updated = W + γ (c 1 y 1 + c y ) (1) here γ s the learnng rate Ths procedure s llustrated n g If there are N output neurons, then the eght vector W s updated as follos: number of possbltes to set c n (1) If e setc to be the dfference beteen the target value and the output value, the mult-gradent algorthm s equvalent to the backpropagaton algorthm In [3], assumng that the target value s ether 1 or 9, as set as follos: c c = t t y y f target value t = 9 and f target value t = 1 and y otherse y < 9 > 1 In other ords, e gnore the output neurons that exceed the target values and concentrate on the output neurons that do not meet the target values, updatng eghts accordngly Snce the classfcaton accuracy s the most mportant crteron hen neural netorks are used as a classfer, ths eght update strategy can be effectve, provdng better classfcaton accuraces W updated = W + γ + + + c ( c1 y1 c y N y N here c f X belongs to class ω and c otherse ) y αy 1 βy y 1 Assumng the sgmod functon s used as the actvaton functon, t can be shon that dfferentatng y 1, y th respect to the eghts beteen the hdden layer and the output layer can be obtaned as follos: k, j y k = y k (1 y k )z j ( k = k) ( k k) k, j here s the eght beteen hdden neuron j and output neuron k and z j s the output of hdden neuron j Smlarly, dfferentatng y 1, y th respect to eghts beteen the nput layer and hdden layer yelds here y k h j, h j, = y k (1 y k ) k, j z j (1 z j )x s the eght beteen nput neuron k, j and hdden neuron j and s the eght beteen hdden neuron j and output neuron k There are a y g Adjustng eghts by addng the gradents 4 Experments and Results Experments ere conducted for a symmetrc channel and a non-symmetrc channel In the frst experment, e generated 1, samples for the follong symmetrc channel: y( = k = a k x(n k ) + e( here =, a 1 = a 5 = 5, a = a 4 = 7, a 3 = 1, and σ e = 1 Among the 1, samples, the frst 1 samples are used for tranng and the rest are used for testng g 3 shos the performance comparson of the multgradent and the backpropagaton algorthms As can be seen, the multgradent notceably outperforms the backpropagaton When the backpropagaton as 1 SYSTEMICS, CYBERNETICS AND INORMATICS VOUME 1 - NUMBER 3
used, the classfcaton accuraces for the tranng and test data are 8% and 5%, respectvely When the netorks are traned by the multgradent, the classfcaton accuraces for the tranng and test data are % and 3%, respectvely In the second experment, e generated 1, samples for the follong channel: y( = k = a k x(n k ) + e( here =, a 1 =, a = 8, a 3 = 1, a4 = 7, a 5 = 3 and σ e = It s noted that the channel s non-symmetrc As prevously, the frst 1 samples are used for tranng and the rest are used for testng g 4 shos the performance comparson Wth the backpropagaton, the classfcaton accuraces for the tranng and test data are 939% and 939%, respectvely When the netorks are traned by the multgradent, the classfcaton accuraces for the tranng and test data are 956% and 949%, respectvely As n the symmetrc channel, the multgradent outperforms the backpropagaton The multgradent also converges faster the backpropagaton 5 Conclusons [] S Chen, B Mulgre, and P M Grant, "A clusterng technque for dgtal communcatons channel equalzaton usng radal bass functon netorks," IEEE Trans Neural Netorks, vol 4, no 4, pp 57-579, July 1993 [3] J Go, G Han, H Km and C ee, "Multgradent: a ne neural netork learnng algorthm for pattern classfcaton," IEEE Trans Geoscence and Remote Sensng, vol 39, no 5, pp 9-993, May 1 [4] R P ppmann, "An Introducton to Computng th Neural Nets," IEEE ASSP Magazne, vol 4, no, pp 4-, 19 (a) Classfcaton accuracy (%) 9 83 1 3 4 5 In ths paper, e appled the recently publshed multgradent tranng algorthm to neural netorks that are used as an equalzer It as reported that the multgradent algorthm outperforms the backpropagaton hen neural netorks are to be used as a classfer Experments th symmetrc and non-symmetrc channels shoed that the multgradent algorthm provded notceable mprovements over the conventonal backpropagaton (b) Classfcaton accuracy (%) 9 No Iter References 83 1 3 4 5 [1] G J Gbson, S Su and C N Coan, "Multlayer perceptron structures appled to adaptve equalzers for data communcatons," Proc IEEE ICASSP, pp 1183-11, May 19 No Iter g 3 Performance comparson for a symmetrc channel (a) tranng data, (b) test data SYSTEMICS, CYBERNETICS AND INORMATICS VOUME 1 - NUMBER 3 13
(a) 96 95 94 Classfcaton accuracy (%) 93 9 9 1 3 4 5 No Iter (b) 96 95 94 Classfcaton accuracy (%) 93 9 9 1 3 4 5 No Iter g 4 Performance comparson for a nonsymmetrc channel (a) tranng data, (b) test data 14 SYSTEMICS, CYBERNETICS AND INORMATICS VOUME 1 - NUMBER 3