Letters. Comments on Approximation Capability in by Multilayer Feedforward Networks and Related Problems

Transcription

1 714 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 9, NO. 4, JULY 1998 Letters Comments on Approxmaton Capablty n by Multlayer Feedforward Networs and Related Problems Guang-Bn Huang and Haroon A. Babr Abstract In the above paper 1 Chen et al. nvestgated the capablty of unformly approxmatng functons n C(R n ) by standard feedforward neural networs. They found that the boundedness condton on the sgmodal functon plays an essental role n the approxmaton, and conjectured that the boundedness of the sgmodal functon s a necessary and suffcent condton for the valdty of the approxmaton theorem. However, we fnd that the conjecture s not correct, that s, the boundedness condton s not suffcent or necessary n C(R n ). Instead, boundedness and unequal lmts at nfntes condtons on the actvaton functons are suffcent, but not necessary n C(R n ). Index Terms Actvaton functons, approxmaton capablty, boundedness, feedforward networs. I. INTRODUCTION In the above paper 1 and n the last several years many researchers (e.g., [1] [9]) have explored the capabltes of standard multlayer feedforward neural networs to approxmate general mappngs from one-dmensonal space to another. Two aspects of the approxmaton problem have been dscussed: unform approxmaton everywhere n C(R n ) 2 [7] and L p approxmaton almost everywhere n compacts of R n [2], [5], [6]. In the above paper Chen et al. clamed that generalzed sgmodal functons 3 can be used as actvaton functons for a sngle hdden layer neural networ to approxmate any arbtrary mappngs n C(R n ). Further, they conjectured that the boundedness of the sgmodal functon s a necessary and suffcent condton for the valdty of the approxmaton theorem (Theorem 1, p. 25 n the above paper). However, we fnd that the conjecture s not correct, that s, the boundedness condton s not suffcent or necessary n C(R n ). Instead, the boundedness and unequal lmts at nfntes condtons on the actvaton functons are suffcent, but not necessary n C(R n ). II. ON SUFFICIENT CONDITIONS Chen et al. proved the followng mportant approxmaton theorem (Theorem 1, p. 25 n the above paper). Lemma 2.1: If s a bounded generalzed sgmodal functon, and f (x) s a contnuous functon n R, for whch lm x!01 f (x) =A and lm x!+1 f (x) =B, where A; B are constants, then for any >0, there exst ; ;w ;b such that j (wx + b) 0 f (x)j < holds for all x 2 R. Chen et al. state that we conjecture the boundedness of (x) s a necessary and suffcent condton for the valdty of (Theorem 1, p. 26). In ths paper, we show that the boundedness of (x) s nether Manuscrpt receved May 13, 1997; revsed March 22, The authors are wth the School of Electrcal and Electronc Engneerng, Nanyang Technologcal Unversty, Sngapore Publsher Item Identfer S (98)04541-X. 1 T. Chen, H. Chen, and R.-W. Lu, IEEE Trans. Neural Networs, vol. 6, pp , Jan C(R n )=ff 2 C(R n ): lmjxj!1 f (x)exstsg. 3 (x) s called a generalzed sgmodal functon f lm x!01 (x) =0 and lm x!+1 (x) =1. a necessary nor a suffcent condton. Instead, the boundedness and unequal lmts at nfntes condtons on the actvaton functons are suffcent, but not necessary. In order to prove our vewpont we only need to extend ther approxmaton theorem (Theorem 1, p. 25) to more general ones. Consder a lnear subspace of C(R), L C(R), where L = f 2 C(R): lm f (x) = lm f (x) : (1) x!01 x!+1 For g 2 C(R), (w; b) 2 R 2, denote g (w; b) 2 C(R) by g (w; b) (x) = g(wx + b), and let F (g) C(R) denote the set of g (w; b), (w; b) 2 R 2. It s obvous that f g 2 L then for all (w; b), g (w; b) 2 L. Thus, we have the followng theorem. Theorem 2.1: For g 2 L, the span of F (g) L s a subset of L. Ths follows from the defnton of the span, and the fact that L s a lnear subspace of C(R). Remar: Because radal bass functons [1] have equal lmts at nfntes, accordng to Theorem 2.1, they cannot be used as actvaton functons by whch a neural networ can approxmate arbtrary contnuous mappngs wth any precson n R although they can be used when consdered n a compact set of R as shown n [1]. Theorem 2.1 shows that gven any (bounded or unbounded) functon g(x) n R, f lm x!01 g(x) = lm x!+1 g(x), then for any arbtrary (contnuous or uncontnuous) mappng f (x) n R wth lm x!01 f (x) 6= lm x!+1 f (x), and for every > 0 there does not exst w, b and 2 R and 2 N such that j g(w x+b )0f(x)j <hold for all x 2 R. Ths means that only boundedness of actvaton functons s not suffcent for standard feedforward neural networs to approxmate any arbtrary mappng n C(R n ). Instead, we fnd that the boundedness and unequal lmts at nfntes condtons on the actvaton functons are suffcent n C(R n ). Theorem 2.2: Gven a bounded functon g(x) n R and there exst lmts lm x!01 g(x), lm x!+1 g(x), and lm x!01 g(x) 6= lm x!+1 g(x), then for any arbtrary mappng f (x) n C(R), for every >0 there exst w, b and, and 2 N such that g(w x + b ) 0 f (x) < holds for all x 2 R. Proof: We can prove the theorem n two steps: 1) lm x!01 g(x) = 0 and lm x!+1 g(x) = M, where M can be any arbtrary nonzero real value and 2) lm x!01 g(x) =N and lm x!+1 g(x) =M, where M and N can be any arbtrary unequal real values. 1) Consder lm x!01 g(x) = 0 and lm x!+1 g(x) = M, where M can be any arbtrary nonzero real value. Let g 1 (x) = g(x)=m. Then lm x!01 g 1(x) =0and lm x!+1 g 1(x) = 1. Accordng to Lemma 2.1, for every >0, there exst 2 N and ;w 0 ;b ; = 1; 111; such that j 0 g 1(w x +

2 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 9, NO. 4, JULY b )0f (x)j <. Let = 0 =M; =1; 111;. Then we have g(w x + b ) 0 f (x) < holds for all x 2 R. 2) Consder lm x!01 g(x) =N and lm x!+1 g(x) =M, where M and N can be any unequal real values. Set g 1 (x) = g(x) 0 N. Thus lm x!01 g(x) =0and lm x!+1 g(x) = M 0 N. Accordng to the result obtaned n the prevous step, for every >0, there exst N and ;w ;b ;= 1; 111;0 1 such that 01 g 1 (w x + b ) 0 f (x) < (2) holds for all x 2 R. Suppose x 0 2 R and g(x 0 ) 6= 0. Then we can choose w =0and b = x 0 such that g(w x + b )= g(x 0 ). Let = 0N 01 =g(x 0 ). Hence we have = = g(w x + b ) 0 f (x) g(w x + b ) 0 N 01 =g(x 0 ) 1 g(x 0 ) 0 f (x) g 1 (w x + b ) 0 f (x) <: (3) Ths completes the proof. Lemma 2.2: If g(x) n R satsfes that all lnear combnatons g(w x +b ) are dense n C(R), then all lnear combnatons g(w 1 x + b) are dense n C(Rn ), where w 2 R n and w 1 x s the nner product of w and x. Usng Theorem 2.2 and Lemma 2.2, the followng theorem can be obtaned. Theorem 2.3: Gven any bounded functon g(x) n R whch has lmts lm x!01 g(x), lm x!+1 g(x), and lm x!01 g(x) 6= lm x!+1 g(x), then all lnear combnatons g(w 1 x + b) are dense n C(R n ), where w 2 R n and w 1 x s the nner product of w and x. holds for all x 2 R. In other words, although (x) s unbounded, for every mappng f (x) 2 C(R), for every >0 there exst w, b, and and 2 N such that holds for all x 2 R. [(w x + b ) 0 (0w x 0 b )] 0 f (x) < ACKNOWLEDGMENT The authors would le to than the anonymous revewers for ther valuable suggestons, especally for the proof structure of Theorem 2.1. REFERENCES [1] F. Gros and T. Poggo, Networs and the best approxmaton property, Artfcal Intell. Lab., Massachusetts Inst. Technol., A.I. Memo 1164, [2] K. Horn, M. Stnchcombe, and H. Whte, Multlayer feedforward networs are unversal approxmators, Neural Networs, vol. 2, pp , [3] G. Cybeno, Approxmaton by superpostons of a sgmodal functon, Math. Contr., Sgnals, Syst., vol. 2, no. 4, pp , [4] K. Funahash, On the approxmate realzaton of contnuous mappngs by neural networs, Neural Networs, vol. 2, pp , [5] K. Horn, Approxmaton capabltes of multlayer feedforward networs, Neural Networs, vol. 4, pp , [6] M. Leshno, V. Y. Ln, A. Pnus, and S. Schocen, Multlayer feedforward networs wth a nonpolynomal actvaton functon can approxmate any functon, Neural Networs, vol. 6, pp , [7] Y. Ito, Approxmaton of contnuous functons on R d by lnear combnatons of shfted rotatons of a sgmod functon wth and wthout scalng, Neural Networs, vol. 5, pp , [8], Approxmaton capablty of layered neural networs wth sgmod unts on two layers, Neural Comput., vol. 6, pp , [9] K. Horn, M. Stnchcombe, and H. Whte, Unversal approxmaton of an unnown mappng and ts dervatves usng multlayer feedforward networs, Neural Networs, vol. 3, pp , III. ON NECESSARY CONDITIONS Chen et al. conjectured that the boundedness of was a necessary and suffcent condton for the valdty of Lemma 2.1. From Theorem 2.2 we can easly fnd that the conjecture s not correct. In fact, we can defne (x) = e 0x + 1 ; f x<0 x2 0; f x =0 (4) 1+e 0x + 1 x ; f x>0. 2 Then lm x!0 (x) = +1 and lm x!0 (x) = +1, (x) s unbounded n R. It s obvous that 1(x) = (x) 0 (0x) s bounded and has lmts lm x!01 1 (x), lm x!+1 1 (x), and lm x!01 1 (x) 6= lm x!+1 1 (x). Accordng to Theorem 2.2 for any arbtrary mappng f (x) 2 C(R), for every >0 there exst w, b, and and 2 N such that j 1 (w x+b )0f (x)j <

3 716 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 9, NO. 4, JULY 1998 On Chaotc Smulated Annealng Lpo Wang and Kate Smth Abstract Chen and Ahara recently proposed a chaotc smulated annealng approach to solvng optmzaton problems. By addng a negatve self-couplng to a networ model proposed earler by Ahara et al. and gradually removng ths negatve self-couplng, they used the transent chaos for searchng and self-organzng, thereby achevng remarable mprovement over other neural-networ approaches to optmzaton problems wth or wthout smulated annealng. In ths paper we suggest a new approach to chaotc smulated annealng wth guaranteed convergence and mnmzaton of the energy functon by gradually reducng the tme step n the Euler approxmaton of the dfferental equatons that descrbe the contnuous Hopfeld neural networ. Ths approach elmnates the need to carefully select other system parameters. We also generalze the convergence theorems of Chen and Ahara to arbtrarly ncreasng neuronal nput output functons and to less restrctve and yet more compact forms. Index Terms Annealng, chaos, energy functon, Hopfeld, neural networ, optmzaton. I. INTRODUCTION Combnatoral optmzaton problems are ever present n scence and technology. Snce Hopfeld and Tan s semnal wor [6] on solvng the travellng salesman problem wth a Hopfeld neural networ (HNN) [5], the HNN s [4], [5] have been recognzed as powerful tools for optmzaton (e.g., [10], [13]). The HNN s have an ntrgung property that as each neuron n an HNN updates, an energy functon s monotonously reduced untl the networ stablzes. One can therefore map an optmzaton problem to a HNN such that the cost functon of the problem corresponds to the energy functon of the HNN and the fnal state of the HNN thus suggests a soluton to the optmzaton problem wth a low cost value. Whle some researchers have descrbed HNN s as nothng more than nave gradent descent machnes, the neural framewor does brngs about some mportant advantages over other gradent descent technques: prncpally the nherent parallelsm and hardware mplementaton whch can potentally result n great speed-ups over conventonal technques for combnatoral optmzaton [12]. The employment of HNN s to solve problems of real-world sgnfcance has been hampered, however, by problems over the last decade wth soluton qualty and slow development of sutable hardware to enable large szed problems to be solved. Earler attempts at solvng varous optmzaton problems wth the HNN s suffered from the fact that a HNN can often be trapped at a local mnmum n the complex energy terran, whch gves an optmzaton soluton wth an unacceptably hgh cost [15]. Several methods whch allow for temporary energy ncreases, such as smulated annealng [7], have been proposed. Recent advances have now made modfed HNN s compettve wth the best heurstcs for solvng combnatoral optmzaton problems, and ths has been demonstrated on a varety of real-world problems [10], [11]. The search stll contnues however, for further or alternatve mprovements to the standard Manuscrpt receved November 9, Ths wor was supported n part by the Australan Research Councl and Dean Unversty. L. Wang s wth the School of Computng and Mathematcs, Dean Unversty, Clayton, Vctora 3168, Australa. K. Smth s wth the Department of Busness Systems, Monash Unversty, Clayton, Vctora 3168, Australa. Publsher Item Identfer S (98)04613-X. neural algorthms to address the ssue of soluton qualty: partcularly mprovements whch are easly mplementable n hardware. Chen and Ahara [1] recently proposed a chaotc smulated annealng approach. By addng a negatve self-couplng to a transently chaotc neural networ (TCNN) and gradually removng ths negatve self-couplng, they used the transent chaos generated by the fadng negatve self-couplng for searchng and self-organzng, thereby achevng remarable mprovement over other neural-networ methods, n terms of frequency of fndng near-optmal solutons. However, a number of networ parameters must be carefully chosen so as to guarantee the convergence of the TCNN and ts mnmzaton of the energy upon the removal of the transent chaos [2]. In addton, Chen and Ahara [2] used n provng ther convergence theorems a partcular sgmodal functon for all neurons. Hardware mplementatons may not easly ensure ths form of nput output (I/O) functon and may also need to allow for some varatons n I/O among the neurons. In ths letter, we frst suggest an alternatve approach to chaotc smulated annealng wth guaranteed convergence and mnmzaton of the energy functon, but wthout the need for choosng any other system parameters. We then generalze the convergence theorems to arbtrarly ncreasng I/O and to less restrctve and yet more compact forms. II. AN ALTERNATIVE APPROACH TO CHAOTIC SIMULATED ANNEALING The dynamcs of a n-neuron contnuous Hopfeld neural networ (CHNN) [5] are descrbed by du dt = 0 u + j T j V j + I (1) where = 1; 2; 111 n; I s the external nput to neuron and s sometmes called frng threshold when replaced wth 0I. The nternal state of neuron ; u 2 (01; +1), determnes the output of neuron V (t) =f [u (t)] 2 [0; 1] (2) or u (t) =f 01 [V (t)]. Here f s the neuronal I/O functon for neuron. f may dffer from neuron to neuron and does not need to have any symmetry propertes; however, we assume that f s monotonously ncreasng so that f 01 exsts. For example, the sgmod functon f (x) = 1 [1 + tanh(x)] (3) 2 s often used, wth beng the gan of the I/O functon. Hopfeld ntroduced the followng functon [5] for the CHNN: E(t) = ; j Tj V(t) Vj (t) 0 IV(t) + 1 V (t) f 01 () d: (4) 0 = 0 j T j V j 0 I + f 01 (V )= = 0du =dt, provded that T j = T j (regardless the sgn of the self-couplng T ), then de=dt = (@E=@V )(dv =dt) =0(du =dt)1 f 0 1 (du =dt) < 0, f du =dt 6= 0 for at least one, and de=dt = 0 f and only f du =dt = 0 for all. Hence E s a Lyapunov (energy) functon that monotonously decreases as the networ updates untl the networ stablzes. Based on the exstence and the fnteness of such a Lyapunov functon, Hopfeld concluded [5] that the CHNN must stablze tself: ~u(t)! ~u o,ast! +1. Here ~u o s ndependent of tme t and represents the stable fxed pont of the networ.

4 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 9, NO. 4, JULY Now let us consder a dfferent networ whch conssts of n neurons wth the same I/O functons f, but obeys the followng dynamc equatons: u (t +1t) = 1 0 1t u (t)+1t T j V j (t)+i : (5) j We note that n the lmt 1t! 0, ths equaton s the same as the CHNN (1). In fact, (5) s the Euler approxmaton of the CHNN (1). The Euler approxmaton of the CHNN (5) s dentcal to the TCNN of Chen and Ahara [1], [2], [9]: f 1t = 1 and u (t +1)=u (t)+ j T 0 j V j (t)+i 0 (6) =10 1t ; T0 j =1tT j ; I 0 =1tI : (7) By addng a large negatve self-couplng T 0 and gradually removng t n the TCNN (6), Chen and Ahara [1] maredly enhanced the probablty of reachng an optmal or near-optmal soluton. In ths approach, other system parameters, such as and [the gan n the sgmodal functon equaton (3)], must be carefully selected wth respect to the synaptc weght matrx accordng to ther stablty theorems [2], n order to assure networ convergence and energy mnmzaton. We suggest an alternatve approach to chaotc smulated annealng: startng from the Euler approxmaton of the CHNN (5) wth a large tme-step 1t, where the dynamcs are chaotc [2], we gradually reduce the tme-step 1t. The system s guaranteed to converge and to mnmze the CHNN energy functon (4), snce n the lmt of 1t! 0, the system approaches the CHNN whch s stable and mnmzes the CHNN energy functon. Ths approach does not requre dffcult choces of any system parameters to assure networ convergence and energy converson. III. GENERALIZED STABILITY THEOREMS Chen and Ahara [2] derved stablty theorems for both the Euler approxmaton of the CHNN (5) and the TCNN (6) for both synchronous and asychronous updatng, usng a partcular form of the neuronal I/O (3). We now generalze ther theorems to arbtrarly ncreasng I/O functons (2) and to less restrctve but more compact forms. Let us calculate the change n energy between two tme steps when the TCNN (6) s updated synchronously, accordng to (4) 1E(t) E(t +1)0E(t) = ; j 1t 1t T j 0 1V (t)1v j (t) + 1V (t) f 01 [V (t)] 0 1V (t) f 01 [V (t + 1)] +(10 ) fg [V (t + 1)] 0 G [V (t)]g: (8) where G (V ) V f 01 0 () d and 1V (t) V (t +1)0V (t). Expandng G at V (t +1)and usng the fact that G s concave-up [8], we obtan G [V (t + 1)] 0 G [V (t)] G 0 [V (t + 1)] 1V (t) mn d G [1V d V (t)] 2 (9) where mnfd 2 G =d 2 V g = mnf1=fg1= 0 max s the mnmum curvature of G ; max beng the maxmum slope of the I/O functons. Furthmore, f 0 0ff 01 [V (t + 1)] 0 f 01 [V (t)]g 1V (t) 0 [1V (t)] 2 : max (10) Combnng (8) (10), n the case where 1 0, we obtan 1E(t) 1t ; j T 0 j + (1+) j 1V (t)1v j (t) (11) where j =0f 6= j and j =1f = j. Hence 1E(t) 0, or the networ s stable, f matrx ft 0 j +[(1+)= max ] j g s postvedefnte. Therefore a suffcent stablty condton for a synchronous TCNN (6) s (1 + ) 1 0; and < 0Tmn 0 (12) max where Tmn 0 s the mnmum egenvalue of matrx Tj. 0 For >1, we expand G at V (t) and obtan, nstead of (9) G [V (t + 1)] 0 G [V (t)] G 0 [V (t)]1v (t) mn d 2 G d 2 [1V (t)] 2 : (13) V We thus have an alternatve stablty condton for the TCNN n synchronous mode, smlar to (12) 2 >1; and > 0Tmn: 0 (14) max The stablty condton for the Euler approxmaton of the CHNN (5) can be derved easly from (7) and (12), wth the usual assumpton that > 0 2 1t ; 1t < ; and T mn 0: (15) 1 0 maxt mn Both the TCNN and the Euler approxmaton of the CHNN are stable f Tmn 0 = 1tT mn > 0, snce the matrx n (11) s automatcally postve-defnte. It s straghtforward to derve that the stablty condtons for both the TCNN and the Euler approxmaton of the CHNN for asynchronous updatng are obtaned from the condtons for synchronous updatng (12), (14), and (15) wth Tmn 0 (or T mn) replaced by mnftg 0 (or T ). Our stablty condtons consst of two parameter regons for the TCNN (12) and (14) and one parameter regon for the Euler approxmaton of the CHNN (15), and are therefore more compact compared to those of Chen and Ahara [2] (three regons for each type of networ). Furthermore, our condtons are less restrctve on the parameters nvolved. For nstance, n the one-neuron TCNN consdered by Chen and Ahara [2], the I/O functon s gven by (3) wth a gan =125. Hence max = =2 =62:5. Wth =0:9, Chen and Ahara s [2] theorems ndcate that the networ s stable f 0T11 0 < 0:0288, whereas our condton (12) gves 0T11 0 < 0:0304 and s closer to the bfurcaton pont 0T11 0 =0:0331. IV. CONCLUSIONS In summary, we have proposed an alternatve approach to chaotc smulated annealng, n whch the tme-step 1t n the Euler approxmaton of the CHNN starts from a large value, where chaos exsts, and reduces to a small value so that the networ stablzes. Ths approach guarantees convergence and mnmzaton of the energy functon, and elmnates the need of choosng other system parameters. It should prove to be a powerful tool for effcently obtan optmal or near-optmal solutons to a varety of optmzaton problems, whch s currently under nvestgaton. We have also generalzed the Chen Ahara convergence theorems for the transently chaotc neural networ and the Euler approxmaton of the CHNN to arbtrarly ncreasng neuronal I/O functons and to less restrctve but more compact forms. Ths should be useful for both hardware mplementatons and software smulatons, as well as further theoretc analyss of the systems.

5 718 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 9, NO. 4, JULY 1998 ACKNOWLEDGMENT Many helpful comments and suggestons from the revewers are gratefully apprecated. REFERENCES [1] L. Chen and K. Ahara, Chaotc smulated annealng by a neuralnetwor model wth transent chaos, Neural Networs, vol. 8, no. 6, pp , [2], Chaos and asymptotcal stablty n dscrete-tme neural networs, Physca, vol. 104, pp , June, [3] F. Fogelman-Soulé, C. Meja, E. Goles, and S. Martnez, Energy functon n neural networs wth contnuous local functons, Complex Syst., vol. 3, pp , [4] J. J. Hopfeld, Neural networs and physcal systems wth emergent collectve computatonal abltes, n Proc. Nat. Academy Sc. USA, Apr. 1982, vol. 79, pp [5], Neurons wth graded response have collectve computatonal propertes le those of two-state neurons, n Proc. Nat. Academy Sc. USA, May 1984, vol. 81, pp [6] J. J. Hopfeld and D. W. Tan, Neural computaton of decsons n optmzaton problems, Bol. Cybern., vol. 52, pp , [7] S. Krpatrc, C. D. Gelatt, Jr., and M. P. Vecch, Optmzaton by smulated annealng, Scence, vol. 220, pp , [8] C. M. Marcus and R. M. Westervelt, Dynamcs of terated-map neural networs, Phys. Rev. A, vol. 40, pp , July [9] H. Nozawa, A neural-networ model as a globally coupled map and applcatons based on chaos, Chaos, vol. 2, no. 3, p , [10] K. Smth and M. Palanswam, Statc and dynamc channel assgnment usng neural networs, IEEE J. Select. Areas Commun., vol. 15, no. 2, pp , [11] K. Smth, M. Palanswam, and M. Krshnamoorthy, Neural technques for combnatoral optmzaton wth applcatons, IEEE Trans. Neural Networs, accepted. [12] M. Verleysen and P. Jespers, An analog VLSI mplementaton of Hopfeld s neural networ, IEEE Mcro, pp , Dec [13] L. Wang, Dscrete-tme convergence theory and updatng rules for neural networs wth energy functons, IEEE Trans. Neural Networs, vol. 8, pp , Mar [14], On the dynamcs of dscrete-tme, contnuous-state Hopfeld neural networs, IEEE Trans. Crcuts Syst. II, to be publshed. [15] G. V. Wlson and G. S. Pawley, On the stablty of the travelng salesman problem algorthm of Hopfeld and Tan, Bol. Cybern., vol. 58, pp. 63, Comments on The Effects of Quantzaton on Multlayer Neural Networs Oh-Jun Kwon and Sung-Yang Bang In ths letter we pont out and correct the errors n the above paper 1 n the dervatons of the followng equatons: 1) n the left column of p y = (10 1 2N ) 2 1 K1 (1) 144 Manuscrpt receved November 9, The authors are wth the Department of Computer Scence and Engneerng, Pohang Unversty of Scence and Technology, Pohang, , Korea. Publsher Item Identfer S (98)04613-X. 1 G. Dündar and K. Rose, IEEE Trans. Neural Networs, vol. 6, pp , Nov ) n the rght column of p y = (11 1 2N ) 2 A A 2 0 tanh A 2 : (2) We thn the dervatons should be as follows. a) Snce the products of the weght and the nput are ndependent, we can rewrte the varance of y 0 as follows: 2 K 01 y =var wx 0 0 =0 K 01 = var[wx 0 0 ]: (3) =0 Also w 0 and x 0 are ndependent and E[w 0 ] = 0 by the assumpton of the unform dstrbuton of the weghts gven by [1]. And, by the assumpton that the nputs are unformly dstrbuted between zero and one, the expectaton of an nput s Therefore E[x 0 ]= 1 2 : (4) var[wx 0 0 ]=E[(wx 0 0 ) 2 ] 0 E 2 [wx 0 0 ] = E[(wx 0 0 ) 2 ] = E[(w) 0 2 ] 1 E[(x 0 ) 2 ] = var[w] 0 1 (var[x]+e 0 2 [x 0 ]) = (10 1 2N ) = (10 1 2N ) 2 : (5) 36 Fnally, substtutng (5) nto (3), we obtan 2 y = (10 1 2N ) 2 1 K1: (6) 36 b) The output of the hdden neurons s between zero and one snce the followng sgmodal functon s used n neurons: f (u) = (1 + e 0u ) 01 : (7) By the assumpton gven by [1] that the outputs of the hdden neurons are unformly dstrbuted, the expectaton of the output of a hdden neuron s E[x 1 l ]= 1 2 : (8) Therefore, as n (a), we can derve the varance of y 1 as follows: 2 K 01 y =var wlx 1 1 l l=0 K 01 = var[wlx 0 1 l ] l=0 K 01 = var[w l ] 1 (var[x 1 l ]+E 2 [x 1 l ]) l=0 = K2 ( N ) A A 2 0 tanh A = (11 1 2N ) A A 0 tanh A 2 : (9)

6 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 9, NO. 4, JULY The new result of a) ndcate that 2 y s four tmes that ndcated by [1] and hence A wll be twce as large. Therefore, accordng to the new results of a) and b), 2 y wll approxmately four tmes as large as that ndcated by [1]. larger for the same reasons. Thus, var(1y 0 )=K =36: (We have used the smpler var() notaton rather than 2 to correct typographcal errors n our manuscrpt caused by confuson of superscrpts.) Thus, the later equatons should be Authors Reply G. Dündar and K. Rose and var(w 1 ) =( N ) 2 =12 var(1 1)=1 2 1=12: We would le to than Kwon and Bang for ther correctons to the equatons n Secton II of our paper. We agree wth ther correctons. They fnd that var(y 0 )=K 1(1 N 0 ) 2 =36 whch s four tmes larger than our expresson. In addton, the expresson whch follows ths n our paper should also be four tmes Manuscrpt receved March 22, G. Dündar was wth the Department of Electrcal and Computer Scence Engneerng, Rensselaer Polytechnc Insttute, Troy, NY USA. He s now wth the Department of Electrcal and Electronc Engneerng, Boğazç Unversty, Bebe 80815, Istanbul, Turey. K. Rose s wth the Department of Electrcal and Computer Scence Engneerng, Rensselaer Polytechnc Insttute, Troy, NY USA. Publsher Item Identfer S (98) Kwon and Bang correct our expresson for var(y 1 ) and we agree wth ther results. However, these changes wll also affect the fnal equaton n Secton II of our paper whch should be var(1y 1 )=K 2 [(( N ) 2 =12)var(1x 1 )+E[(x 1 ) 2 jvar(1 1 )]: The error s relatvely small snce we are replacng var(x 1 ) n the orgnal equaton by E[(x 1 ) 2 ]=var(x 1 )+1=4: Snce smlar errors were made n the dervatons of var(y 0 ) and var(1y 0 ), etc., these have lttle effect on the results of Fg. 3 where we dvde the rms error for the output by the rms value for the output sne wave. Thus these errors tend to cancel and our conclusons are not affected.