An Accurate Measure for Multilayer Perceptron Tolerance to Weight Deviations

Transcription

1 Neural Processng Letters 10: , Kluwer Acadec Publshers. Prnted n the Netherlands. 121 An Accurate Measure for Multlayer Perceptron Tolerance to Weght Devatons JOSE L. BERNIER, J. ORTEGA, M. M. RODRÍGUEZ, I. ROJAS and A. PRIETO Dpto. Arqutectura y Tecnología de Coputadores, Unversdad de Granada, E18071 Granada, Span, e-al: berner@atc.ugr.es, apreto@atc.ugr.es Abstract. The nherent fault tolerance of artfcal neural networks ANNs) s usually assued, but several authors have claed that ANNs are not always fault tolerant and have deonstrated the need to evaluate ther robustness by quanttatve easures. For ths purpose, varous alternatves have been proposed. In ths paper we show the drect relaton between the ean square error MSE) and the statstcal senstvty to weght devatons, defnng a easure of tolerance based on statstcal senttvty that we have called Mean Square Senstvty MSS); ths allows us to predct accurately the degradaton of the MSE when the weght values change and so consttutes a useful paraeter for choosng between dfferent confguratons of MLPs. The experental results obtaned for dfferent MLPs are shown and deonstrate the valdty of our odel. Key words: ean square error degradaton, ultlayer perceptron, fault tolerance, statstcal senstvty 1. Introducton Based on the analogy between Artfcal Neural Networks ANNs) and natural ones, t s frequently assued that ANNs are nherently tolerant to faults. Ths assupton has been shown to be false [1, 2]; consequently, soe confguratons of weghts for a fxed structure of ANN are ore tolerant than others, although they provde slar perforance wth respect to learnng ablty. Thus t would be useful to have an accurate eans of easurng ths tolerance. Dfferent easureents have been proposed. In [3] the probablty of errors n Multlayer Perceptrons MLPs) that use the sple-step functon as actvaton s used to study the tolerance of such structures, and n [4] the study s extended to neurons wth a ultple-step actvaton functon. The authors found that the probablty of error s not affected by an ncrease n the nuber of neurons per layer. In [1] a sulaton-derved quanttatve easure akng use of the worst case hypothess to consder the nuber of neurons that present a stuck-at fault s used, whle n [2] a procedure based on the replcaton of hdden unts s proposed

2 122 JOSE L. BERNIER ET AL. to acheve fault-tolerant ANNs, provdng etrcs for tolerance as a functon of redundancy. The authors of [5] show that the nserton of synaptc nose durng the learnng process ncreases the fault tolerance of ANNs. They also showed that enlargng the networks does not prove fault tolerance at all, but on the contrary akes the ore susceptble to faults. Hence, t s clear that the supposed fault tolerance of MLPs s relatve, and that the degree of fault tolerance can and should be easured as s the learnng perforance. Ths becoes sgnfcant when the learnng process s perfored n eda dfferent fro where the MLP s pleented, as there ay be dfferences n the accuracy used to store the values of the weghts or other agntudes. These dfferences ay serously degrade the perforance of the MLP. A dscusson of hardware error sources s presented n [5]. Cho et al. [7] proposed statstcal senstvty as a easure of tolerance n MLPs and thus as a crteron for selectng a confguraton of weghts fro dfferent alternatves presentng slar perforance wth respect to learnng. Statstcal senstvty easures the output changes of the MLP when the values of the weghts change, a low value plyng less degradaton of the learnng perforance. Ths fact s plcty proved wth the results obtaned. In ths letter, an explct relaton between the statstcal senstvty to weght devatons and the ean square error MSE) between the desred and obtaned output n the MLP s shown, such that t s possble to accurately predct the degradaton of the MSE for a partcular value of ths devaton. A new fgure that we call Mean Square Senstvty MSS) s proposed as a easureent of the fault tolerance to weght devatons. The MSS can be easly coputed after tranng and s shown to consttute a good easureent of the tolerance of the network. The statstcal senstvty can be coputed locally for each neuron and so, dfferent alternatves arse for future work and resarch possbltes; for exaple, the study of tolerance to weght perturbatons of partcular eleents n the network or the odfcaton of the tranng algorth n order to reduce the MSS. Other easures proposed affect the whole network [1, 2, 3, 4] and so ther use to study tolerance s ore lted. Furtherore, as the MSS s drectly related to the MSE degradaton, an accurate and quanttatve easure of MLP tolerance s obtaned, wthout akng use of any hypotheses. 2. The Statstcal Senstvty of a Multlayer Perceptron A ultlayer perceptron MLP) s coposed of M layers, where each layer = 1...M)has N neurons. The neuron of layer s connected to the N neurons of the prevous layer by a set of weghts w = 1,...,N ).

3 AN ACCURATE MEASURE FOR MULTILAYER PERCEPTRON TOLERANCE 123 The output y of a neuron belongng to a layer, s a functon f actvaton functon) of the weghted su of the outputs cong fro the neurons n the prevous layer z ): N y = f z ) = f 1) =1 w y where y consttutes the outputs of the N neurons of the prevous layer. Specfcally, y 0 = 1,...,N 0) are the nputs to the network. Durng learnng, the weghts are adapted n order to nze the MSE by usng a gradent descent algorth known as backpropagaton. Dependng on the ntal values of the weghts and other paraeters of the algorth, dfferent values are reached after tranng. These posble solutons ay present a slar MSE, but they dffer wth respect to the fault tolerance obtaned. Fault tolerance s related to a unfor dstrbuton of learnng, such that the salency of the weghts s regular [6]. The values of the weghts n an electronc pleentaton can be changed f a crcut presents a defect. Moreover, the backpropagaton algorth s often executed n a general purpose coputer and the weghts obtaned are physcally pleented; t s thus possble for dfferences n the loaded values to occur. The statstcal senstvty S allows us to easure the degradaton of the expected output of a neuron n layer n a quanttatve way when the values of the weghts change. The statstcal senstvty s defned n [7] by the followng expresson: var y S = l ) 2) σ 0 σ where σ represents the standard devaton of the changes n the weghts, and var y ) s the varance of the devaton n the output wth respect to the output of the MLP n the absence of perturbatons) due to these changes and can be coputed as: var y ) = E[ y ) 2 ] E[ y ]) 2 3) where E[ ] s the expected value of [ ]. Statstcal senstvty s fundaentally dfferent fro output senstvty as used n [6]. The output senstvty s defned as the dervatve of the output of the neuron wth respect to the value of a weght and so t easures the dependence of the output wth respect to ths partcular value of the weght. The statstcal senstvty easures the varaton range of the output of a neuron due to changes n the weghts. To copute expresson 2), a ultplcatve odel of weght devatons s assued that satsfes: a) E[ w ]=0 b) E[ w )2 ]=σw )2 c) E[ w w )]=0f = or = or =

4 124 JOSE L. BERNIER ET AL. If the devatons of the weghts of a neuron n layer are sall enough, the correspondng devaton n the output can be approxated as: where y f = =1 = f df dz y w =1 w + y y y ) y w + w y ) 4) Proposton 1: f E[ w ]=0,, then E[ y ]=0,. Proof 1: t wll be shown by nducton over.for = 1: E [ ] N 0 ) N 0 y 1 = E f 1 y 0 w 1 = f 1 y 0 E [ ] w 1 = 0 5) =1 =1 Assung that E[ y ]=0, for layer : [ ] = E E [ y = f = 0 f =1 =1 y y w + w y [ ] [ E w + w E ) ] y ]) 6) Proposton 2: the statstcal senstvty to ultplcatve perturbatons of a neuron n layer can be expressed as: S = f N ) N 2 y w + w wk C k 7) Ck = =1 k=1 where the ters Ck can be recursvely coputed as: ) 2 f ) 2 wr y r + w r f f k r=1 w r r=1 s=1 s=1 w s C rs ) f = k w ks C rs otherwse 8)

5 AN ACCURATE MEASURE FOR MULTILAYER PERCEPTRON TOLERANCE 125 Proof 2: akng use of Proposton 1, var y ) = E[ y ) 2 ]. The ters E[ y yk ] >0 can be obtaned as: [ N E[ y y k ] = E = r=1 r=1 N fk s=1 N N f y r y s +w r w ks E [ y r +y s = f f k w r + w r y r ) wks + ) ] w ks y s [ ] f f k yr ys E wr w ks + s=1 ] [ ys + y r wks E wr E [ ] ) wks y r [ ] yr ys E wr w ks + r=1 s=1 ] ) w r w ks E [ y r y s wr y s If Ck s defned as C k E[ y y k ])/σ2 and the odel for perturbatons consdered s appled n 9), t s straghtforward to obtan Equaton 8). The ntal condton for 8) s that Ck 0 = 0,k as the nputs y0 are supposed to be free of errors. At ths pont, takng nto account that E[ y ) 2 ] = σ 2 C and substtutng n 2) Proposton 2 s proved. In the partcular case of MLPs wth only one hdden layer, expresson 7) can be coputed n a ore copact for: S = f N w )2 =1 y ) ) 2 + S ) 2 ] + 9) = 1, 2 10) takng nto account that the statstcal senstvty of the nputs s zero,.e., S 0 = 0. In ths way, as the above expresson 10) shows, the coputaton of the statstcal senstvty can be perfored n a relatvely easy way for each neuron of the MLP and for each nput pattern n the case of MLPs wth one hdden layer, because there are no cross ters as n expresson 7).

6 126 JOSE L. BERNIER ET AL. 3. The Mean Square Senstvty The goal of the backpropagaton algorth s to reduce the ean square error MSE) whch can be expressed as: ε = 1 2N p N p N M p=1 =1 d p) y M p) ) 2 11) where N p s the nuber of nput patterns consdered, N M s the nuber of neurons n the output layer, and d p) and y M p) are the desred and obtaned outputs of the neuron of the output layer for the nput pattern p, respectvely. If the weghts of the MLP suffer any devaton, the MSE s altered and so, by developng expresson 11) wth a Taylor expanson near the nonal MSE found after learnng, ε 0, t s obtaned that: ε = ε N p + 1 2N p N p N M p=1 =1 N p M p=1 =1 d p) y M p) ) y M p) y M p) ) ) Now, f we copute the expected value of ε and take nto account that E[ y M ]= 0andthatE[ y M ) 2 ] can be obtaned fro expressons 2) and 3) as E[ y M ) 2 ] σ 2 S M ) 2, the followng expresson s obtaned: E[ε ]=ε 0 + σ2 2N p N p N M p=1 =1 S M ) 2 13) By analogy wth the defnton of MSE, we defne the followng fgure as Mean Square Senstvty MSS): MSS = 1 2N p N p N M p=1 =1 S M ) 2 14) The MSS can be coputed fro the statstcal senstvty of the neurons belongng to the output layer, as expresson 14) shows. In ths way, cobnng expressons 13) and 14), the expected degradaton of the MSE, E[ε ] can be coputed as: E[ε ]=ε 0 + σ 2 MSS 15) Thus, expresson 15) shows the drect relaton between the MSE degradaton and the MSS. As the MSS can be drectly coputed after tranng t s possble to predct the degradaton of the MSE when the weghts are devated fro ther

7 AN ACCURATE MEASURE FOR MULTILAYER PERCEPTRON TOLERANCE 127 Table I. MSE and MSS obtaned after tranng. Approxator Predctor ε MSS nonal values nto a range wth standard devaton equal to σ. Moreover, as can be observed n the expresson obtaned, a lower value of the MSS ples a lower value of the degradaton of the MSE, so we propose usng the MSS as a sutable easure of the tolerance of MLPs to weght devatons. Note that as the statstcal senstvty of a partcular neuron can be coputed ndependently, several lnes of research are open to study the tolerance of partcular eleents or to develop new tranng algorths that take nto account the MSS as another ter to nze durng learnng. In [7] t s proposed to use the average statstcal senstvty as a crteron to select a weght confguraton fro dfferent possbltes whch present slar MSE after tranng. However, the MSS s drectly obtaned fro the MSE degradaton, as expresson 15) shows, and thus consttutes a better easureent of MLP tolerance aganst weght perturbatons. 4. Results In order to valdate expresson 15) we copared the results obtaned for the MSE when the MLPs are subect to devatons wth the predcted value obtaned by usng ths expresson. Two MLPs were consdered: an approxator of the sne functon [8] and a predctor of the Mackey-Glass teporal seres [9]. The approxator had 1 nput neuron, 11 neurons n the hdden layer and 1 output neuron, and the predctor conssted of 3 nput neurons, 11 neurons n the hdden layer and 1 output neuron. All the neurons consdered contaned a bas nput. Table I shows the values of MSE and MSS obtaned after tranng wth the test patterns dfferent fro those used for tranng). All the weghts obtaned after learnng have been devated fro ther nonal values by usng the ultplcatve odel such that each weght w has a value equal to w 1 + δ ),whereδ s a rando varable wth standard devaton equal to σ and average equal to zero. Note that n the case of consderng analogue crcuts, σ s the sae as the tolerance of the analogue eleents. Table II shows the values of the MSE predcted and obtaned experentally for dfferent values of σ analogue tolerance). For each value of σ consdered, the experental values of MSE are averaged over 100 tests where each test conssts of a rando devaton of all the weghts of the MLP. The confdence nterval at 95% s also presented n Table II.

8 128 JOSE L. BERNIER ET AL. Table II. Coparson between predcted and experental E[ɛ ]. Approxator Predctor σ %) Predcted Experental Predcted Experental x 1e-3) x 1e-3) x 1e-3) x 1e-3) ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± ± Approxator Experental Predcted 0.06 Mean Square Error MSE) Standard devaton of weght perturbatons Fgure 1. MSE predcted and experental for the approxator of the sne functon. Expresson 15) s shown to be vald; t accurately predcts the degradaton of the MSE when the weghts present perturbatons. It s also proven that the lower the value of the MSS, the lower the degradaton of the MSE. Thus, even f a partcular confguraton presents a lower MSE after tranng, f the MSS s hgh, ths nonal MSE s strongly degraded when devatons are present, and so the MSS ust be consdered when a weght confguraton s to be chosen.

9 AN ACCURATE MEASURE FOR MULTILAYER PERCEPTRON TOLERANCE Predctor Experental Predcted Mean Square Error MSE) Standard devaton of weght perturbatons Fgure 2. MSE predcted and experental for the predctor of the Mackey-Glass teporal seres. Fgures 1 and 2 show the degradaton of the MSE for dfferent values of σ.the values predcted and obtaned experentally are represented for the approxator and the predctor, respectvely. Each experental value s plotted wth ts respectve confdence level at 95% obtaned wth 100 saples. In a slar way to Table II, the predcted values of the MSE accurately ft those obtaned experentally. For exaple, n Fgure 2 the predcton s vald up to weght devatons of about 20% fro ther nonal values; n an analogue crcut ths eans that the MSE degradaton can be predcted for tolerance argns of ther eleents of about 20%. The atchng between the predcted and the experental values of MSE s better when weght devatons are saller; however, for noral values of analogue tolerance, the predcted value s accurate and, n any case consttutes an upper bound for the MSE degradaton. 5. Conclusons In ths letter we have presented the relaton between the ean square error MSE) and the statstcal senstvty. As the statstcal senstvty easures the devaton n the output of a MLP when ts weghts are perturbed, ths relaton allows us to obtan a useful crteron to evaluate the fault tolerance of the network. To copare dfferent weght confguratons, we propose the use of ean square senstvty MSS), whch s coputed fro the statstcal senstvty. Lower values of MSS ply lower degradatons of MSE. Results show the correctness of the expressons

10 130 JOSE L. BERNIER ET AL. obtaned. To dstngush MSS fro other easures proposed to assess the tolerance of MLPs, t s drectly related to MSE degradaton and also, as statstcal senstvty can be coputed for each neuron of the MLP, new research possbltes are opened for the study of related aspects. As future work, a new backpropagaton algorth that ncludes the obectve of nzng MSS, ontly wth MSE, wll be developed n order to obtan weght confguratons that axze fault tolerance whle antanng learnng perforance. As MSS s an accurate easure of MSE degradaton, the perforance of such an algorth wll probably be better than that descrbed n [10] for a slar tranng algorth based on average statstcal senstvty nzaton. References 1. Segee, B. E. and Carter, M. J.: Coparatve fault tolerance of parallel dstrbuted processng networks, IEEE Trans on Coputers 4311) 1994), Pathak, D. S. and Koren, I.: Coplete and partal fault tolerance of feedforward neural nets, IEEE Trans. on Neural Networks 62) 1995), Stevenson, M., Wnter, R. and Wdrow, B.: Senstvty of neural networks to weght errors, IEEE Trans. on Neural Networks 11) 1990), Alpp, C., Pur, V. and Sa, M.: Senstvty to errors n artfcal neural networks: a behavoral approach, In: Proc. IEEE Int. Syp. on Crcuts & Systes, pp , May Edwards, P. J. and Murray, A. F.: Fault tolerance va weght-nose n analogue VLSI pleentatons a case study wth EPSILON, IEEE Proc. on Crcuts and Systes II: Analog and Dgtal Sgnal Processng 459) 1998), Edwards, P. J. and Murray, A. F.: Can deternstc penalty ters odel the effects of synaptc weght nose on network fault-tolerance?, Int. Journal of Neural Systes 64) 1995), Cho, J. Y. and Cho, C.: Senstvty analyss of ultlayer perceptron wth dfferentable actvaton functons, IEEE Trans. on Neural Networks 31) 1992), Sudkap, T. and Haell, R.: Interpolaton, copleton and learnng fuzzy rules, IEEE Trans. on Systes, Man & Cybernetcs 242) 1994), Wang, L.: Adaptve Fuzzy Systes and Control. Desgn and Stablty Analyss, Englewood Clffs, Prentce Hall, Berner, J. L., Ortega, J. and Preto, A.: A odfed backpropagaton algorth to tolerate weght errors,lecture Notes n Coputer Scence 1240, 1997), Sprnger-Verlag,