Evaluation for Prediction Accuracies of Parallel-type Neuron Network

Transcription

1 Proceedngs of the Internatonal ultconference of Engneers and Computer Scentsts 009 Vol I, IECS 009, arch 8-0, 009, Hong Kong Evaluaton for Predcton Accuraces of Parallel-type Neuron Networ Shunsue Kobayaawa and Hroazu Yoo Abstract The parallel-type neuron networ (PNN) s researched to mprove on the decrease n capabltes of the neuron networ by the nterference of the learnng caused between the outputs of BP networ (BPN) of two outputs or more and the dffculty of the common achevement of the mddle layer used for each output. The research to compare predcton accuraces of nonlnear tme seres sgnals predcton systems usng BPN and PNN has been performed so far. However, t has not attaned demonstratng the exstence of domnance of all predcton accuraces of PNN to BPN. Then, the expermental evaluaton of the domnance of all outputs of PNN whch could exst for the theory by results of the comparson of learnng rules of BPN and PNN was performed usng nonlnear tme seres sgnals predcton systems n ths research. As a result, the domnance was showed. Index Terms BP Networ, earnng Rule, Nonlnear Tme Seres Sgnals Predcton System, Parallel-type Neuron Networ, Predcton Accuracy I. INTRODUCTION The parallel-type neuron networ (PNN) [] [5] s researched to mprove on the decrease n capabltes of the neuron networ by the nterference of the learnng caused between the outputs of BP networ (BPN) [6] of two outputs or more and the dffculty of the common achevement of the mddle layer used for each output. It s nown that causes of these problems are n the learnng rule and constructon of the neuron networ. To enable the learnng for connecton weghts n mddle layers of the perceptron [7], the learnng rule of BPN was created wth the standpont of psychology. The nconsstence s caused between the physcal events to the neuron networ and the mathematcs processng by the learnng of BPN of two outputs or more. BPN of two outputs or more has total calculatons usng calculaton results whch are gotten ndependently by each output n calculatons to change weghts and thresholds n the last mddle layer at the learnng, because t has the mddle layer used by each output n common. Ths s an orgn of cause of mutual nterference to the learnng for each output. And, ths nterference bac propagates one after another n the mddle layers. oreover, anuscrpt receved December 30, 008. S. Kobayaawa s wth Graduate School of fe Scence and Systems Engneerng, Kyushu Insttute of Technology, -4 Hbno, Waamatsu-u, Ktayushu-sh, Fuuoa , Japan (phone: ; fax: ; e-mal: obayaawa-shunsue@edu.lfe.yutech.ac.jp). H. Yoo s wth Graduate School of fe Scence and Systems Engneerng, Kyushu Insttute of Technology, -4 Hbno, Waamatsu-u, Ktayushu-sh, Fuuoa , Japan (phone: ; fax: ; e-mal: yoo@lfe.yutech.ac.jp). the ndependent learnng for each output s performed only n the output layer. The research from the standpont whch the learnng rule, the neuron and the networ are devsed has been done to mprove lmt of outputs accuraces of BPN caused from the problem concernng such a learnng capablty though BPN has approxmate realzaton capablty of an arbtrary contnuous map [8], [9]. Especally, t s mportant to construct correctly the networ whch s the prme cause for ths problem. The learnng rule of PNN s processed wth physcal events n the neuron networ only one output approprately reflected n mathematcs. Furthermore, the learnng only one output s performed n all layers. Therefore, t s theoretcally shown that accuraces to all outputs of PNN are more domnant than ones of BPN when the learnng rule of PNN s compared wth one of BPN. The research to compare the predcton accuracy has been performed about the nonlnear tme seres sgnals predcton systems usng PNN and BPN from the above-mentoned vewpont so far. However, t has not attaned demonstratng the exstence of domnance to BPN concernng all predcton accuraces of PNN. On the other hand, there are the detecton of mddle cerebral artery spasm usng learnng vector quantzaton neuron networs by. Swercz et al. [0] and the determnng coronary artery dsease and predctng leson localzaton by I. Babaoglu et al. [] n the research whch apples a parallel-type of networ n recent years. Then, the purpose of ths research s to demonstrate the exstence of domnaton to BPN concernng the predcton accuraces of all outputs of PNN. II. PARAE-TYPE NEURON NETWORK A. I/O Characterstcs Fg. shows a dscrete tme parallel-type neuron networ (DTPNN). The neuron used for a parallel-type neuron networ (PNN) can apply varous types. In ths research, a general artfcal neuron s used for PNN. The I/O characterstcs of DTPNN are shown () n the nput layer and from () to (4) n the mddle layers and the output layer. z x (,,, n ) () n ( τ) ( τ) u w x j j j,3,,,,, n ; j,,, n ;,,, n () ISBN: (revsed on November 06) IECS 009 ISSN: (Prnt); ISSN: (Onlne)

2 Proceedngs of the Internatonal ultconference of Engneers and Computer Scentsts 009 Vol I, IECS 009, arch 8-0, 009, Hong Kong x x () τ st layer Parallel unt ( -)th layer nd layer th layer Parallel unt n z z () τ ( τ + ) ( τ) ( τ) w w +Δw j j j,3,, (7),,, n ; j,,, n ;,,, n ( τ) ( τ ) E β ( ) h τ Δ h + Δ h ( τ) ( τ ) r + βδh (,3,, ;,,, n ;,,, n ) (8) x n z n n Fg. Dscrete tme parallel-type neuron networ ( τ) ( τ) s u h (,3,, ;,,, n ;,,, n ) (3) ( τ) ( τ) ( τ) tan z f s A s (,3,, ;,,, n ;,,, n ) where the upper shows a layer number, the lower shows an element number n < >, the left row shows an element of output sde, the rght row shows an element of nput sde n tow rows mar of < >, x s an nput sgnal, z s an output sgnal, w s a connecton weght, u s the nput weght sum, h s a threshold, s s the nput sum, f s an output functon, A s the output coeffcent, s the output layer, the suffx of each sgn s a parallel unt number, τ s dscrete tme. oreover, w and h are changed by the tranng. B. earnng Rule The learnng rule of DTPNN s shown by from (5) to (). The bac-propagaton for BP networ of one output s appled to ths learnng rule for a parallel unt. E y z ( n ) (4),,, (5) ( τ) ( τ ) E Δ w + β ( ) Δw τ j j w j ( τ) ( τ) ( τ ) r z + βδw j j,3,,,,, n ; j,,, n ;,,, n (6) ( τ + ) ( τ) ( τ) h h +Δ h (,3,, ;,,, n ;,,, n ) r (9) s E y z A + s,,, n (0) ( ) r j s E j n ( τ) ( τ) A r w ( τ ) j + s j,3,, (),,, n; j,,, n;,,, n + where E s an evaluaton functon, y s a teacher sgnal, Δw s a changed value of connecton weght, Δh s a changed value of threshold, s a renforcement coeffcent of gradent-based method, β s a renforcement coeffcent of momentum, r s a renforcement sgnal. III. EXPERIENT A. ethod Fg. shows a nonlnear tme seres sgnals predcton system of fve nputs four outputs whch the output at τ+ s obtaned from the nput at τ. BP networ (BPN) and the parallel-type neuron networ (PNN) of three layers are appled to ths system. Next, the experment to obtan each root mean square error () s performed under condtons n Table. Furthermore, averages of ther values are compared. 400 data of nonlnear tme seres sgnals whch s obtaned from moton equaton of a nonlnear plant at dscrete tme ISBN: (revsed on November 06) IECS 009 ISSN: (Prnt); ISSN: (Onlne)

3 Proceedngs of the Internatonal ultconference of Engneers and Computer Scentsts 009 Vol I, IECS 009, arch 8-0, 009, Hong Kong 00 ms are used for the nput sgnals and the teacher sgnals. The gan tunng s done as for these sgnals, and these values are set wthn the range from - to. Fg. 3 shows these sgnals. Table shows condtons for the evaluaton experment to ther predcton accuraces. Here, ntal values of the connecton weghts and the thresholds are decded by random numbers wthn the range shown n Table every one tme of the learnng experment. Ranges of ther mddle layer elements are decded for the number of elements an output to be the same about each neuron networ. oreover, mn and mn n Table are a learnng renforcement coeffcent of gradent-based method at each the mnmum obtaned from the rough and fne search tranng. B. Results From Fg. 4 to Fg. 9 show the average s and the standard devatons of BPN and PNN whch are obtaned from ther rough, fne and hgh fne search tranng. The length bar n these fgures shows the range of the standard devaton of plus and mnus. Table and Table 3 show condtons at the mnmum average s of BPN and PNN. Inputs x x x 5 System neuron networ Outputs ^x (τ+) - + ^x (τ+) - + ^ (τ+) - + x^ 4 (τ+) - + Teacher sgnals x (τ+) x (τ+) (τ+) (τ+) Fg. Nonlnear tme seres sgnals predcton system of fve nputs four outputs Input x x x 5 Tme[s] 3 4 Table Condtons of experment for predcton accuraces evaluaton Items of data Types BPN PNN (unt) earnng rules Bac earnng rule -propagaton for PNN Intal condtons Connecton weghts Thresholds -0.3~ ~0.3 ddle layer elements Range ~6 ~54 Interval Range 0 Rough ~ Interval 0 tmes 0. Range mn ~0.9 mn Fne Gradent mn ~9 mn earnng Interval 0. -based mn, mn renforcement The nterval before and after method coeffcents Range used by the fne search Hgh centerng on fne mn /0 n the above-mentoned Interval each nterval omentum 0 Tranng cycles Processng tmes 30,000 4 Fg. 3 Input sgnals and teacher sgnals for the tranng ddle layer elements Fg. 4 The average s and the standard devatons of BPN after the rough search ISBN: (revsed on November 06) IECS 009 ISSN: (Prnt); ISSN: (Onlne)

4 Proceedngs of the Internatonal ultconference of Engneers and Computer Scentsts 009 Vol I, IECS 009, arch 8-0, 009, Hong Kong ddle layer elements (a) x Fg. 6 Average of s and ther standard devatons of BPN after the fne search ddle layer elements (b) x (a) x ddle layer elements (c) (b) x ddle layer elements (d) (c) Fg. 5 Averages s and ther standard devatons of PNN after the rough search ISBN: (revsed on November 06) IECS 009 ISSN: (Prnt); ISSN: (Onlne)

5 Proceedngs of the Internatonal ultconference of Engneers and Computer Scentsts 009 Vol I, IECS 009, arch 8-0, 009, Hong Kong (d) Fg. 7 Averages of s and ther standard devatons of PNN after the fne search (c) Fg. 8 Average of s and ther standard devatons of BPN after the hgh fne search (d) Fg. 9 Averages of s and ther standard devatons of PNN after the hgh fne search Table Condton of BPN at the mnmum average of s Waveform name ddle layer elements Elements an output earnng renforcement coeffcent Each Average Varance Standard devaton x x (a) x Table 3 Condton of PNN at the mnmum average of s Waveform name ddle layer Each elements Total Elements an output earnng renforcement coeffcent Each Average Varance Standard devaton x x (b) x As a result whch the mnmum average s are compared, t s shown that the accuraces of all predcton ISBN: (revsed on November 06) IECS 009 ISSN: (Prnt); ISSN: (Onlne)

6 Proceedngs of the Internatonal ultconference of Engneers and Computer Scentsts 009 Vol I, IECS 009, arch 8-0, 009, Hong Kong outputs of PNN are hgher about each waveform n these tables than ones of BPN. oreover, t s shown that the mean value of the mnmum average s of all outputs of PNN decreases from one of BPN by 39.0 %. thought that the outputs accuraces are more amelorable f PNN s changed to an error convergence parallel-type neuron networ system [] whch error convergence-type neuron networ systems are appled to parallel unts of PNN. IV. DISCUSSION At the begnnng, from a vewpont of the learnng rule s consdered. There s () to calculate the renforcement sgnal led from the term of gradent-based method of (6) and (8) n the last mddle layer of BP networ (BPN) of two outputs or more. The calculaton for the amount of the product of the renforcement sgnals and connecton weghts of all outputs of the output layer whch causes the nterference to the amount of change about the connecton weghts and the thresholds s generated n (). Ths s to be changed the connecton weghts and thresholds n the last mddle layer by other outputs whch the tranng has not converged even f the tranng s converged completely by an arbtrary output. Therefore, t s thought that t s very dffcult for the output whch the tranng has been converged completely to eep the state contnuously by ths change. That s, ths output vbrates to the teacher sgnal. Furthermore, ths vbraton has the vbratng nfluence to other outputs. Ths s the mutual nterference of the learnng caused between the outputs. The nterferences bac propagate too because the change calculatons of the connecton weghts and the thresholds based on a renforcement sgnals ncludng ths nterference are executed toward the nput layer one after another n the mddle layers. A parallel type neuron networ (PNN) does not have the above-mentoned nterference, and excellent capabltes of the neuron networ can be expected because t s an output a parallel unt. Next, from the vewpont of common for the mddle layer s consdered. BPN has the mddle layer correspondng to each output n common. The condton to construct the mddle layers prepared at each output to one, that s, the common condton of the mddle layers s the case whch the output sgnal vectors of the mddle layers prepared at each output to all the nput sgnal vectors becomes equal. Ths common condton of the mddle layers s obvously met f a constructon of coeffcents of elements n the mddle layers prepared at each output s equal n any mddle layer. However, t s very dffcult to meet such a condton actually. Therefore, BPN should have enough the learnng capablty to obtan the amed output n the output layer by usng an output sgnal vector of the last mddle layer whch does not satsfy the common condton of the mddle layers. Such learnng of BPN s le a perceptron learnng only the output layer. On the other hand, PNN has the mddle layer an output to avod the above-mentoned problem whch BPN s dffcult to have mddle layer correspondng to each output n common. Therefore, the learnng s executed n the mddle layers and the output layer for the tranng of one output, and excellent capabltes of the neuron networ can be expected. From above two vewponts, t s thought that all predcton accuraces of PNN are hgher than ones of BPN. As a result, outputs accuraces of neuron networs for an applcaton are amelorable by changng BPN for PNN. oreover, t s V. CONCUSION It was shown that all predcton accuraces of the parallel-type neuron networ (PNN) was hgher than ones of BP networ (BPN), as the result of comparng the mnmum average root mean square errors (s) whch are obtaned from nonlnear tme seres sgnals predcton systems of fve nputs four outputs usng BPN and PNN. oreover, t was shown that the mean value of the mnmum average s of all outputs of PNN decreased from one of BPN by 39.0 %. The future wor s to perform smulaton experment of an error convergence parallel-type neuron networ system whch mproves PNN further and to evaluate the effectveness. ACKNOWEDGENT We wsh to express our grattude to members n our laboratory who cooperate always n the academc actvty. REFERENCES [] S. Kobayaawa and H. Yoo, The Volterra Flter Bult-n Neural Networ for the Arcraft Ptch Atttude Control, Proc. of the 58 th Jont Conf. of Electrcal and Electroncs Engneers n Kyushu Japan, 005, p. 49. [] S. Kobayaawa and H. Yoo, Evaluaton for Predcton Capablty of Parallelzed Neuron Networs, Proc. of the 8 th SOFT Kyushu Chapter Annu. Conf. Japan, 006, pp [3] S. Kobayaawa and H. Yoo, Applcaton to Predcton Problem of Parallelzed Neuron Networs n the Arcraft, Techncal Report of IEICE, vol. 06, no. 47, SANE , Jan. 007, pp [4] S. Kobayaawa and H. Yoo, Evaluaton of the earnng Capablty of a Parallel-type Neuron Networ, Proc. of the st Internatonal Symp. on Informaton and Computer Elements 007 Japan, 007, pp [5] S. Kobayaawa and H. Yoo, Expermental Study for Domnance to Accuracy of Predcton Output of Parallel-type Neuron Networ, Techncal Report of IEICE, vol. 08, no. 54, NC008 0, ay 008, pp [6] D.E. Rumelhart, G.E. Hnton, and R.J. Wllams, earnng Representatons by Bac-propagatng Errors, Nature, vol. 33, no. 6088, Oct. 986, pp [7] F. Rosenblatt, The Perceptron: A Probablstc odel for Informaton Storage and Organzaton n the Bran, Psychologcal Revew, vol. 65, no. 6, Nov. 958, pp [8] K. Funahash, On the Capabltes of Neural Networs, Techncal Report of IEICE, vol. 88, no. 6, BE88 5, July 988, pp [9] K. Funahash, On the approxmate realzaton of contnuous mappngs by neural networs, Neural networs, vol., no. 3, 989, pp [0]. Swercz, J. Kochanowcz, J. Wegele, R. Hurst, D. S. ebesnd, Z. ara, E. R. elhem, and J. Krejza, earnng Vector Quantzaton Neural Networs Improve Accuracy of Transcranal Color-coded Duplex Sonography n Detecton of ddle Cerebral Artery Spasm Prelmnary Report, Neuronformatcs, vol. 6, no. 4, Aug. 008, pp [] I. Babaoglu, O. K. Bayan, N. Aygul, K. Ozdemr, and. Bayra, Assessment of exercse stress testng wth artfcal neural networ n determnng coronary artery dsease and predctng leson localzaton, Expert Systems wth Applcatons, vol. 36, ssue, part, ar. 009, pp [] S. Kobayaawa and H. Yoo, Proposal of Error Convergence-type Neuron Networ System, Extended Abstracts of 008 Internatonal Symp. on Intellgent Informatcs Japan, 008, p. 9. ISBN: (revsed on November 06) IECS 009 ISSN: (Prnt); ISSN: (Onlne)

7 Proceedngs of the Internatonal ultconference of Engneers and Computer Scentsts 009 Vol I, IECS 009, arch 8-0, 009, Hong Kong ERRATA Date of modfcaton s November, 06. Errors from lne from last lne of p. 57 to lne of p. 58 are corrected as Each 00 data of nonlnear tme seres sgnals from x to x4 whch are obtaned from moton equaton of a nonlnear plant at dscrete tme 0 ms are used for nput sgnals and teacher sgnals of the system, respectvely. 00 data of x5 s a control sgnal to ths plant.. ISBN: (revsed on November 06) IECS 009 ISSN: (Prnt); ISSN: (Onlne)