Neural Networks: Algorithms and Special Architectures

Transcription

1 Internatonal Journal of Electrcal Engneerng. ISSN Volume 3, Number 3 (2), Internatonal Research Publcaton House htt:// Neural Networks: Algorthms and Secal Archtectures Bharat Bhushan and Madhusudan Sngh 2 Delh College of Engneerng, Delh, Inda Emal: bharatdce@yahoo.co.n 2 Delh College of Engneerng, Delh, Inda Emal: madhusudan@dce.ac.n Abstract he aer s focused on neural networks, ther learnng algorthms, secal archtecture and SVM. General learnng rule as a functon of the ncomng sgnals s dscussed. Other learnng rules such as Hebban learnng, delta learnng, ercetron learnng, Least Mean Square (LMS) learnng, Wnner ake All (WA) learnng are resented as a dervaton of the general learnng rule. Archtecture secfc learnng algorthms for cascade correlaton networks, functonal lnk networks, counterroagaton networks and Radal Bass Functon (RBF) networks are descrbed. Case study has been done for neural network based dentfcaton and control for temerature control of a water bath. Suort Vector Machnes (SVMs) an extremely owerful method of dervng effcent models for multdmensonal functon aroxmaton and classfcaton have been dscussed. Introducton Artfcal neural networks are systems that are delberately constructed to make use of some organzatonal rncles resemblng those of the human bran. hey reresent the romsng new generaton of nformaton rocessng systems. Artfcal neural networks (ANNs) have a large number of hghly nterconnected rocessng elements (nodes or unts) that usually oerate n arallel and are confgured n regular archtectures. he collectve behavor of an ANN, lke a human bran, demonstrates the ablty to learn, recall, and generalze from tranng atterns or data. ANNs are nsred by modelng network of real (bologcal) neurons n the bran. A survey [],[2],[3],[4],[5],[6],[7],[8],[9],[],[],[2],[3] on neural networks learnng methods and secal feedback archtectures have been done. Sngle-layer and multlayer feedforward networks and ther assocated suervsed learnng rules have

2 76 Bharat Bhushan and Madhusudan Sngh been descrbed. Case study on temerature control of water bath usng feedforward NN have been done. Varous learnng algorthms lke Hebban learnng rule, Correlaton learnng rule, Instar learnng rule, Wnner akes All, Outstar learnng rule, Percetron learnng rule, Wdrow-Hoff learnng rule, Delta learnng rule and Error Backroagaton learnng are dscussed. Secal feedforward archtectures lke Functonal lnk networks, feedforward verson of the counterroagaton network, LVQ learnng vector quantzaton, WA archtecture, Cascade Correlaton archtecture and RBF-Radal bass functon networks have been dscussed. Both the multlayer ercetron and the radal bass network are based on the oular learnng aradgm of error-correcton learnng. he synatc weghts of these networks are adusted to reduce the dfference (error) between the desred target value and corresondng outut. SVM offers an extremely owerful method of dervng effcent models for multdmensonal functon aroxmaton and classfcaton. Neural Networks he feedforward neural networks allow only for one drectonal sgnal flow. Furthermore, most of feedforward neural networks are organzed n layers. An examle of the three-layer feedforward neural network s shown n Fgure. H ID D E N L A Y E R S O U P U LAYER INPUS OUPUS Fgure : Feedforward Neural Network. he feedforward neural network can be used for nonlnear transformaton (mang) of a multdrectonal nut varable nto another multdmensonal varable n the outut. Presently, there s no satsfactory method to defne how many neurons should be used n hdden layers. Usually ths s found by tral and error method. In general, t s known that f more neurons are used, more comlcated shaes can be maed. On the other sde, neural networks wth large number of neurons lose ther ablty for generalzaton, and t s more lkely that such network wll try to ma nose suled to the nut also.

3 Neural Networks: Algorthms and Secal Archtectures 77 Case Study: Neural network based dentfcaton & control for temerature control of a water bath. Nonlnear lant model [5] s y ( k + ) = F * y( k) + ( g /+ ex(.5* y( k) 4)* u( k) + ( F) * y o (a) β where F = ex( α ); g = [ ex( α )] (b) α α =.5 β = Y = 25 C o 4 3 = 3 sec u = Inut, lmted between and 5 volts. Wth these arameters, smulated system s equvalent to a SISO temerature control system of a water bath that exhbts lnear behavour uto about 7 o C and then becomes nonlnear and saturates at about 8 o C. he task s to control the lant usng MLP neural network and lant to follow a secfed reference y r (k). Feedforward neural network s created and traned usng seven neurons n hdden layer and one neuron n outut layer. Inut s a tran of ulses of random heght between and 5 & random wdth between and Plant Inut-- NN Inut- 3 2 Plant outut-b NN outut--r m e S t e m e S t e Fgure 2: Plant nut-nn nut and Plant outut-nn outut. Core Learnng Algorthms for Neural Networks Smlarly to the bologcal neurons, the weghts n artfcal neurons are adusted durng a tranng rocedure. Varous learnng algorthms were develoed and only a few are sutable for multlayer neuron networks. Some use only local sgnals n the neurons, other requre nformaton from oututs, some requre a suervsor who knows what oututs should be for the gven atterns, other-unsuervsed algorthms do not requre such nformaton.

4 78 Bharat Bhushan and Madhusudan Sngh A. Hebban Learnng Rule he Hebb [2] learnng rule s based on the assumton that f two neghbour neurons must be actvated and deactvated at the same tme, then the weght connectng these neurons should ncrease. For neurons oeratng n the ooste hase, the weght between them should decrease. If there s no correlaton, the weght should reman unchanged. hs assumton can be descrbed by the formula Δ w = ηy x, =,2,.,n; =,2,.,m. (2) Where y a( w X ) = Where the sgnal on the th nut and w s the weght from -th to -th neuron, η s the learnng constant, (3) x s y s the outut sgnal. hus, the Hebban learnng rule s an unsuervsed learnng rule for a feedforward network snce t uses only the roduct of nuts and oututs to modfy the weghts. No desred oututs are gven to generate the learnng sgnal to udate the weghts. B. Correlaton learnng rule he correlaton-learnng rule s based on a smlar rncle as the Hebban learnng rule. It assumes that weghts between smultaneously resondng neurons should be largely ostve, and weghts between neurons wth ooste reacton should be largely negatve. Mathematcally, ths can be wrtten that weghts should be roortonal to the roduct of states of connected neurons. Δw = ηx d (4) C. Percetron learnng rule In ths learnng arorate weghts of a smle ercetron are found so that t can fnd a decson lane. It rocesses the tranng data one by one and adusts the weghts ncrementally. For the ercetron-learnng rule, the learnng sgnal s the general weght-learnng rule s set as the dfference between the desred and actual PE resonses. hat s, Δw = η ( d y ) x, Where y sgn( W X ) = Weghts are adusted only when the actual outut y dsagrees wth weghts are ntalzed at any value n ths method. (5) d. he D. Wdrow-Hoff learnng rule Wdrow and Hoff develoed a suervsed tranng algorthm, whch allows tranng a neuron for the desred resonse. A cost functon E(W) measures the system s erformance error by m ( k) ( ) = ( d 2 k= = E W w x ( k) ) 2 (6)

5 Neural Networks: Algorthms and Secal Archtectures 79 E(W ) s normally ostve but aroaches zero when =,2,----,. where s the number of aled atterns (k ) y aroaches (k ) d for k ( k ) ( k ) ( k ) Δw = η ( d W X ) x, =,2, ,m. (7) k= he learnng rule s called the Adalne learnng rule or the Wdrow-Hoff learnng rule. It s also referred to as the least mean square (LMS) rule. he Wdrow-Hoff learnng rule s very smlar to the ercetron learnng rule, dfference s that ercetron learnng rule orgnated n an emrcal Hebban assumton, whle the Wdrow-Hoff learnng rule was derved from the gradent-descent method whch can be easly generalzed to more than one layer. Furthermore, the ercetron learnng rule stos after a fnte number of learnng stes, whle, n rncle, the gradent descent aroach contnues forever, convergng only asymtotcally to the soluton. E. Instar learnng rule If nut vectors, and weghts, are normalzed, or they have only bnary bolar values (- or +), then the net value wll have the largest ostve value when the weghts have the same values as the nut sgnals. herefore, weghts should be changed only f they are dfferent from the sgnals Δw = η ( x w ) (8) F. WA- Wnner akes All Learnng s based on the clusterng of nut data to grou smlar obects and searate dssmlar ones. In ths algorthm weghts are modfed only for the neuron wth the hghest net value. Weghts of remanng neurons are left unchanged. hs suervsed algorthm (because we do not know what are desred oututs) has a global character. he net values for all neurons n the network should be comared n each tranng ste. he WA algorthm, develoed by Kohonen [7] s often used for automatc clusterng and for extractng statstcal roertes of nut data. G. Outstar learnng rule In the outstar learnng rule t s requred that weghts connected to the certan node should be equal to the desred oututs for the neurons connected through, those weghts Δw = η( d w ) (9) Where d s the desred neuron outut and η s small learnng constant whch further decreases durng the learnng rocedure. hs s the suervsed tranng rocedure because desred oututs must be known. Both nstar and outstar learnng rules were develoed by Grossberg [8].

6 8 Bharat Bhushan and Madhusudan Sngh H. Delta learnng rule In ths rule, cost functon s n m ( k ) ( k) 2 E( W ) = [ d a( w x )] 2 k= = = () where k ndcates the kth tranng data and s the total number of tranng data. ( k ) ( k ) ' ( k ) ( k ) Δw = η [ d a( net )] a ( net ) x k= () ( k ) ( k ) where net = W X s the net nut to the th PE when the kth nut attern s resented. Δw = η[ d ( k) a( W X ( k ) ' )] a ( W X ( k) ) x (2) In case of the ncremental tranng for each aled attern the weght change should be roortonal to nut sgnal, to the dfference between desred and actual oututs, and to the dervatve of the actvaton functon. I. Error backroagaton learnng rule he delta-learnng rule can be generalzed for multlayer networks. Usng a smlar aroach, as t s descrbed for the delta rule, the gradent of the global error can be comuted n resect to each weght n the network. ( k) d( E) dw = 2 n = [( d o ' ' ) F { z } f ( net ) x ] (3) Secal Feedforward Archtectures A. Functonal lnk networks One layer neural networks are relatvely easy to tran, but these networks can solve only lnearly searated roblems. Functonal lnk networks shown n fgure 3 [9] are sngle layer neural networks that are able to handle lnearly nonsearable tasks usng the arorately enhanced nut reresentaton. hus, fndng a sutably enhanced reresentaton of the nut data s the key ont of the method. he roblem wth the functonal lnk network s that roer selecton of nonlnear elements s not an easy task.

7 Neural Networks: Algorthms and Secal Archtectures 8 Non-Lnear Elements Inuts Oututs + Fgure 3: One layer neural network wth arbtrary nonlnear terms. B. WA archtecture he WA network was roosed by Kohonen [7]. hs s bascally a one-layer network used n the unsuervsed tranng algorthm to extract a statstcal roerty of the nut data. Learnng s based on the clusterng of nut data to grou smlar obects and searate dssmlar ones. At the frst ste all nut data s normalzed so the length of each nut vector s the same, and usually equal to unty. he actvaton functons of neurons are unolar and contnuous. he learnng rocess starts wth a weght ntalzaton to small random values. 5 = Out=F(net) Fgure 4: Neuron as the Hammng dstance classfer. If nuts of the neuron of fgure 3 are bnares, for examle X=[, -,, -, -] then the maxmum value of net 5 Net= x w =XW (4) = s when weghts are dentcal to the nut attern W=[,-,,-,-]. In ths case net =5. For bnary weghts and atterns net value can be found usng equaton:

8 82 Bharat Bhushan and Madhusudan Sngh n Net= x w = = XW = n 2HD (5) Where n s the number of nuts and HD s the Hammng dstance between nut vector X and weght vector W. hs concet can be extended to weghts and atterns wth analog values as long as both lengths of the weght vector and nut attern vectors are the same. he Eucldean dstance between weght vector W and nut vector X s W X = ( w x) + ( w2 x2) + + ( w n xn ) (6) n 2 W X = ( w x ) (7) = W X = ( WW 2WX + XX ) when the lengths of both the weght and nut vectors are normalzed to value of one X = and W = (9) hen the equaton smlfes to W X = (2 2WX ) (2) Maxmum value of net = when W and X are dentcal. C. Feedforward verson of the counterroagaton network he counterroagaton network was orgnally roosed by Hecht-Nlsen [2]. hs network, whch s shown n Fgure 5 requres numbers of hdden neurons equal to the number of nut atterns, or more exactly, to the number of nut clusters. When bnary nut atterns are consdered, then the nut weghts must be exactly equal to the nut atterns. In ths case, (8) Kohonen Layer normalzed nuts oututs unolar n e u r o n s Sum mng Crcuts Fgure 5: Counter roagaton Network.

9 Neural Networks: Algorthms and Secal Archtectures 83 Net = X W = (n 2HD(X, W)) (2) where n s the number of nuts, w are weghts, x s the nut vector, and HD (w, x) s the Hammng dstance between nut attern and weghts. Snce for a gven attern, only one neuron n the frst layer may have the value of one and remanng neurons have zero values, the weghts n the outut layer are equal to the requred outut attern. he counterroagaton network s very easy to desgn. he number of neurons n the hdden layer should be equal to the number of atterns (clusters). he weghts n the nut layer should be equal to the nut atterns and, the weghts n the outut layer should be equal to the outut atterns. D. LVQ Learnng Vector Quantzaton At LVQ network the frst layer detects subclasses. he second layer combnes subclasses nto sngle class (Fgure 6). Frst layer comutes Eucldean dstance between nut attern and stored atterns. Wnnng neuron s wth the mnmum dstance. Comettve Layer W Lnear Layer V normalzed nuts oututs unolar neurons Fgure 6: LVQ Learnng Vector Quantzaton. E. Cascade Correlaton Archtecture he cascade correlaton archtecture was roosed by Fahlman and Lebere (Fgure 7). he rocess of network buldng begns wth some nuts and one or more outut nodes, but wth no hdden nodes. Every nut s connected to every outut node. he outut nodes may be lnear unts, or they may emloy some nonlnear actvaton functon such as a bolar sgmodal actvaton functon. In each learnng ste, the new hdden neuron s added and ts weghts are adusted to maxmze the magntude of the correlaton between the new hdden neuron outut and the resdual error sgnal on the network outut that we are tryng to elmnate. Each hdden neuron s traned ust once and then ts weghts are frozen. he network learnng and buldng rocess s comleted when satsfed results are obtaned.

10 84 Bharat Bhushan and Madhusudan Sngh + + Outut Neurons + Oututs Inuts + Fgure 7: Cascade correlaton archtecture. F. RBF- Radal Bass Functon Networks he structure of the radal bass network s shown n fgure 8. hs tye of network usually has only one hdden layer wth secal neurons. Each of these neurons resonds only to the nut sgnals close to the stored attern. Hdden "neurons" Inuts Stored Stored D D Oututs Stored Stored D D Outut Normalzaton Summng Crcut Fgure 8: Radal Bass Functon Networks. he behavor of the neurons sgnfcantly dffers from the bologcal neuron. In ths neuron, exctaton s not a functon of the weghted sum of the nut sgnals. Instead, the dstance between the nut and stored attern s comuted. If ths dstance s zero then the neuron resonds wth a maxmum outut magntude equal to one.

11 Neural Networks: Algorthms and Secal Archtectures 85 hs neuron s caable of recognzng certan atterns and generatng outut sgnals beng functons of a smlarty. Suort Vector Machnes Suort vector machnes (SVMs) offer an extremely owerful method of dervng effcent models for multdmensonal functon aroxmaton and classfcaton. SVMs can be used to classfy lnearly and nonlnearly searable data. hey can be used as nonlnear classfers and regresson machnes by mang the nut sace to a hgh-dmensonal feature sace. A suort vector machne [] has a basc format, as dected n Fgure (9), where φ k (X) s a nonlnear transformaton of the nut feature vector X nto a hgh dmensonal sace new feature vector φ(x)=[ φ (X) φ 2 (X) φ 3 (X) ---- φ (X)]. he outut y s comuted as: Fgure 9: An SVM neural network structure. y( X ) = w ϕ ( X ) + b = ϕ( X ) W + b (22) k = k k Where W=[w w w ] s the x weght vector, and b s the bas term. he dmenson of φ(x) (=) s usually much larger than that of the orgnal feature vector(=m). It has been argued that mang a low-dmensonal feature nto a hgherdmensonal feature sace wll lkely make the resultng feature vectors lnearly searable. In other words, usng φ as a feature vector s lkely to result n better attern classfcaton results. Gven a set of tranng vectors {X (); N}, weght vector W as: N W = γ ϕ ( X ( )) = γφ = where φ = [ ϕ ( X ()) ϕ ( X (2)) ϕ ( X ( N ))] s an Nx matrx, and γ s a xn vector. Substtutng W nto y(x) yelds: (23)

12 86 Bharat Bhushan and Madhusudan Sngh N y( X ) = ϕ ( X ) W + b = γ ϕ( X ( )) + b = γ K ( X, X ( )) + b = = (24) where the kernel K ( X, X ( )) s a scalar-valued functon of the testng samle X and a tranng samle X (). By usng kernel functons, SVMs can be used to fnd dscrmnant functons for nonlnearly searable data. However, roblems ersst n the a ror choce of the kernel functon and ts varous arameters. he attrbutes that make SVMs attractve are: () hey rovde a unque searatng surface that maxmzes the margn or searaton. (2) Usng an adequately chosen kernel, they rovde a way to obtan nonlnear classfcaton boundares by roectng data from the nut sace to a hgher dmensonal feature sace. (3) hey generalze well from relatvely few data onts. (4) hey rovde model selecton usng conventonal mathematcal rogrammng technques and are solvable by readly avalable software. N Concluson Core learnng algorthms for neural networks have been dscussed. Secal feedback archtectures usng neural networks have been resented. he feedforward neural network can be used for nonlnear transformaton (mang) of a multdmensonal nut varable nto another multdmensonal varable n the outut. Neural networks could erform better than teacher (traner). SVM archtecture have been llustrated and dscussed. A case study has been erformed on temerature control of water bath usng feedforward NN. References [] Grossberg, S., 969, Embeddng felds: a theory of learnng wth hysologcal mlcatons, Journal of Mathematcal Psychology 6: [2] Beren, H., 99, Neural Networks and Fuzzy Logc n Intellgent Control, n roc. of IEEE nternatonal symosum on Intellgent Control, 5-7 Setember 99, ages: [3] Kohonen, 99, he Self Organzed mas, n roceedng of IEEE 78 (9): [4] Lng, C.., and Lee, C.S.G., 99, Neural Network-Based Fuzzy Logc Control and Decson System, n IEEE transactons on Comuters, Vol.4, No.2, December 99,ages [5] eng, L.C., and George, L.C.S., 996, Neural Fuzzy systems, Prentce Hall P R, Uer Saddle Rver, NJ [6] Eberhart, R.C., 998, Overvew of Comutatonal Intellgence, n the roc. of IEEE nternatonal conference on Engneerng n Medcne and Bology Socety, vol. 2, No.3, 998, ages:

13 Neural Networks: Algorthms and Secal Archtectures 87 [7] Kng, R.E., 999, Comutatonal Intellgence n Control Engneerng, Marcel Dekker, Inc., 999 [8] Burns, R.S., 2, Advanced Control Engneerng, Butter worth Henemann, 2. [9] Hu, Y.H., and Hwang, J.N., 22, Handbook of Neural Network Sgnal Processng, CRC Press. [] Verr, B.H.A., and Poggo,., Learnng and Vson Machnes, n the roc. of IEEE nternatonal conference vol.9, No.7, July22. [] Jang, J.S., Sun, C.., and Mzutan, E., 24, Neuro-Fuzzy and Soft Comutng, Prentce Hall, 24. [2] Wlamowsk, B.M., 24, Methods of Comutatonal Intellgence, n the roc. of IEEE nternatonal conference on Industral echnology 24, ages: -8. [3] Lu, H.J., Wang, Y.N., and Lu, X.F., 25, A method to choose Kernel functon and ts arameters for Suort Vector Machnes, n the roc. of IEEE nternatonal conference on Machne, Learnng and Cybernetcs, 8-2 August 25, ages: Vtae Madhusudan was born n Ghazur (U.P.) Inda, n 968. He receved B.Sc (Electrcal Engneerng) degree from the Faculty of echnology, Dayalbagh Educatonal Insttute, Dayalbagh Agra, Inda, n 99 and the M.E. degree from the Unversty of Allahabad, Allahabad, Inda, n 992. He receved hs Ph.D. degrees n Deartment of Electrcal Engneerng, Unversty of Delh, Inda n 26. In 992, he oned the Deartment of Electrcal Engneerng, NERIS-Nrul, Arunachal Pradesh, as lecturer. In June 996, he oned Electrcal Engneerng Deartment, IE Lucknow, Inda as a lecturer. In March 999, he oned Deartment of Electrcal Engneerng, Delh College of Engneerng, Delh Inda as an Assstant Professor. In October 27 he became the rofessor of Electrcal engneerng n Delh College of Engneerng, Delh Inda. Hs research nterests are n area of modelng and analyss, of Electrcal Machnes, voltage control asects of Self Excted Inducton Generator, Power electroncs, and drves. He s a member of the Insttuton of Engneers (IE), Inda, member of Insttuton of Electroncs and elecommuncaton Engneers, New Delh, Inda. He s a Lfe Member of the Indan Socety for echncal Educaton, New Delh, Inda. He s also a member of IEEE (USA). Bharat Bhushan receved B.E. (Electrcal Engneerng) degree from the Delh College of Engneerng, Unversty of Delh, Delh, n 992 and the M.E.(Electrcal Engneerng) degree from the Delh College of Engneerng, Unversty of Delh, Delh, n 996.

14 88 Bharat Bhushan and Madhusudan Sngh In June 997, he oned the Deartment of Instrumentaton Engneerng, Sant Longowal Insttute of Engneerng & echnology, Longowal, Punab as Lecturer. In January 998, he oned Instrumentaton & Control Engneerng Deartment, Ambedkar Polytechnc, New Delh as Lecturer. In May 999, he oned Deartment of Electrcal Engneerng, Delh College of Engneerng, Delh as Lecturer. From December 28, he s Assstant Professor n Deartment of Electrcal Engneerng, Delh College of Engneerng, and Delh, Inda. Hs research nterests are n the area of Fuzzy Logc, Neural Networks, Comutatonal Intellgence and Robotc manulators. He s member of he Insttuton of Engneerng & echnology, UK.