Removal of Hidden Neurons by Crosswise Propagation

Transcription

1 Neural Informaton Processng - etters and Revews Vol.6 No.3 arch 25 ETTER Removal of dden Neurons by Crosswse Propagaton Xun ang Department of anagement Scence and Engneerng Stanford Unversty CA 9535 USA Insttute of Computer Scence and Technology Peng Unversty Bejng 87 Chna Emal: langxun@cst.pu.edu.cn (Submtted on December 3 24) Abstract The hdden neuron s removed by analyzng the orthogonal projecton correlatons among the outputs of other hdden neurons. The technque of crosswse propagaton (CP) s used to update the remanng weghts and thresholds. Experments llustrate that the method gves better ntal ponts for retranngs and retranngs cost less epochs. eywords hdden neurons orthogonal projecton crosswse propagaton (CP). Introducton Scholars have showed enormous nterests for removng the superfluous weghts and hdden neurons n neural networs [][3][2][3][2]. Too many weghts or hdden neurons may lead to overfttng of data and poor generalzaton whle too few weghts and hdden neurons may not allow the neural networ to learn the data suffcently accurately. The frequently used technques can be classfed nto two categores the methods of removng weghts and the methods of removng hdden neurons. Ths paper focuses on the second category. Ths paper presents a method based on the senstvty and mportance of hdden neurons by the orthogonal projecton. In addton the method does not smply tae away the less mportant hdden neurons as most technques. Instead an approach called the weght crosswse propagaton (CP) s appled to elmnate the weght nformaton loss to the least and the CP smply apples the coeffcents of the orthogonal projectons. Frst we defne the hdden output row vector or smply the hdden row vector. A hdden row vector s composed of outputs of one hdden neuron wth respect to all the tranng patterns. Second we defne the augmented hdden output row vector or smply the augmented hdden row vector. An augmented hdden row vector s ether a hdden row vector or row vector (- -). As t s well nown the hdden layer can be compressed f the augmented hdden row vectors are lnearly dependent [][3][2][3][2]. owever n applcaton there are few cases n whch the augmented hdden row vectors are lnearly dependent. The common practce s as follows. When people tran a networ they always choose as few hdden neurons as possble n the begnnng. When the networ cannot learn the mappng they add hdden neurons. After the networ s traned they remove some hdden neurons usng the prunng algorthms and retran the networ. Ths paper s the drect extenson of the case of lnear dependence [3]. Instead of usng lnear dependence correlatons ths paper employs an orthogonal projecton crteron. In addton the method can be appled to both bnary and real nputs and outputs. The method s dvded nto two stages. In the frst stage for each hdden row vector we calculate ts orthogonal projecton n the space spanned by the other augmented hdden row vectors and obtan a dstance between ts orthogonal projecton and the hdden row vector tself. Then we remove the hdden neuron wth the smallest dstance. The second stage apples the CP method. Our motvaton s that f a hdden row vector can be best approxmately expressed by the other augmented hdden row vectors then the other augmented hdden row vectors mght also best express the nformaton provded by ths hdden row vector. [6] realzed ths dea n the cascade-correlaton neural networs. owever the general neural networ archtecture was not addressed n [6]. Ths paper accomplshes ths wor. In a word the wor n ths paper s more general and has wder applcaton potentals. 79

2 Removal of dden Neurons by Crosswse Propagaton X. ang w z p x p w N w θ ψ y p x N p N w N y p p... P θ z p ψ Fgure. A three-layer perceptron. We demonstrate our method n the archtecture of standard three-layer (one hdden layer) perceptrons. It s not dffcult to extend t to a multlayer perceptrons. In a three-layer perceptron (see Fgure ) there are N nput ports (wthout neurons) hdden neurons output neurons. W{w hn } N denotes the nput weght matrx (the one between the nput layer and the hdden layer) or smply nput-hdden matrx { mh } denotes the output weght matrx (the one between the hdden layer and output layer) or smply hdden-output matrx Θ (θ θ )τ denotes the threshold column vector n the hdden layer Ψ(ψ...ψ ) τ denotes the threshold column vector n the output layer and ( ) τ denotes the transpose of ( ) n N h... m.... The actvaton functon n the hdden layer and output layer s the sgmodal functon f ( x) () γ ( xϕ ) e where γ > ϕ R. et P be the number of tranng pattern pars X p (x p... x Np ) τ [] N be the nput pattern column vector Z p (z p... z p ) τ [] be the hdden layer output column vector T p (t p... t p ) τ [] be the output pattern column vector or target vector Y p (y p... y p ) τ [] be the networ output vector p... P. We wrte the matrx of outputs of hdden neurons or smply the hdden matrx as ] P [ Z Z P ] [ (2) where h (z h... z hp ) [] P (h... ) are the hdden row vectors. For a fxed tranng set h s a constant vector. If a vector contans the th element (augmented vector) or a matrx contans the th row or the th column vector (augmented matrx) we assgn a to ts superscrpt. et (z... z P )(-... -). Then the augmented hdden matrx s and the augmented hdden-output matrx s [] ( ) P (3) ( ) [ Ψ ] []. (4) If a vector does not contan the th element or a matrx does not contan the th row or the th column vector we assgn a - to ts superscrpt ( s the ndex of the hdden neuron to be removed wth runnng from to ). Then the correspondng augmented hdden matrx s 8

3 Neural Informaton Processng - etters and Revews Vol.6 No.3 arch 25 8 P ] [ (5) and the correspondng augmented hdden-output matrx s ] [ Ψ ] [ Ψ ] [ ) ( ) ( ) ( ) ( ψ ψ. (6) Ths paper s organzed as follows. In secton 2 we frst present the method to determne a superfluous hdden neuron by the orthogonal projecton crteron. Then we show how to update the new weghts and thresholds n the reduced networ. Smulatons are dscussed n secton 3. Secton 4 concludes the paper. 2. Removal of dden Neurons by Crosswse Propagaton (CP) A hdden layer can be compressed f the hdden row vectors are lnearly dependent [3]. Suppose that after prunng as n the above we have hdden neurons then... are lnearly ndependent. ence P ran ran (7) or P-. Ths means that the number of hdden neurons s always smaller than or equal to the number of tranng pattern pars mnus one after the networ s pruned. If > P- the method n [3] can be used untl P-. At ths tme f hdden row vectors... are stll lnearly dependent we contnue usng the method n [3] untl hdden row vectors... are lnearly ndependent (for clarty we always use to denote the number of hdden neurons namely whenever we remove a hdden neuron we redenote the new number of hdden neurons as ). Consequently n the followng we always suppose that hdden row vectors... are lnearly ndependent namely ran and ran. The space S - spanned by (-... -) and the other hdden row vectors j (j ) can be expressed as S - span{ }. (8) If and only f the rows of matrx -j are lnearly ndependent then for vector j there exsts a unque row vector of the coeffcents for the lnear combnaton of j Π Π [] ) ( ) ( (9) mnmzng the Eucldean dstance [2][5] between Π -j -j and j such that Proj( j ) Π -j -j S -j (S -j s obvously complete) where Proj( ) s the orthogonal projecton of ( ) nto space S -j Π -j -j - j mn ξ ξ( -j )- j. () It s well nown [2] that the vector Π -j -j - j s orthogonal to the rows of -j

4 Removal of dden Neurons by Crosswse Propagaton X. ang [Π -j -j - j ]( -j ) τ () and Π -j s determned by Π -j [ -j ( -j ) τ ] j ( -j ) τ. (2) Snce ran -j and ran -j ran[( -j ) τ ] - ran[ -j ( -j ) τ ] mn{ran -j ran( -j ) τ } (3) we have ran [ -j ( -j ) τ ] where -j ( -j ) τ an matrx. Thus -j ( -j ) τ s reversble [2][5] and Π -j j ( -j ) τ [ -j ( -j ) τ ] -. (4) We obtan the orthogonal projecton of j (j... ) n space S -j and the dstance between j and Proj( j ) We want to remove the hdden neuron wth the least dstance. From Proj( j ) Π -j -j j... (5) d j j -Proj( j ) j.... (6) d mn{d j j... } (7) we now that the th hdden row vector can be best expressed by the other augmented hdden row vectors. As a result we can remove the th hdden neuron. We want to use Proj( ) Π - - to approxmate the hdden row vector then I ( nety )' Π the new augmented hdden-output matrx s Proj( ) ( Π ) ( ) ' (8) )' Π. (9) ( By (9) the th hdden neuron can be removed whle the output weghts and the thresholds n the output layer should be updated. The process (9) s called the weght crosswse propagaton (CP) for the nformaton of the removed hdden neuron s propagated crosswsely to the other hdden neurons. In the nput augmented matrx we could smply tae away the weghts connected to the th hdden neuron and the threshold n the th hdden neuron whle the remanng weghts for the other hdden neurons eep unchanged (see Fgure 2). After the removal of the th hdden neuron the new networ s retraned. Snce the above process s rrelevant to the actvaton functon t can be used n any forms of actvaton functons. In addton the method s also rrelevant to the nputs and outputs; hence t can be used n the cases of bnary and real nputs and outputs wthout lmtatons. When the orthogonal projecton of a hdden row vector s the vector tself the orthogonal projecton s degenerated nto the case of lnear dependence n [3][4]. Clearly the lnear dependence crteron leads to a precse transform whle the orthogonal projecton crteron results n an approxmaton after prunng and the approxmaton should be reduced or elmnated by the retranng. We defne d m /d as the relatve mportance comparng wth the bggest d m where d m max j {d j j...; j }. If 3 we also defne d s /d as the relatve mportance of the second smallest d s where d s mn j {d j j...; j m}. In smulaton we pre-set the values of δ m and δ s. If d m >δ m or d >δ s (2) d ds we remove the th hdden neuron. If more than one hdden neurons satsfes one of the condtons we randomly choose one to remove. Snce the magntudes of d m /d and d s /d are relevant to the dmenson of hdden row vectors P the pre-set values of δ m and δ s are normally functons of P. 82

5 Neural Informaton Processng - etters and Revews Vol.6 No.3 arch 25 z p - z p m m ψ m y p - z (-) p m ( -) z p m z () p m () m y m p y p z p m p... P Fgure 2. The method of prunng away the hdden neurons. The dotted part wll be removed. The hdden-output matrx for the other hdden neurons should be updated by the CP operaton of (9) whle the nput-hdden matrx for the other hdden neurons eeps unchanged. 3. Smulatons and Dscussons We use the functon thp2.m n the neural networ toolbox of ATAB pacages on Sun s Solars worstatons. In the followng examples the actvaton functon s shown n (). The weghts and thresholds are randomly ntalzed n the begnnng. The learnng rate s. Snce the closest cousn method of removng hdden neurons s the method n [] based on the magntudes of norms of hdden row vectors we only compare our method wth that n []. Other famous methods le optmal bran damage or surgeon are most lely to be used n removng weghts [4] often leadng to dfferent networ topologes after prunng. It s therefore not perfectly comparable between the optmal bran damage methods and our method. ore than one hdden neurons could be removed at a tme before retranng by the method n ths paper. The process s the smlar. In the frst stage we fnd hdden neurons to remove based on orthogonal projectons. If d j of more than one hdden row vectors are too small we can remove them. In the second stage the followng two CP processes are equvalent () removng one hdden neuron at a tme and mplementng the CPs one by one (2) removng a batch of hdden neurons and mplementng the CP at one tme by smlar matrx operatons. Ths s because the CP process s actually the row operatons n matrx performng the operatons of lnear combnatons row by row s equvalent to performng them a batch of rows together. owever f more than one hdden neurons are removed n one tme the new ntal pont mght not be as close to the global mnmum as n the method of removng only one hdden neuron at a tme. ence n the followng we always remove hdden neurons one by one. Example. The party problem. We pre-set δ m P and δ s P/3. The permssble sum-square-root error of the networ e δ s.5. Experment. N3 6 P8. After 723 epochs e δ.5. The norms and d j of the augmented hdden row vectors are shown n Table. Snce /.3d 4 /d 3 d m /d >δ m P8 the 3rd hdden neuron can be removed. So the 6 networ s compressed nto a networ wth 5. The retranng costs 78 epochs to reach e δ.5. The number of retranng epochs depends on many factors such as d s /d d m /d the learnng rate and the shape of error hypersurfaces around the startng pont for the retranng. For comparson we also remove the hdden neuron respectvely from the st to the 6th wthout the CP process and then retran the reduced networ. The results are shown n Table. Notcng that the st hdden neuron has the smallest norm t can be removed by the method n []. The retranng costs 244 epochs. Whle 83

6 Removal of dden Neurons by Crosswse Propagaton X. ang based on our method the 3rd hdden neuron s selected to remove and the retranng costs only 78 epochs savng about 2/3 of retranng epochs comparng wth the method n []. Table. The epoch for removng the 3rd hdden neurons wth the CP s shown n row 5 and the epochs for removng each of the hdden neurons wthout the CP respectvely are shown n row 4. j norms 2.54 (smallest) d j (smallest) epochs of retranng wthout the CP epochs of retranng wth the CP 78 Experment 2. N3 4 P8. We test 2 tmes. All of them successfully remove one hdden neuron and reach to an 3 networ repeatedly usng steps to 5. In Table 2 after 79 epochs e δ.5. We fnd that 2 and /.92d 4 /d 2 d m /d >δ s P/38/ Then the 2nd hdden neuron can be removed by the orthogonal projecton crteron. Notcng that the 2nd hdden neuron has the smallest norm t can be removed by the method n []. The retranng costs 53 epochs (see row 4 n Table 2). The epochs of retranng wth and wthout the CP are also shown n Table 2. In comparson based on our method also the 2nd hdden neuron s selected to remove and the retranng wth the CP costs only 492 epochs (see row 5 n Table 2) savng about 2/3 of retranng epochs. Table 2. The epoch for removng the 2nd hdden neurons wth the CP s shown n row 5 and the epochs for removng each of the hdden neurons wthout the CP respectvely are shown n row 4. j norms (smallest) d j (smallest) epochs of retranng wthout the CP epochs of retranng wth the CP 492 Table 3. The shortest second shortest and the maxmal dstances dj and the epochs of retranngs n each archtecture for are shown. In columns 2-4 >P-7 j S -j the mnmum the second mnmum and maxmum of dj are all zeros. In columns 2-5 and 7 no retranngs are needed snce the orthogonal projectons are degenerated nto the case of lnearly dependence d (mnmum of d j ) d s (second mnmum of d j ) d s /d d m (maxmum of d j ) d m /d epochs of retranng wth the CP For comparson the method n [] s used wth networs startng from. The total retranng epochs are 23 before the networ s cut down to 3. Whle n the frst tme the retranng s trapped nto a local mnmum when 4. The reason of the large number of retranng epochs s that at each tme when a hdden neuron s removed the retranng starts almost from the very begnnng. Example 2. Neurocontrol system. We pre-set δ m 2 and δ s (d m -d )/3. δ m s set to an absolute value because we do not want t too large. The permssble sum-square-root error of the networ e δ s.. The tradtonal way to do t s to eep the networ archtecture unchanged. owever n practce n order to tran fast onlne sometmes some hdden neurons have to be added whle tranng. When the networ has been traned wth the patterns for the specfc control the superfluous hdden neurons are removed for better generalzaton. As n [8][9] at the begnnng we set N6 4 n the neural networ and let the system follow the square wave r(t). The elements n the nput pattern vector are r(t) r(t-) u(t) u(t-) y(t) y(t-) or the nput pattern vector s X t (r(t) r(t-) u(t) u(t-) y(t) y(t-)) τ [] 6 and the element n the output pattern vector 84

7 Neural Informaton Processng - etters and Revews Vol.6 No.3 arch 25 s y(t) or the output pattern vector s T t y(t) [] t 2... The neural networ learns the control onlne wthout any pror nowledge n the begnnng. If the neural networ cannot learn the patterns a new hdden neuron s added. At the tme when the control law s learned 2. Then we try to prune away some hdden neurons. Fnally a networ wth 5 s obtaned. The learnng process of neurocontroller s shown n Fgure 3 (the dotted wave). number of epochs Fgure 3. The neurocontrol result for fast learnng the square wave. The dotted wave s the learnng process of the neurocontroller. The dotted wave n the frst valley of the square wave s the tranng process whle the hdden neurons are added. The dotted wave n the second valley of the square wave s the retranng process whle some hdden neurons are removed. From the above experments we have the followng observatons: () The smallest d s not always correspondng to the smallest norm of the hdden row vector when compressng especally when s relatvely large (see Table ). owever when s approachng to ts lower bound the smallest length of the hdden row vector frequently has the smallest d (see Table 2). (2) In large scale networs the smaller d can often be found. As a result they are easer to compress. Clearly when the networ s qute superfluous d j are qute small namely the hdden row vectors j are of great potental to express each others. As decreases the maxmal and mnmal values of d j have also the same tendency to ncrease along wth the redundancy of the networ decreases (see Table 3). (3) The CP loses less weght nformaton. So t gves to better ntal ponts for retranng or the ntal ponts for retranng are not too far away from the global mnma and the retranngs are not from the very begnnng. Ths normally leads to less retranng tme (see Tables -2). (4) Removal of the hdden neuron wth a larger value of d normally needs more retranng tme of the networ than that for a smaller value of d. If d s very small sometmes no retranng tme s even needed to reach the permssble error e δ (see Tables -3). 4. Concludng Remars The orthogonal projecton crteron s very smple to use. The hdden neurons wth small norms often have the small d j but the hdden neurons wth small d j may not have small norms. In another word the applcablty feld of method n ths paper s wder than the magntude-based prunng method []. At least the orthogonal projecton crteron s an excellent alternatve method of []. The archtecture that cannot be compressed by one of the two methods may stll be compressed by the other method. Both methods are complementary and a proper usage of ther man strengths and weanesses should lead to synergetc effects benefcal to ther common goals. It should be mentoned that the error hypersurfaces are very complex [5] and there are many factors nfluencng the selecton of hdden neurons and compressng effectveness. It s acnowledged that the method n ths paper s defntely not a method of realzng the smallest networ archtecture. Instead t s just one of effectve and practcal prunng methods. 85

8 Removal of dden Neurons by Crosswse Propagaton X. ang References [] F.. Bauer Elmnaton wth weghted row combnatons for solvng lnear equatons and least square problems In: J.. Wlnson C. Rensch (ed.) near Algebra. Sprnger-Verlag 97 pp [2] A. Ben-Israel T. E. Grevlle Generalzed Inverses - Theory and Applcaton Wley-Interscence 974. [3] E. Cantu-Paz Prunng neural networs wth dstrbuton estmaton algorthms In: Cantu-Paz E. (ed.) ecture Notes n Computer Scence Vol Sprnger-Verlag 23 pp [4] Y.. Cun J. S. Dener S. A. Solla Optmal bran damage Proc of IEEE Conf on Neural Informaton Processng Systems Denver 989 pp [5] C.. Devto Functonal Analyss and near Operator Theory Addson-Wesley 99. [6] V. Egel-Danelson. F. Augustejn Neural networ prunng and ts effect on generalzaton some expermental results Neural Parallel and Scentfc Computatons Vol. 993 pp [7] A. P. Engelbrecht A new prunng heurstc based on varance analyss of senstvty nformaton IEEE Trans on Neural Networs Vol.2 No.6 2 pp [8] A. P. Engelbrecht. Fetcher I. Cloete Varance analyss of senstvty nformaton for prunng multlayer feedforward neural networs Proc of Int Jont Conf on Neural Networs Washngton D C 999 pp.379. [9]. Fetcher V. atovn F. E. Steffens Optmzng the number of hdden nodes of a feedforward artfcal neural networ Proc of IEEE World Congress on Computatonal Intellgence Anchorage 998 pp []. agwara A smple and effectve method for removal of hdden unts and weghts Neurocomputng Vol pp [] B. assb D. G. Stor Second order dervatves for networ prunng: optmal bran surgeon Proc of Neural Informaton Processng Systems Vol pp [2] Y. rose Y. och S. jya Bac-propagaton algorthm whch vares the number of hdden unts Neural Networs Vol.4 99 pp [3] X. ang ethods of dggng tunnels nto the error hypersurfaces Neural Parallel and Scentfc Computatons Vol. 993 pp [4] X. ang Networ expanson and networ compresson Proc of IEEE Int Conf on Neural Networs Perth 995 pp [5] X. ang Complexty of error hypersurfaces n multlayer perceptrons wth bnary pattern sets Int Journal of Neural Systems Vol.4 No.3 24 pp [6] X. ang A study of removng hdden neurons n cascade-correlaton neural networs Proc of Int Jont Conf on Neural Networs Budapest 24 pp.5-2. [7] B. uller J. Renhardt Neural Networs - An Introducton Sprng-Verlag 99. [8] S. Narendra. Parthasarathy Gradent methods for the optmzaton of dynamcal systems contanng neural networs IEEE Trans on Neural Networs Vol.2 99 pp [9] Y.. Pao S.. Phllps D. J. Sobajc Neural computng and the ntellgent control systems Int Journal of Control Vol pp [2] R. Reed Prunng algorthms - a survey IEEE Trans on Neural Networs Vol pp Xun ang receved hs Ph.D. n Computer Engneerng from Tsnghua Unversty and BA from Stanford Unversty. e s currently an assocate professor at Insttute of Computer Scence at Peng Unversty. s research nterests nclude neural networs fnancal nformaton processng fnancal nformaton systems. (omepage: 86