Non-Linear Back-propagation: Doing. Back-Propagation without Derivatives of. the Activation Function. John Hertz.

Transcription

1 Non-Lnear Bac-propagaton: Dong Bac-Propagaton wthout Dervatves of the Actvaton Functon. John Hertz Nordta, Begdamsvej 7, 200 Copenhagen, Denmar Ema: Anders Krogh Eectroncs Insttute, Technca Unversty of Denmar, Budng Lyngby, Denmar, Ema: Benny Lautrup The Nes Bohr Insttute, Begdamsvej 7, 200 Copenhagen, Denmar Ema: Torsten Lehmann Eectroncs Insttute, Technca Unversty of Denmar, Budng Lyngby, Denmar, Ema: March 4,994 Abstract The conventona near bac-propagaton agorthm s repaced by a non-near verson, whch avods the necessty for cacuatng the dervatve of the actvaton functon. Ths may be expoted n hardware reazatons of neura processors. In ths paper we derve the non-near bac-propagaton agorthms n the framewor of recurrent bac-propagaton and present somenumerca smuatons of feed-forward networs on the NetTa probem. A dscusson of mpementaton n anaog VLSI eectroncs concudes the paper.

2 Introducton From a smpe rewrtng of the bac-propagaton agorthm [RHW86] a new famy of earnng agorthms emerges whch we ca non-near bac-propagaton. In the norma bac-propagaton agorthm one cacuates the errors on the hdden unts from the errors on the output neurons by means of a near expresson. The non-near agorthms presented here have theadvantage that the bac-propagaton of errors goes through the same non-near unts as the forward propagaton of actvtes. Usng ths method t s no onger necessary to cacuate the dervatve of the actvaton functon for each neuron n the networ as t s n standard bac-propagaton. Whereas the dervatves are trva to cacuate n a smuaton, they appear to be of major concern when mpementng the agorthm n hardware, be t eectronc or optca. For these reasons we beeve that non-near bac-propagaton s very we suted for hardware mpementaton. Ths s the man motvaton for ths wor. In the mt of nntey sma earnng rate the non-near agorthms become dentca to standard bac-propagaton. For sma earnng rates the performance of the new agorthms s therefore comparabe to standard bac-propagaton, whereas for arger earnng rates t performs better. The agorthms generaze easy to recurrent bac-propagaton [Am87, Pn87] of whch the standard bac-propagaton agorthm for feed-forward networs s a speca case. In ths paper we derve the non-near bac-propagaton (NLBP) agorthms n the framewor of recurrent bac-propagaton and present somenu- merca smuatons of feed-forward networs on the NetTa probem [SR87]. A dscusson of mpementaton n anaog VLSI eectroncs concudes the paper. 2 The Agorthms In ths secton we present the agorthms for non-near bac-propagaton. The dervaton can be found n the next secton. We consder a genera networ, recurrent or feed-forward, wth N neurons. The neuron actvtes are denoted V and the weght of the connecton from neuron j to neuron s denoted w j. Threshod vaues are ncuded by means of a xed bas neuron numbered =0. The actvaton (output) of a unt n the networsthen V = g(h ) h = P j w j V j + () 2

3 where g s the actvaton functon and s externa nput the unt. These equatons are apped repeatedy for a neurons unt the state of actvty converges towards a xed pont. For a feed-forward networ ths s guaranteed to happen, and the actvtes shoud be evauated n the forward drecton,.e. from nput towards output. For a recurrent networ there s no guarantee that these equatons w converge towards a xed pont, but we sha assume ths to be the case. The equatons are, however, smpest n the genera form. The error of the networ on nput pattern s dened as = ; V (2) where are the target vaues for the output unts when the nput s pattern and V s the actua output for that same nput pattern. For non-output unts 0. We dene the bacward actvatons as y = g(h + [ P (y ; V )w + ]) (3) where the constants and w be dscussed shorty. These varabes are \eectve" or \movng" targets for hdden unts n the networ. For output unts n a feed-forward networ the sum on s empty, and f the errors are a zero, these equatons have the smpe souton y = V.For non-zero error teraton of these equatons s ewse assumed to ead to a xed pont n the bacwards actvaton state (one can easy show that f the forward equatons () converge, the bacwards equatons w aso converge). Notce that durng teraton of the bacward actvatons we eep the forward actvatons xed. Now, consder a set of nput-output patterns ndexed by = ::: p, and assume that the squared error s used as the cost functon, E sq = 2 px X = ( ) 2 : (4) In terms of these new varabes the non-near bac-propagaton s then e deta-rue earnng, w j = X (y ; V )V j : (5) The constants and are repacements for the usua earnng rate, and t s requred that = s \sma". The reason we spea of a famy of of agorthms s that derent choces of yed derent agorthms. Here the parameters are aowed to der from unt to unt, but usuay they w be 3

4 the same for arge groups of unts (e.g. ones formng a ayer) whch smpes the equatons. We consder two choces of partcuary nterestng: = and =. For the rst of these pays a roe smar to the earnng rate n deta-rue earnng, snce s repaced by n (5). For the entropc error measure [HKP9] the weght update s the same (5), but y s dened as y = g(h + P (y ; V )w )+ : (6) In ths case the weght update for an output unt s exacty e the standard one, w j = P V j.for a networ wth near output unts optmzng the squared error the two equatons for y concde. Obvousy these agorthms can be used onne,.e. changng the weghts after each pattern, just as s common when usng the standard bac-propagaton agorthm. Fnay we woud e to expcty show the mportant cases of = and = for a feed-forward networ. For smpcty the ndex w be dropped. Notaton: abes the ayers from 0 (output) to L (nput). w j s the weght from unt j n ayer + to unt n ayer. Any other varabe (e y and V ) wth superscrpt refers to that varabe n ayer. It w be assumed that s the same for a unts n a ayer, =. The error on the output unts are denoted as before. Here are the two versons: 4

5 =. Output unt: y 0 = g(h0 + 0 ) (squared error). y 0 = V 0 + (entropc error). Hdden unt: y = g(h + ; P(y ; ; V ; )w ; ) Weght updatew j =(y ; V )V + j =. Output unt: y 0 = g(h0 + ) (squared error). y 0 = V 0 + (entropc error). Hdden unt: y = g(h + P (y ; ; V ; )w ; ) Weght updatew j = (y ; V )V + j 3 Dervaton of NLBP In ths secton we derve the non-near bac-propagaton n the framewor of recurrent bac-propagaton. As an ntroducton, we foow the dervaton of recurrent bac-propagaton n [HKP9, p. 72{75]. See aso [Pn87]. 3. Standard recurrent bac-propagaton Assume xed ponts of the networ are gven by (), and that the earnng s governed by an error measure E e (4). Ifwedene (4) s dentca to (2), the gradent descent earnng rue s w pq pq : (7) Derentaton of the xed pont equaton () for pq =(L ; ) p V 0 pv q (8) 5

6 wth V 0 = g 0 (h ) and the matrx L gven by L j = j ; V 0 w j : (9) If ths matrx s postve dente, the dynamcs w be contractve around a xed pont. Accordng to our assumptons ths must therefore be the case. Denng p P = V 0 p (L ; ) p (0) the weght update can be wrtten as and the 's are the soutons to w pq = p V q () = V 0 (P w + ): (2) These are the standard bac-propagaton equatons for a genera networ. In a feed-forward networ they converge to a xed pont when terated n the bacwards drecton from output towards nput. For a genera recurrent net they w converge towards a xed pont when the L-matrx s postve dente, as may easy be demonstrated. 3.2 Non-near bac-propagaton If the error measure s gven by (4) the dervatves of the error measure are = ; V for the output unts, 0 otherwse (3) For an output unt n a feed-forward networ we thus nd = V 0 ( ; V ). One of the deas of the non-near bac-propagaton s to force that nterpretaton on a the unts, denng `eectve targets' y such that / y ; V : (4) For sma eq. expanson: () can then be nterpreted as a rst order Tayor w j = V j = V P (P 0 w + )V j ' [g(h + [ j w j + ]) ; g(h )]V j (5) j where g(h )=V. The rst term n the bracets s just the output of a unt wth the norma nput and the bac-propagated error added { the eectve target: P y = g(h + [ w + ]): (6) 6

7 Note however that the \ntegraton" n (5) s qute arbtrary t s just one possbty out of many gven by w j = V j ' [g(h + [ P w + ]) ; g(h )]V j (7) where the 's are arbtrary parameters smar to the earnng rates.for consstency one now has to repace by = (y ; V ) (8) Then y s nay gven by (3) and the weght update by (5). Formay the \ntegraton" n eq. (5) s ony vad for sma (or sma = n (7)). But for arger there s no guarantee that the cean gradent descent converges anyway, and these nonnear versons mght we turn out to wor better. By mang very arge compared to, one can mae the NLBP ndstngushabe from standard bac-propagaton (the Tayor expansons w be amost exact). That woud be at the expense of hgh numerca nstabty, because y woud be very cose to V and the formua for the weght update, w j = (y ; V ), woud requre very hgh precson. On the other hand, very sma 's are ey to tae the agorthm too far from gradent descent. For these reasons we beeve that the most nterestng range s (assumng that < ). The mt = s the most stabe, numercay, and = s the most gradent-descent-e mt. Notce that f the ratos = = are the same for a neurons n the networ then the equatons tae the smper form and P y = g(h + (y ; V )w + ) (9) w j = (y ; V )V j (20) 3.3 Entropc error measure The entropc error measure s E = X " # ( + V 2 ) og +V ( ; V )og ; V ; (2) f the actvaton functon g s equa to tanh. A smar error measures exsts for other actvaton functons e g(x) =(+e ;x ) ;. It can be shown that 7

8 for ths and smar error E Instead of (3) y shoud then be dened as (6). 3.4 Interna representatons = : (22) For a feedforward archtecture wth a snge hdden ayer, the weght change formuas resembe those obtaned usng the method of nterna representatons [AKH90]. However, they are not qute the same. Usng our present notaton, n the present method we nd a change for the weght from hdden unt j to output unt of [ ; g( P w V )]V j, whe the nterna representaton approach ts[ ; g( P w y )]y j.for the nput-to-hdden ayer the expressons for the weght changes n the two approaches oo the same, but the eectve targets y j n them are derent. They are both cacuated by bac-propagatng errors ; V from P the output unts, but n the present case these V are smpy the resut g( j w j V j ) of the forward P propagaton, whe n the nterna representatons approach, V = g( j w j y j ),.e. they are obtaned by propagatng the eectve targets on the hdden ayer forward through the hdden-to-output weghts. 4 Test of Agorthm The agorthms have been tested on the NetTa probem usng a feed-forward networ wth an nput wndow of 7 etters and one hdden ayer consstng of 80 hdden unts. The agorthms were run n onne mode and a momentum term of 0:8 was used. They were tested for these vaues of :,0:25, 0:5, 0:75, and :0. The earnng rate was scaed accordng to the number n of connectons feedng nto unt, suchthat = = p n a standard method often used n bac-propagaton. Severa vaues were tred for. The nta weghts were chosen at random unformy between ;0:5= p n and 0:5= p n. For each vaue of and, 50 cyces of earnng was done, and the squared error normazed by the number of exampes was recorded at each cyce. For a runs the na error was potted as a functon of, see gure. Ceary the runs wth = are amost ndstngusabe from standard bacpropagaton, whch s aso shown. As a \coroary" these pots show that the entropc error measure s superor { even when the object s to mnmze squared error (see aso [SLF88]). In gure 2 the tme deveopment of the error s shown for the extreme vaues of and xed. 8 V 0

9 squared error eta squared error eta Fgure : Pots showng the squared error after 50 tranng epocs as a functon of the sze of the earnng rate. The eft pot s for the squared error cost functon and the rght pot for the entropc error measure. The sod ne represents the standard bac-propagaton agorthm, the dashed ne the nonnear bac-propagaton agorthm wth =, and the dot-dashed ne the one wth =. The NLBP wth =0:25 (*), =0:5 (x), and =0:75 (o) are aso shown..5 squared error tranng epocs Fgure 2: The decrease n error as a functon of earnng tme for standard bac-propagaton (sod ne), NLBP wth = (dashed ne), and = (dot-dashed ne). In a three cases the squared error functon and = 0:05 was used. 9

10 5 Hardware Impementatons A the agorthms can be mapped topoogcay onto anaog VLSI n a straght-forward manner, though seectng the most stabe s the best choce because of the mted precson of ths technoogy. In ths secton, we w gve two exampes of the mpementaton of the agorthms for a feed-forward networ usng = and the squared error cost functon: Denng a neuron error,, for each ayer (note for the output ayer) we can wrte (8) for = as ( = ; V 0 for =0 P ; w ; for >0 (23) = g(h + ) ; g(h ): (24) Thus, topoogcay ths verson of non-near bac-propagaton maps n exacty the same way on hardware as the orgna bac-propagaton agorthm the ony derence beng how s cacuated ( = g 0 (h ) for bacpropagaton) [LB93]: The system conssts of two modues: A \synapse modue" whch cacuates w, propagates j h forward and propagates bacward + j anda \neuron modue" whch forward propagates V and bacward propagates. To change the earnng agorthm to non-near bac-propagaton, t s thus necessary ony to change the \neuron modue". V DD h Bas V V SS Fgure 3: Contnuous tme non-near bac-propagaton \neuron modue" wth hyperboc tangent actvaton functon. The precson s determned by the matchng of the two derenta pars. Asmpeway to mpement a sgmod-e actvaton functon s by the use of a derenta par. Fgure 3 shows a non-near bac-propagaton \neuron 0

11 modue" wth a hyperboc tangent actvaton functon. Appyng h and ; as votages gves the V and outputs as derenta currents (the \synapse modue" can just as we cacuate ; as ). It s nterestng to notce that the crcut structure s dentca to the one used n [Bog93] to cacuate the dervatve of the actvaton functon: Repacng by a sma constant,, the output w approxmate g 0 (h ). Usng the crcut n the proposed way, however, gves better accuracy: The s not a \sma" quanttywhchmaes the nherent naccuraces ess sgncant, reatvey. Further, the crcut cacuates the desred drecty, emnatng the need of an extra mutper and thus emnatng a source of error. The accuracy of the crcut s determned by the matchng of the two derenta pars and of the bas sources. Ths can be n the order of %of the output current magntude. In a \standard mpementaton" of the \synapse modue", the h and outputs w be avaabe as currents and the V and nputs must be apped as votages. Thus the above mpementaton requres accurate transresstances to functon propery. Aso, as the same functon s used to cacuate the V s and the s, t woud be preferabe to use the same hardware as ths emnates the need of matched components. Ths s possbe f the system s not requred to functon n contnuous tme, though the output has to be samped (whch ntroduces errors). V DD h 2 Bas Bas V max : V : V mn V SS Fgure 4: Dscrete tme non-near bac-propagaton \neuron modue" wth non-near actvaton functon. The precson s determned by the accuracy of the swtched capactor. In gure 4 such a smped dscrete tme \neuron modue" whch reuses the actvaton functon boc andwhch has current nput/votage outputs s shown. Durng the coc phase, V s avaabe at the output and s samped at the capactor. Durng the 2 coc phase, s avaabe at the output. The actvaton functon saturates such that V mn V V max, though t s not very we dened. Ths s of no major concern, however the accuracy s determned by the swtched capactor. Usng desgn technques to

12 reduce charge njecton and redstrbuton, the accuracy can be n the order of 0.%of the output votage swng. As ustrated, the non-near bac-propagaton earnng agorthm s we suted for anaogue hardware mpementaton, though oset compensaton technques st have tobeempoyed. It maps topoogcay on hardware n the same way as ordnary bac-propagaton, but the crcut to cacuate the ssmuch more ecent: It can approxmate the earnng agorthm equatons more accuratey and as the agorthm requres ony smpe operatons apart from the actvaton functon, desgn eorts can be put on the eectrca speccatons of the hardware (nput mpedance, speed, nose mmunty, etc.) and on the genera shape of the sgmod-e actvaton functon. Further, as the agorthm requres ony one \speca functon", t has the potenta of very hgh accuracy through reuse of ths functon boc. 6 Concuson A new famy of earnng agorthms have been derved that can be thought of as \non-near gradent descent" type agorthms. For approprate vaues of the parameters they are amost dentca to standard bac-propagaton. By numerca smuatons of feed-forward networs earnng the NetTa probem t was shown that the performance of these agorthms were very smar to standard bac-propagaton for the range of parameters tested. The agorthms have two mportant propertes that we beeve mae them easer to mpement n eectronc hardware than the standard bacpropagaton agorthm. Frst, no dervatves of the actvaton functon need to be cacuated. Second, the bac-propagaton of errors s through the same non-near networ as the forward propagaton, and not a nearzed networ as n standard bac-propagaton. Two exampes of how anaogue eectronc hardware can utze these propertes have been gven. These advantages may aso be expected to carry over to optca mpementatons. 2

13 References [AKH90] G. I. Thorbergsson A. Krogh and J. A. Hertz. A cost functon for nterna representatons. In Neura Informaton Processng Systems 2, pages 733{740, D. S Touretzy, ed. San Mateo CA: Morgan Kaumann, 990. [Am87] [AR88] L.B. Ameda. A earnng rue for asynchronous perceptrons wth feedbac n a combnatora envronment. In M. Caud and C. Buter, edtors, IEEE Internatona Conference onneura Networs, voume 2, pages 609{68, San Dego 987, 987. IEEE, New Yor. J.A. Anderson and E. Rosenfed, edtors. Neurocomputng: Foundatons of Research. MIT Press, Cambrdge, 988. [Bog93] G. Bogason. Generaton of a neuron transfer functon and ts dervatve. Eectroncs Letters, 29:867{869, 993. [HKP9] J.A. Hertz, A. Krogh, and R.G. Pamer. Introducton to the Theory of Neura Computaton. Addson-Wesey, Redwood Cty, 99. [LB93] T. Lehmann and E. Bruun. Anaogue VLSI mpementaton of bac-propagaton earnng n artca neura networs. In Proc. 'th European Conference on Crcut Theory and Desgn, pages 49{496, Davos 993, 993. [Pn87] F.J. Pneda. Generazaton of bac-propagaton to recurrent neura networs. Physca Revew Letters, 59:2229{2232, 987. [RHW86] D.E. Rumehart, G.E. Hnton, and R.J. Wams. Learnng representatons by bac-propagatng errors. Nature, 323:533{536, 986. Reprnted n [AR88]. [SLF88] S.A. Soa, E. Levn, and M. Fesher. Acceerated earnng n ayered neura networs. Compex Systems, 2:625{639, 988. [SR87] T.J. Sejnows and C.R. Rosenberg. Parae networs that earn to pronounce engsh text. Compex Systems, :45{68,