Approximation Capabilities of Adaptive Spline Neural Networks

Transcription

1 Approxmato Capabltes of Adaptve Sple eural etworks Lorezo Vecc Paolo Campolucc Fracesco Pazza ad Aurelo Uc Dpartmeto d Elettroca e Automatca - Uverstà d Acoa Italy Va Brecce Bache 63 Acoa-Italy Fax:+39 (7) emal: aurel@eealab.ua.t Abstract I ths paper we study the propertes of eural etworks based o adaptve sple actvato fuctos (AS). Usg the results of regularzato theory we show how the proposed archtecture s able to produce smooth approxmatos of ukow fuctos; to reduce hardware complexty a partcular mplemetato of the kerels expected by the theory s suggested. Ths soluto although sub-optmal greatly reduces the umber of euros ad coectos as t gves a creased expressve power to each euro whch s also able to produce a smooth actvato fucto just cotrollg oe fxed parameter of a Catmull-Rom cubc sple. Expermetal results demostrate that there s also a advatage terms of the umber of free parameters that together wth smoothess leads to a mproved geeralzato capablty.. Itroducto T HE SUCCESSFUL applcato of eural etworks to the soluto of real-world problems requres the satsfacto of some costrats: the most mportat propertes whch should be guarateed are a good geeralzato capablty a short trag tme ad f we are terested hardware mplemetato a small sze terms of hdde uts. I [4] a ew archtecture has bee devsed whose ma feature s a sple-based adaptve actvato fucto allowg a small umber of hdde euros ad of trag epochs f compared to classcal sgmodal etworks. The purpose of ths paper s to further vestgate some Ths work s supported by the Mstero della Rcerca Scetfca e Tecologca of Italy expermetal results related to geeralzato from a theoretcal pot of vew. Recetly adaptve actvato fuctos have attracted a certa amout of research because they seem to provde better fttg propertes ad a good reducto the etwork s sze caused by the boosted capabltes of the euros. I [6] [9] polyomal fuctos were tested whle [7] the drect adaptato of the coeffcets a Look-Up-Table (LUT) was proposed: both cases a great advatage was otced terms of coecto complexty ad also a terestg smlarty wth Volterra flters; but also serous problems were ecoutered because of the lack of smoothess of the actvato fuctos ad of the adaptato mechasm. A smpler but effectve dea cossts troducg the sgmod s slope ad ga also the learg process. I [] ths approach s reported wth expermetal results hghlghtg the ablty of such etworks to better ft ukow fuctos wth a reduced umber of euros ad wth a learg algorthm whch s oly a slght modfcato of backpropagato. I spte of ths the theoretcal aspect of these mproved performace s ot deeply vestgated. The approach followed here s to derve drectly from regularzato theory a eural etwork archtecture able to provde a good geeralzato capablty through smooth actvato fuctos based o a cubc kerel. To reduce the umber of hdde uts each euro s a combato of several kerels but stead of usg the exact umber mposed by the theory

2 we have adopted a sub-optmal ad ecoomcal soluto workg o a adaptve approxmato of the cubc tract through a Catmull-Rom sple. Ths dea s very effectve for the cotrol of the total umber of free parameters.. Sple-based Actvato Fuctos. Regularzato theory ad cubc sples Regularzato theory (see [3] ad [8]) provdes a very terestg way of solvg ll-posed problems. I partcular our terest s drected to the recostructo of hypersurfaces kow oly o a fte set of pots costtutg the trag set T = {( x t) =... }. A eural model should be able to produce a smooth approxmato of the ukow fucto ad to avod the overfttg of the avalable data. Ths characterstc ot always preset whe classcal sgmods are employed ca be mposed o the etwork s archtecture by a term added to the sum of the squared errors o T ad weghted by a costat λ. H( f ) = [ t f( x )] + λ Pf () = The stablzer P s the dfferetal operator determg the kd of smoothess ad the shape of the approxmator whle. s a sutable orm. The goal s to fd the fucto f whch mmzes (). Cosderg the oe-dmesoal case a possble choce for P s d f( x) Pf = dx R ; () dx f s the expressed as a lear combato of kerels G(x)= x 3 traslated o the posto of the examples f( x) = c G( x x ); (3) = the coeffcets c are obtaed as soluto vector c of the lear system (G+λI)c=t where G s the Gree s matrx I s the detty matrx ad t s the target vector. I [3] the exteso to the multdmesoal case s proposed; the trck cossts usg a global kerel whch each dmeso s treated depedetly ad weghted by a costat µ j ; wth dmesos we have G( x) = µ G( x ) = µ x j j j= j= j 3 j. (4) It s also ofte suggested to choose a proper referece frame to better approxmate the multvarate target fucto as the sum of a umber of fuctos equal to the dmeso of the put space; ths leads to a fal expresso easly mapped o a sgle hdde layer eural archtecture: f ( x) = µ c w x α j j= = 3 j j (5) where w j s the vector detfyg the j-th drecto of the ew frame ad α j s the ceter of the -th kerel alog the j-th drecto. The f(x) gve by equato (5) has some drawbacks: frst of all coeffcets c =... are the solutos of a lear system wth geerally very large whch meas qute hard umercal problems. The a evaluato of the parameters µ j j=... s eeded.e. by crossvaldato. A possble soluto s to cosder both c ad µ j as free parameters to be foud by a learg algorthm lke backpropagato; ayway there s stll a too large umber of uts ( that s euros for each oe of the drectos) leadg to a uacceptable hardware complexty most of all terms of coectos. Istead of usg the classcal techque of prug the kerels our dea cossts costrag the whole cubc sple alog the geeral drecto w j o a sgle euro. 3 j j j j j = ϕ ( wx+ w ) = c wx-α (6) ow f(x) ca be wrtte as f( x) = µϕ j j( w jx+ wj ). (7) j= The euro mplemetg equato (6) s stll too

3 complcated ad requres the kowledge of all the coeffcets c so t s advsable to choose a sub-optmal structure made of a smpler cubc sple. We have adopted a Catmull-Rom cubc sple whose cotrol pots are adapted together wth µ j w j ad the compoets of w j for all the dexes j by a backpropagato-lke learg algorthm. Ths soluto s a good compromse betwee the optmalty of the orgal model ad complexty; fact LUTs make the realzato of ths ew kd of euro ot much harder tha usg sgmods; besdes t allows to ft the data properly wthout losg the smoothess eeded to obta a good geeralzato. Fally we ll exted our approach to eural etworks wth more tha hdde euros also f ths tme a drect dervato from regularzato theory s ot possble: may occasos the target does t have a addtve structure ad the troducto of some extra dmesos mproves the qualty of the approxmato... Catmull-Rom mplemetato of the cubc actvato fucto As we have atcpated stead of workg o expresso (6) whch each of + tracts of the sple s charactersed by a dfferet lear combato of the coeffcets c the actvato fucto wll be composed of smpler cubc structures of local ature. The -th tract s expressed by equato (8) where u [] ad Q=(q x q y ). Such a sple terpolates the pots Q + (u=) ad Q + (u=) ad has a cotuous frst dervatve; the secod dervatve s ot cotuous oly at the kots. Equato (8) represets a fucto f x-coordates are ordered accordg to the rule q x < q x+ < q x+ < q x+3. The thrd degree equato F x (u)=x where x s the actvato of the euro gves the value of the local coordate u: f the cotrol pots are uformly spaced alog the x-axs (by a fxed step x) F x (u) reduces to a frst degree polyomal. 3 3 Q Fu ()= [ u u u ] 5 4 Q 3 + (8) Q+ Q + 3 Fx ( u)= u x + qx + It s worth-otg that fxg the values of q x reduces the total umber of free parameters ad the rsk of overfttg. The q y coordates are talzed by samplg a fucto lke the sgmod whch supples the eural etwork wth uversal approxmato capablty. Alog the x-axs P+ pots are take; outsde the samplg terval the euro s output wll be held costat at the values q y for the egatve x-coordate ad q yp- for the postve x. To evaluate the degree of smoothess of the Catmull-Rom sple-based actvato fucto t s possble to use expresso (); callg f j the actvato fucto of the j-th euro we ca approach the tegrato: d f j( sj) ds R j = ds j P 3 3 3( qjp + 3qjp + 3qjp + + qjp + 3) x p= 3( qjp 3qjp + 3qjp + qjp + 3) ( qjp 5qjp + + 4qjp + qjp + 3) ( qjp 5qjp + 4qjp + q jp+ 3) (9) + + where the dex y the q pots has bee omtted for smplcty. ow wth a proper choce of just oe parameter that s x the desred compromse betwee data fttg a smoothess ca be reached. Several expereces demostrated that the best values for x le the terval [.5.].

4 .3. Learg algorthm Eve f a sgle hdde layer s suffcet to obta uversal approxmato capabltes ad to create a smooth recostructo of the target accordg to regularzato theory for the sake of geeralty we ll descrbe a learg algorthm sutable for multlayer structures too. Let s beg by defg some symbols. If the actvty level of the k-th euro the l-th layer (l=...l ( l ad k=... l ) s s ) k the parameters of the local tract of the sple are sk P z k = + ak = x zk () uk = zk ak where a k s a dummy varable. The output of a ( l geerc euro s x ) k ; the case of a vector fucto the k-th compoet of the target s t k. The learg step p wll be sometmes omtted the followg descrpto of the backpropagato rules. The adaptato rates are µ w for the coecto weghts ad bas ad µ q for the ( l cotrol pots; c ) km (. ) s the m-th colum of the matrx (8). Table I - LEARIG ALGORITHM FOR l = L... FOR k =... l FOR j =... l tk xk l = M e l k = + ( l+ ) ( l+ ) δm wmk l = M... m= df l u ( ) l l ( ) () () ka k δk = ek du x u= uk ( l ) wkj = µδ w k xj wkj [ p+ ] = wkj [ p] + wkj [ p] FOR m =... 3 q = ka k + = µ e ( ) () e c u k a m q k l µ q k km k k ( ) q l k( ak ( l k( ak k( ak q [ p+ ] = q F [ p] + q [ p] ) k( ak ( ) 3. Smulato ad Dscusso 3.. Two-dmesoal test fucto I Table we summarze the results of a test o a two-dmesoal fucto: 5 pots are gve [-] [-]. gxy ( ) = s( π x) + 4( y 5. ) () We compared etworks wth oe hdde layer ad a lear euro the output: _x_ s used for etworks wth sgmods ad S_ x_ for sple actvato fuctos ( puts x hdde uts ad output). I the secod ad the thrd colum the reader ca fd the average geeralzato error ad ts varace respectvely; the the trag epochs are reported. Ad.Par. s the total umber of parameters updated durg the learg phase: t should be otced that the adaptato of the cotrol pots of the sples s local ad so oly those q y wll be moved that correspod to tracts whch the actvty levels belog to. Ths meas that oly a few of the 8 cotrol pots usually used to sample the sgmod are really updated ad ther umber ( for fxed shape actvato fuctos) plus the umber of weghts s Ad.Par It s clear that our archtecture has may advatages over etworks wth sgmods. Frst of all the geeralzato error s lower as expected from the theory; ts varace s lower too whch meas that the etwork s more relable. The umber of adaptable parameters Ad.Par. s dcated by may authors (see [] ad [5]) as oe of the key factors the choce of a model to reach a good geeralzato performace: the table shows that our etwork the value of Ad.Par s ot very dssmlar or s eve lower tha the oe requested whe sgmods are used. I other words ASs seem to make a better use of ther adaptve capabltes. Fally we obta such good results wth a shorter trag tme ad wth a more ecoomcal hardware structure terms of euros ad coectos.

5 Fgure shows a plot of the orgal fucto whle Fgure we show the fucto produced by a AS wth just two hdde uts: the good level of the approxmato s evdet. Table II - SIMULATIOS O FUCTIO () < e > et. ad # of g σ e Epochs Ad.Par. uts [db] g _8_ _5_ S SP-4_ phase a smlar umber of free parameters are adapted usg a close CPU tme for the ASs ad stadard-mlps famles etworks. The ASs are deoted as S56_x _7 whle stadard-mlp are deoted as 56_x _7 where x x represet the euros the hdde layer. Here order to have a close umber of adaptable parameters for both etworks famles the parameters x ad x assume the followg value x =[ ] ad x =[3 4 5]. For example the et S56_46_7 cotas 363 adaptable parameters whle the et 56_5_7 that s the closest terms of umber of free parameters cotas 37 adaptable parameters (see Table III for detals) Fgure PLOT OF FUCTIO () Characters recogto problem The trag set cossts 64 PC-CGA characters (from 33 to 96 of the ASCII code) defed as a 7x8 pxel matrx whle the target s the respectve 7 bt ASCII code. The smulatos cosst o the trag of dfferet sze of ASs ad stadard-mlps wth (about) the same umbers of adaptable parameters. Obvously at the ed of the trag Fgure APPROXIMATIO OF FUCTIO () OBTAIED WITH A TWO HIDDE EUROS AS We traed the etworks usg epochs of 64 characters each. Durg the learg phase the put patters are corrupted by vertg radomly the % of the pxels (salt & pepper ose). The percetage of salt & pepper ose s evaluated cosderg the etre epoch such that the verted pxels umber for each character ca be dfferet for each trag step. I order to have some statstcal formato 3 etworks for each sze are traed. I order to evaluate the geeralzato performace of the ASs a smple test have bee also carred out. The test cossts o the -

6 evaluato of the <MSE> ad the percetage of correct characters classfcato (ht-rate) presece of dfferet realzato ad percetage of salt & pepper ose. I partcular the etworks traed wth % of flpped pxel are tested wth 3% ad 5% of salt & pepper ose. I Table III the comparso performace are reported. Refereces [] Amar S. Murata.(993). Statstcal Theory of Learg Curves uder Etropc Loss Crtero. eural Computato [] Che C. T. Chag W. D. (996). A Feedforward eural etwork wth Fucto Shape Autotug. eural etworks [3] Gros F. Joes M. Poggo T. (995). Regularzato Theory ad eural etworks Archtectures eural Computato [4] Guarer S. Pazza F. Uc A. (995). Multlayer eural etworks wth Adaptve Sple-based Actvato Fuctos. I Proceedgs of the Iteratoal eural etwork Socety Aual Meetg WC 95 Washgto D.C. USA I695-I699. [5] Moody J. E. (99). The Effectve umber of Parameters: A Aalyss of Geeralzato ad Regularzato olear Learg Systems. I Advaces eural Iformato Processg Systems 4 J. E. Moody S. J. Haso ad R. P. Lppma edtors Morga Kauffma [6] Pazza F. Uc A. Zeob M. (99). Artfcal eural etworks wth Adaptve Polyomal Actvato Fucto. I Proc. of the IJC Bejg Cha [7] Pazza F. Uc A. Zeob M. (993). eural etworks wth dgtal LUT actvato fucto. I Proc. of the IJC agoya Japa [8] Poggo T. Gros F. (99a). etworks for Approxmato ad Learg. Proceedgs of the IEEE 78(9) [9] Wag J.. Lay S. R. Maecler M. Mart R. D. Schmert J. (994) Regresso Modelg Back-Porpagato Projecto Pursut Learg IEEE Tras. o eural etworks 5(3) Table III - CHARACTERS RECOGITIO GEERALIZATIO PERFORMACE. THE TERM <MSE> REPRESETS THE AVERAGE MEA SQUARE ERROR <H.R.> IS THE HIT-RATE AD THE TERM <W.P.> IS THE UMBER OF WROG PATTERS; AVERAGED OVER 3 ETWORKS FOR FORWARD EPOCHS FOR A TOTAL OF 64 TEST CHARACTERS PRESETED. THE ETWORKS SIZE IS CHOSE I ORDER TO MAITAI THE UMBER OF ADAPTABLE PARAMETERS AS CLOSE AS POSSIBLE. et. sze um. of adapt. param. Salt & pepper ose level 3% flpped pxels Salt & pepper ose level 5% flpped pxels x or x - <MSE> <H.R.> <W.P.> <MSE> <H. R.> <W.P.> [db] % [db] % S56_7_ _3_ S56_37_ _4_ S56_46_ _5_