A Sensitivity-Based Adaptive Architecture Pruning Algorithm for Madalines

Transcription

1 Advaced Scece ad echology etters, pp A Sestvty-Based Adaptve Archtecture Prug Algorthm for Madales Sa J, Pg Yag, Shumg Zhog, J Wag, Jeog-Uk Km Jagsu Egeerg Ceter of Network Motorg, Nag Uversty of Iformato Scece ad echology, Nag 0044,Cha; Departmet of Eergy Grd, Sagmyug Uversty, Seoul 0-743, Korea Abstract. I ths paper, we proposed a ew sestvty-based adaptve archtecture prug algorthm for Madales. he algorthm establshes a prug measure based o the etwork sestvty to ts structure varato ad a mmal dsturbace prcple. he measure ca be used to evaluate the performace loss due to ts structure chages more or less. Ad the loss ca be compesated by relearg. hus, the ew adaptve prug mechasm s developed wth measurg, prug, ad compesatg. Keywords: Adale; Madales; sestvty; etwork costructo; prug algorthm Itroducto I terms of archtecture s fucto, eural etwork realzes a put-output mappg that based o the certa archtecture ad gve weght. herefore, the etwork costructo ad weght settg of eural etwork are hot topcs the research of eural etwork, ad have draw more ad more researchers atteto [, ]. However, what s a proper archtecture of a eural etwork for solvg a gve problem? he aswer to ths questo s ot easy. Accordg to some achevemets [3~5] of costructo, there are some broad upper lmt archtectures, though those are ot very sgfcat to etwork costructo. As a certa problem, f the etwork archtecture s too small, t may cost less both mplemetato ad computato, whle t may lear very slow or be ot able to lear at all. Whe the etwork archtecture s too bg, t may be traed quckly ad ft trag data accurately, whle t may cost more mplemetato ad computato ad have bad performace geeralzato [6]. Hece, the am of etwork costructo s to fd a etwork wth small but reasoable archtecture ad be able to lear well. hs paper measures the performace loss based o the exstg theory of Madales sestvty computato. By calculatg the varato o output caused by archtecture prug for Madales, whch s more substatve ad ca further coduct euros prug the etwork. he ma cotrbuto of ths paper s desged a ISSN: AS Copyrght 06 SERSC

2 Advaced Scece ad echology etters adaptve archtecture prug algorthm for Madales based o sestvty. he algorthm ca fd the target etwork wth smaller archtecture adaptvely, ad optmze the costructo of Madales ad reduce the computato of the etwork ad the cost of hardware mplemetato effectvely. he prug algorthm has the followg characterstcs ad ovatos: () computg the varato o output of etwork caused by archtecture chagg drectly, the results are based o the whole put space whch s more practcal; () creatg the sestvty measure follows the dea of mmal dsturbace prcple, whch s more reasoable; (3) establshg the compesato mechasm to reduce the performace loss of etwork caused by prug etwork s archtecture, whch ehaces the effectveess. Prelmares A. Madales model Fg.. Adale model Fg.. Madales model Copyrght 06 SERSC 85

3 Advaced Scece ad echology etters Fg. s a Adale model. Where,,,, W w, w,, w R y s the output, ad f X x x x s the put, ad s the weght correspod to ts put, w0, g s ts actvato fucto. Rs a bas, ad For coveece, ths paper, bas w s see as a elemet of Adale s weght, 0 ad let ts correspodg put elemet as x. I ths way, the weght ad put 0 of Adale ca be further wrtte as W w, w, w,, w R ad X x0, x, x,, x,. 0 Fg. s a Madale model. Seeg from frot to back, a Madale s cossted of put layer, hdde layers ad output layer. For dscusso, we let 0... represets a Madale wth certa archtecture. Where l ( l ) ot oly stads for a layer but also dcates the umber of Adales the layer, whle ad 0 s a excepto, t represets the dmeso of Madale s put, represets the output layer. B. Madales sestvty Defto. For a Adale s put s X,, the gve weght sw R, ad the correspodg varato o weght s W R, the sestvty s defed as the probablty of the Adale s output verso due to the varato o weght for all put pots, whch ca be expressed as Vvar sw, X P f W X f W W X X, () V where V s the umber of all put pots, adv var s the umber of the put pots that cause the Adale s output chaged due to the varato o weght. Defto. For a Madale s put s, X ad the correspodg varato o weght s, the gve weght sw l l R, l l W R, where l 0, l, the sestvty of a layer s a vector ad defed as all the euros sestvty ths layer, whch s expressed as l l l l S s, s, s l () Defto 3. he sestvty of a Madale s the sestvty of the output layer s sestvty, whch s expressed as S S s, s, s (3) et 86 Copyrght 06 SERSC

4 Advaced Scece ad echology etters C. Sestvty-Based Adaptve earg Rules Dfferet from the well-kow BP algorthm [7], the learg rules o Madale are ot so mature, ad ths also oe of the motvatos of ths study. SBAR (Sestvty-Based Adaptve earg Rules) [8] s a weght adaptve learg algorthm for Madales proposed based o perceptro learg rules [9]. he algorthm maly follows three prcples: mmal dsturbace prcple, the beeft prcple ad the task allocato prcple, ad three learg rules are desged based o the three prcples. 3 Sestvty of Madales Based o Archtecture A. Sestvty Defto Based o Archtecture Defto 4. For a Madale, t s a etwork wth the archtecture 0 of, f the th ( ) Adale the th (l ) layer whch marked as ode s prued, the sestvty of Madale s archtecture s defed as for ay put pots, the probablty of the Madale s output verso s ode. because of the archtecture prug, ad expressed as et Accordg to formula (4), set ode ca be expressed as: s ode S s, s, s. (4) et It s obvously that the sestvty of Madales expressed formula (4) s a vector, for coveece, formula (4) ca be further quatfed as the followg: S where s s V s et V, (5) represets the th Adale s sestvty of the output layer, ad V s the umber of all put pots. B. Sestvty Computato based o Archtecture I a Madale, f the th ( ) Adale hdde layer was prued, the all the Adales output layer would loss the th put elemet, k x k, whch would make the correspodg weght equals to 0, ad the whole result of W k k X chages from 0 w x k, k, to 0 wk, xk,, whch meas, Copyrght 06 SERSC 87

5 Advaced Scece ad echology etters pruethe thadale wk, xk, 0,,. hddelayer wk x k (6) 0 Sce there s 0wk, xk, 0wk, xk, w w,, x, (7) k k k, where, k. From formula (7), t ca be see that f prug the th Adale hdde layer, all the Adales output layer wll loss the th elemet of put. It ca make a equvalet trasform that the varato o weght cause the weght equals to 0, therefore there s w w. k, k, I the stuato of prug the th Adale hdde layer, the sestvty computato of ay Adale output ca trasform to a dsturbace weght, whch ames: 0 sk ode sk Wk, wk,, (8) wk, where w, 0 k s the th elemet of. All the Adales sestvty output layer caused by prug the th Adale hdde layer ca fgure out, ad the lke formula (5), the sestvty of the Madale ca fgure out too, that s s ode s. (9) W k et I fact, the sestvty computato formulae of the Madale prug a Adale derved above ca also be geeralzed to prue Adales. 4 Prug Measure Based o Sestvty o seek a target etwork s the drect motvato of explorg etwork archtecture prug, as well as the target etwork ca meet the task learg wth smplfed archtecture by prug Adales hdde layer. However, amog all the euros hdde layer, whch oe should be prued ad what s the bass for selecto must be gve reasoable aswers whe desg of prug algorthm. For a traed etwork, t s obvously that prug Adales hdde layer wll cause varato o the output of the Madale, t wll also chage the performace of the exstg etwork. However, how to measure the performace qualtatvely. It s dffcult to measure the learg ad geeralzato of the etwork gettg better or worse, though, quattatve calculato for ths chage the Madale s feasble. Accordg to the defto of sestvty ad the result of formula (9), sestvty ca more drectly reflect the degree of varato o Madale s output caused by prug Madale s archtecture, as the followg etoutput chage s ode (0) et 88 Copyrght 06 SERSC

6 Advaced Scece ad echology etters Where ode represets prug the th ( ) Adale th ( ) the layer. Sce a successful prug approach s the oe that ca make the prued Madale recover the covergece to the state before prug, therefore, selectg whch Adale to prue should be aalyzed from the vew of learg. he exstg Madales learg mechasms all follow a mportat prcple, whch s amely mmal dsturbace prcple [8, 0]. Upo trag a Madale, t should meet reducg the output error or ted to reduce the output error for the curret trag sample as well as the weght adustmet should reduce break the mappg relatoshp establshed by other trag samples. Itutvely, the smaller varato o a Madale s output, the easer to recover to the orgal performace by relearg; whle the greater varato o a Madale s output, the more dffcult to recover to the orgal performace by relearg. he fluece of the etwork s output ad performace caused by archtecture prug s ot what we wat, whle t s obectve exstece. Hece, we hope the varato o output as small as possble. Based o the above dscusso, the Adale prug approach ca be descrbed as: amog the multple Adales the hdde layer, prug whch oe should be based o the rule that the smaller varato o the Madale s output after prug should be gve prorty. he selecto strategy further evolved to prug selecto measure, whch ca be deoted as: prug selecto measure m s ode, s ode,, s ode () et et et 5 Expermetal verfcatos hs secto presets some expermets whch maly verfy two pots. Oe s to verfy the reasoable of the prug measure ad strategy; aother s to verfy the effectveess of the sestvty-based adaptve archtecture prug algorthm for Madales. Ad these expermets are based o some publc data sets. I order to verfy the reasoable of the prug measure ad strategy, the tal Madale wth the archtecture of 7-8- was selected, as well as the target Madale should wth the archtecture of I the expermets, the problem of the 7 bt party (oe party problem) was solved. I order to verfy the effectveess of the sestvty-based adaptve archtecture prug algorthm for Madales, the problems lke AND-XOR (A logcal operato problem), MONKS- ad BANCE-SCAE from the UCI repostory are solved wth SBAR algorthm ad prug algorthm ths paper. I order to guaratee the feasble of the expermets ad the valdty of expermetal results, the umber of trag teratos s 00, 000 ad 60000, respectvely, ad three tmes for cotuous prug a Adale are allowed. Each result comes from the average of 00 rus. Smulato expermets were carred out ad the expermetal results are show able ad table. Copyrght 06 SERSC 89

7 Advaced Scece ad echology etters able. he effcecy of etwork retrag uder dfferet hdde layer odes Ital Madale No. of Adale hdde layer sestvty he tmes of ode adustmet after relearg able. Comparso of the effectveess ad effcecy of the etwork trag wth the prug algorthm ad SBAR Data set he prug algorthm ths paper SBAR IM M CR Iterato Adustmets M CR Iterato Adustmets AND-XOR MONKS BAANCE SCAE *IM: Ital Madale; M: arget Madale; CR: covergece rate %. Ackowledgemet. hs work was supported by the Idustral Strategc echology Developmet Program (004740) fuded by the Mstry of rade, Idustry ad Eergy (MOIE) Korea ad the PAPD. Prof. Jeog-Uk Km s the correspodg author. Refereces. Reed, R.: Prug algorthms-a survey [J]. IEEE rasactos o Neural Networks, 993, 4(5): Augasta, M.G., Kathrvalava, K., Prug,., Algorthms of eural etworks-a comparatve study [J]. Ope Computer Scece, 03, 3(3): Zhag, Z., Ma, X., Yag, Y.: Bouds o the umber of hdde euros three-layer bary eural etworks [J]. Neural Networks, 003, 6 (7): Copyrght 06 SERSC

8 Advaced Scece ad echology etters 4. Cho, B., ee, J. H., Km, D. H.: Solvg local mma problem wth large umber of hdde odes o two-layered feed-forward artfcal eural etworks [J]. Neurocomputg, 008, 7(6): Puma-Vllaueva, W. J., Satos, E. P. D., Zube, F. J. V. A.: costructve algorthm to sythesze arbtrarly coected feedforward eural etworks [J]. Neurocomputg, 0, 75(): Wag, X., Shao, Q., Mao, Q.: Archtecture selecto for etworks traed wth extreme learg mache usg localzed geeralzato error model [J]. Neurocomputg, 03, 0(): Rumelhart, D., Mcclellad, J.: Parallel Dstrbuted Processg: Exploratos the Mcrostructure of Cogto: Foudatos [M]. MI Press, Zhog, S., Zeg, X., Wu, S.: Sestvty-Based Adaptve earg Rules for Bary Feedforward Neural Networks [J]. IEEE rasactos o Neural Networks ad earg Systems, 0, 3(3): Roseblatt, F.: O the covergece of reforcemet procedures smple perceptros [C]// ech. Rep. VG-96-G-4, Corell Aeroautcal aboratores Yeug, D. S., Cha, P. P. K., Ng, W. W. Y.: Radal Bass Fucto etwork learg usg localzed geeralzato error boud [J]. Iformato Sceces, 009, 79(9): Copyrght 06 SERSC 9