Speeding Up Back-Propagation Neural Networks

Transcription

1 Prcdings f t 005 Infrming Scinc and IT Educatin Jint Cnfrnc Spding Up Back-Prpagatin Nural Ntwrks Mammd A. Otair Jrdan Univrsity f Scinc and Tcnlgy, Irbd, Jrdan tair@just.du.j Walid A. Salam Princss Summaya Univrsity fr Scinc and Tcnlgy, Amman, Jrdan walid@psut.du.j Abstract Tr ar many succssful applicatins f Backprpagatin (BP fr training multilayr nural ntwrks. Hwvr, it as many srtcmings. Larning ftn taks lng tim t cnvrg, and it may fall int lcal minima. On f t pssibl rmdis t scap frm lcal minima is by using a vry small larning rat, wic slws dwn t larning prcss. T prpsd algritm prsntd in tis study usd fr training dpnds n a multilayr nural ntwrk wit a vry small larning rat, spcially wn using a larg training st siz. It can b applid in a gnric mannr fr any ntwrk siz tat uss a backprpgatin algritm trug an ptical tim (sn tim. T papr dscribs t prpsd algritm, and w it can imprv t prfrmanc f backprpagatin (BP. T fasibility f prpsd algritm is swn trug ut numbr f xprimnts n diffrnt ntwrk arcitcturs. Kywrds: Nural Ntwrks, Backprpagatin, Mdifid backprpagatin, Nn-Linar functin, Optical Algritm. Intrductin T Backprpagatin (BP algritm (Rumlart, Hintn, & Williams, 1986; Rumlart, Durbin, Gldn, & Cauvin, 199 is praps t mst widly usd suprvisd training algritm fr multi-layrd fdfrward nural ntwrks. Hwvr, in sm cass, t standard Backprpagatin taks unndurabl tim t adapt t wigts btwn t units in t ntwrk t minimiz t man squard rrrs btwn t dsird utputs and t actual ntwrk utputs (Callan, 1999; Carling, 199; Frman, &, Skapura, 199; Hakin, 1999; Maurn, Tr as bn muc rsarc prpsd t imprv tis algritm; sm f tis rsarc was basd n t adaptiv larning paramtrs,.g. t Quickprp (Falman, 1988, t RPROP (Ridmillr, & Braun, 1993, dlta-bar-dlta rul (Jacbs, 1988, and Extndd dlta-bar-dlta rul (Minai, Cmbinatins f diffrnt tcniqus can ftn lad t an imprvmnt in glbal ptimizatin mtds (Hagan, 1996; L, Tis papr prsnts an ptical backprpagatin (OBP algritm, wit analysis f its bnfits. An OBP algritm is dsignd t vrcm sm f t prblms assciatd Matrial publisd as part f ts prcdings, itr n-lin r in print, is cpyrigtd by Infrming Scinc. Prmissin t mak wit standard BP training using nnlinar functin, wic applid n t digital r papr cpy f part r all f ts wrks fr prsnal r classrm us is grantd witut f prvidd tat t cpis ar nt mad r distributd fr prfit r cmmrcial advantag AND utput units. On f t imprtant aspcts f t prpsd algritm is its tat cpis 1 bar tis ntic in full and giv t full citatin n t first pag. It is prmissibl t abstract ts wrks s lng as crdit is givn. T cpy in all tr cass r t rpublis r t pst ability t scap frm lcal minima n a srvr r t rdistribut t lists rquirs spcific prmissin wit ig spd f cnvrgnc during t training prid. In rdr frm t publisr at Publisr@InfrmingScinc.rg t Flagstaff, Arizna, USA Jun 16-19

2 Spding Up Back-Prpagatin Nural Ntwrks valuat t prfrmanc f t prpsd algritm, simulatins ar carrid ut n diffrnt ntwrk arcitcturs, and t rsults ar cmpard wit rsults btaind frm standard BP. Tis papr is dividd int fur sctins. In sctin, w will intrduc t standard Backprpagatin (BP. In sctin 3, ur xtnsin f t nn-linar functin f t BP is dclard. In sctin 4, a cmparativ study will b dn vr diffrnt ntwrk arcitcturs. Finally, t cnclusins ar prsntd in sctin 5. Standard Backprpagatin (BP T Backprpagatin BP larns a prdfind st f utput xampl pairs by using a tw pas prpagat adapts cycl. As sn in Figur 1, aftr an input pattrn as bn applid as a stimulus t first layr f ntwrk units, it is prpagatd trug ac uppr layr until an utput is gnratd. Tis utput pattrn is tn cmpard t t dsird utput, and an rrr signal is cmputd fr ac utput unit. Figur 1 - T Tr-layr BP Arcitctur T signals ar tn transmittd backward frm t utput layr t ac unit in t intrmdiat layr tat cntributs dirctly t t utput. Hwvr, ac unit in t intrmdiat layr rcivs nly a prtin f t ttal rrr signal, basd rugly n t rlativ cntributin t unit mad t t riginal utput. Tis prcss rpats, layr by layr, until ac unit in t ntwrk as rcivd an rrr signal tat dscribs its rlativ cntributin t t ttal rrr. W fllw Frman and Skapura s bk (199 t dscrib t prcdur f training fdfrward nural ntwrks using t backprpagatin algritm. T dtaild frmulas ar dscribd in t Appndix. Optical Backprpagatin (OBP In tis sctin, t adjustmnt f t nw algritm OBP (Otair & Salam, 004 will b dscribd at wic it wuld imprv t prfrmanc f t BP algritm. T cnvrgnc spd f t larning prcss can b imprvd significantly by OBP trug adjusting t rrr, wic will b transmittd backward frm t utput layr t ac unit in t intrmdiat layr. In BP, t rrr at a singl utput unit is dfind as: δ pk = (Y pk O pk (1 Wr t subscript P rfrs t t p t training vctr, and K rfrs t t k t utput unit. In tis cas, Y pk is t dsird utput valu, and O pk is t actual utput frm k t unit, tn δ pk will prpagat backward t updat t utput-layr wigts and t iddn-layr wigts. Wil t rrr at a singl utput unit in adjustd OBP will b as: 168

3 Otair & Salam Nwδ pk = (1 + (, if ( Y pk O pk >= zr. Nwδ pk = (1 + (3, if ( Y pk O pk < zr. Wr Nwδ pk is cnsidrd as t nw prpsd in t OBP algritm. An OBP uss tw frms f Nwδ pk, bcaus t xp functin always rturns zr r psitiv valus (and t adapts pratin fr many utput units nd t dcras t actual utputs ratr tan incrasing it. Tis Nwδ pk will minimiz t rrrs f ac utput unit mr quickly tan t ld δ pk, and t wigts n crtain units cang vry larg frm tir starting valus. T stps f an OBP: 1. Apply t input xampl t t input units.. Calculat t nt-input valus t t iddn layr units. 3. Calculat t utputs frm t iddn layr. 4. Calculat t nt-input valus t t utput layr units. 5. Calculat t utputs frm t utput units. 6. Calculat t rrr trm fr t utput units, but rplac Nwδ pk wit δ pk (in all quatins in appndix. 7. Calculat t rrr trm fr t utput units, using Nwδ pk, als. 8. Updat wigts n t utput layr. 9. Updat wigts n t iddn layr. 10. Rpat stps frm stp 1 t stp 9 until t rrr (Y pk O pk is accptably small fr ac training vctr pairs. Prf: T utput f BP and OBP fr any utput unit must b qual, if t BP utput units multiply it by Factr (A, wr (A is dfind as fllws: 1 A = (1 + (4 A = (1 + (5 T sigmid functin fr ac utput unit in BP must b qual t t Nwδ pk in OBP trug multiplying it by tis Factr. Tr is antr way t find t factr (A using t fllwing assumptins: 169

4 Spding Up Back-Prpagatin Nural Ntwrks Assumptin 1: 1 1 A1 = 1 (6 + (1 + In tr wrds, OBP uss a sigmid functin n ac rrr f ac utput unit (Y pk O pk, and it assums tat if t sigmid functin (n t lft sid in quatin 6 is multiplid by (A1, it must b qual t ts sigmid functin wic applid n t rrr fr t utput units, tn: 1 + A1 = (7 (1 + Assumptin : An OBP assums tat if t sigmid functin wic is applid n t rrr f utput units - is multiplid by (A, tn it must b qual t Nwδ pk: 1 A = (1 + (8 A = (1 + (1 + (9 (1 + Assumptin 3: Frm quatins (7 and (9, OBP assumd tat: A = A1 * A (10 = (1 + (1 + (11 A = (1 + (1 + (1 Cmparativ Study T OBP will b tstd using t fllwing ral xampl tat cnsidr nural ntwrk wit 6 units fr input layr, 4 units fr iddn layr, and 3 units fr utput layr. Tis xampl will train t ntwrk using OBP, and tn it trains it using standard BP, and it cmpars t final rsults frm OBP and BP. Aftr t MSE (Man Squar Errr racd t t training prcss discntinud. T initial wigts slctd randmly frm 0.5 t +0.5, and t sam initial wigts av bn usd fr t tw algritms. Larning rat quals t 0.01 as bn takn in t training prcss Figur, sws t diffrncs btwn t final wigts frm input layr t t iddn layr using an OBP, and BP ar vry small. 170

5 Otair & Salam Als, tr ar n ug diffrncs btwn t tw algritms t adapt t wigts frm t iddn t t utput layr. Figur (3 sws tat: T surprising rsults in t prvius figur ar t numbr f pcs using t tw algritms, wic prvs tat t OBP is an ptical algritm if it is cmpard wit t standard BP. Tis ntwrk was tstd using t tw algritms using diffrnt larning rat, Tabl 1, sws t rsults. Frm Tabl1, it is bsrvd tat t rsults f OBP ar muc bttr and fastr tan t BP fr all training prcsss wit diffrnt larning rat. Tabl 1 Training prcsss using diffrnt (η Larning Rat Cnclusin Tis papr intrducd a nw algritm OBP, wic as bn prpsd fr t training f multilayr nural ntwrks, and it nancd t vrsin f t Backprpagatin BP algritm. T study sws tat OBP is bnficial in spding up t larning prcss. T simulatin rsults cnfirmd ts bsrvatins. Training prcss dfind as adapting wigts fr ac unit in nural ntwrk, s t OBP is a gd algritm, bcaus it can adapt all wigts wit ptical tim. T simulatin rsults sw tat wn a vry small valu is usd fr larning rat (η wit OBP maks t adaptd final wigts vry clsd bcm t t final wigts tat intrducd frm BP. S, it can scap frm lcal minim. Futur Wrk Futur wrk will tst t prpsd algritm acrss a wid rang f imprtant prblms and applicatins. (η OBP Epcs BP Epc 171

6 Spding Up Back-Prpagatin Nural Ntwrks Appndix: Training a Fd Frward Nural Ntwrks Using t Backprpagatin Algritm Assum tr ar m input units, n iddn units, and p utput units. 1. Apply t input vctr, X p =(X p1, X p, X p3,.., X pn t t t input units.. Calculat t nt- input valus t t iddn layr units: nt pj= ( N W ji X pi i = 1 3. Calculat t utputs frm t iddn layr: i pj = f j ( nt pj 4. Mv t t utput layr. Calculat t nt-input valus t ac unit: O nt pk= ( L W O kj i pj j = 1 5. Calculat t utputs: O pk = f j ( nt pk 6. Calculat t rrr trms fr t utput units: f k δ pk = ( Y pk O pk f k ( nt pk Wr, ( nt pk = f k ( nt pk (1 f k ( nt pk 7. Calculat t rrr trms fr t iddn units δ pj = f j M ( nt pj ( δ K = 1 pk W kj Ntic tat t rrr trms n t iddn units ar calculatd bfr t cnnctin wigts t t utput-layr units av bn updatd. 8. Updat wigts n t utput layr W kj( t+ 1 = W kj( t + ( η δ pk i pj 9. Updat wigts n t Hiddn layr W ji( t+ 1 = W ji( t + ( η δ pj X i Rfrncs Callan, R. (1999. T Essnc f Nural Ntwrks, Sutamptn Institut. Carling, A., (199. Back prpagatin. Intrducing Nural Ntwrks, Caudill, M. & Butlr, C. (1993.Undrstanding nural ntwrks. Cmputr Explratins, 1,

7 Otair & Salam Falman, S. E. (1988. Fastr-larning variatins n backprpagatin: An mpirical study. Prcdings f t 1988 Cnnctinist Mdls Summr Scl, Frman, J. A. &, Skapura, D. M. (199. Backprpagatin. Nural Ntwrks Algritm Applicatins and Prgramming Tcniqus, Hagan, M. T. & Dmut, H. (1996. Nural Ntwrks Dsign, Hakin, S. (1999. Nural ntwrks: A cmprnsiv fundatin ( nd d.. pp Jacbs, R. A. (1988. Incrasd rats f cnvrgnc trug larning rat adaptatin. Nural Ntwrks, 1, L, Y., O, S. H., & Kim, M.W. (1991. T ffct f initial wigts n prmatur saturatin in back prpagatin larning. Prcdings f t Intrnatinal Jint Cnfrnc n Nural Ntwrks, Sattl, Minai, A. A., & Williams, R. D (1990. Acclratin f back-prpagatin trug larning rat mmntum adaptatin. Prcdings f t Intrnatinal Jint Cnfrnc n Nural Ntwrks, Otair, M. A. & Salam, W. A. (004. An imprvd back-prpagatin nural ntwrks using a mdifid nn-linar functin. Prcdings f t IASTED Intrnatinal Cnfrnc, 004, Ridmillr, M & Braun, H. (1993. A dirct adaptiv mtd fr fastr backprpagatin larning t PROP algritm. Prcdings f t IEEE intrnatinal cnfrnc n Nural Ntwrks (ICNN, Vl. I, San Francisc, CA, Rumlart, D. E., Durbin, R. Gldn, R., & Cauvin, Y. (199. Backprpagatin: Trtical fundatins. In Y.Cauvin & D. E Rumlart (Eds., Backprpagatin and Cnnctinist Try. Lawrnc Erlbaum. Rumlart, D. E., Hintn, G. E., & Williams, R. J. (1986. Larning intrnal rprsntatins by rrr prpagatin. In D. E. Rumlart & J. L. McCllland (Eds., Paralll Distributd Prcssing, pp Bigrapis Mammd A. Otair is an Instructr f cmputr infrmatin systms, at t Jrdan Univrsity f Scinc and Tcnlgy, Irbd- Jrdan. H rcivd is B.Sc. in Cmputr Scinc frm IU-Jrdan, and is M.Sc. and P.D in 000, 004, rspctivly, frm t Dpartmnt f Cmputr Infrmatin Sysms-Arab Acadmy. His majr intrsts ar Macin Larning, Nural Ntwrk Larning Paradigms, Wb-cmputing, E-Larning. Walid A. Salam is an Assciat Prfssr f cmputr Scinc, at t PSUT-RSS, Amman-Jrdan. Hrcivd is B.Sc. in Cmputr Scinc frm YU-Jrdan, and is M.Sc. and P.D in 1987, 1991, rspctivly, frm t Dpartmnt f Cmputr Enginring-METU. His majr intrsts ar Macin Larning, Nural Ntwrk Larning Paradigms, and Intllignt Larning Paradigm trug t Wb (Intllignt Wb-larning basd Paradigms. 173