CHEBYSHEV POLYNOMIAL APPROXIMATION FOR ACTIVATION SIGMOID FUNCTION

Transcription

1 CHEBYSHEV POLYNOMIAL APPROXIMATION FOR ACTIVATION SIGMOID FUNCTION Miroslav Vlče Abstract: A alterative polyomial approximatio for the activatio sigmoid fuctio is developed here. It ca cosiderably simplify the iput/output operatios of a eural etwor. The recursive algorithm is foud for Chebyshev expasio of all costitutig polyomials. Key words: Sigmoid fuctio, Chebyshev polyomials, recursive algorithms Received: Jauary 3, Revised ad accepted: August 3,. Itroductio We assume a sigle eural etwor cosistig of a distict umber of iput odes ad oe output ode [6]. I order to evaluate the output we have to apply the stadard activatio sigmoid fuctio σy to the sum y N i w ix i, where x i are the values computed by the ode s predecessor, ad w i are the weights of the correspodig edges. As the activatio sigmoid fuctio σy of a eural etwor satisfies the Riema itegrable coditio, it ca be approximated by the Chebyshev series. We preset the recursive algorithm [8] for Chebyshev approximatio of the activatio sigmoid fuctio ad its atural geeralizatio to a multiple umber of iputs. These results ca be applied i several eural etwors[5], such as multilayer perceptro MLP, wavelet etwors, radial basis fuctio etwors RBFN, piecewise smooth etwors PWSN, the time delay iput multilayer perceptro, geeral regressio eural etwors GRNN, recurret eural etwors, ad the uified model UM. A commo activatio sigmoid fuctio σy is usually represeted through σy ey e y e y tahy, +e y σy +e y. Miroslav Vlce Czech Techical Uiversity i Prague, Faculty of Trasportatio Scieces, Departmet of Applied Mathematics, vlce@fd.cvut.cz c ICS AS CR 387

2 Neural Networ World 4/, There are differet approaches for evaluatig these fuctios whe usig digital implemetatios, such as a trucated series expasio [5], loo-up tables, or liear piecewise approximatio [3], [4] σy tahy y y3 3 + y5 5, 3 σy +e y c y +c. 4 I the followig sectios we preset a efficiet algorithm for coefficiets a for Chebyshev represetatio of the activatio sigma fuctio σy at w, 5 ad develop a explicit expasio of σx+y i biliear form employig Chebyshev polyomials.. Sigmoid Fuctio ad Polyomial Represetatio The Berstei basis fuctios of degree o t, are defied i [] by b, t t t. 6 By simple trasformatio w t we obtai the basis fuctios o iterval w, with b, w +w w. 7 I [8] we have recogized that the itegral of a ormalized Berstei basis fuctio C p,q w p+q + p+q dw q q p +w w 8 gives the maximally flat step fuctio i the form q+ p +w µ+q C p,q w µ µ µ w. 9 We have also derived the differetial equatio w C p,q w+[p q +p+qw]c p,q w 388

3 Vlče M.: Chebyshev polyomial approximatio for activatio sigmoid fuctio from which the expasio of C p,q w i Chebyshev polyomials follows C p,q w at w, d where N p + q +. As dw T w U w we ca write the Chebyshev expasio of the first derivative d dw C p,qw d dw C p,qw αu w. A maximally flat step fuctio is a cotiuous fuctio with all cosecutive derivagive p ad q, N p+q + iitializatio αn p p+q N αn q pαn body for µ N to ed loop o µ N + µ p+q q αµ q pαµ N ++µ αµ+ for µ N to ed loop o µ aµ αµ µ a a Tab. I Recursive algorithm for the evaluatio of coefficiets α ad a. tives, ad it ca be easily idetified as a sigmoid fuctio q+ p µ +w µ+q w σw C p,q w µ µ at w. 3 This sigmoid fuctio is cofied to the iterval,. The study of a stadard represetatio of the sigmoid fuctio is typically limited to the rage 8,8 [3]. We ca compare our defiitio with the stadard sigmoid fuctio σw +e 8w reduced to this iterval ad coclude that the step fuctio C,w hasequivalet propertieswith the differece w C, w σw of%. Usig differet values for p,q, we ca move the origi of the switchig process alog the w axis see Fig

4 Neural Networ World 4/, σ w d dw Cp,qw σw C p,qw Fig. Step fuctio C 8,8 w ad its derivative related to a sigmoid fuctio..8 σw C,w w Fig. Step fuctio C, w, sigmoid fuctio σw differece w. +e 8w, ad their.8.6 C 6,4w Fig. 3 Step fuctios C 4,6 wc 6,4 w,c 8, w,c, w,c,8 w,c 4,6 w, C 6,4 w. 39

5 Vlče M.: Chebyshev polyomial approximatio for activatio sigmoid fuctio 3. Iput/Output Operatio of a Simple Neural Networ For a eural etwor it is importat to represet cosistetly the additio of argumets w x + y. We have developed additio ad multiplicatio theorems for Chebyshev polyomials [7] i alterative forms T x+y l C xt ly, 4 where C x are ultraspherical polyomials []. Formula 4 provides a decompositio of the argumet w x+y betwee two types of orthogoal polyomials, Chebyshev ad ultraspherical. I order to mae the polyomial represetatio uiform we ca assume eq. 4 i the form T x+y a,l T xt l y 5 ad use the stadard recursive formula for Chebyshev polyomials T + x+y+t x+y x+yt x+y. 6 Formula 6 is used to develop a algorithm for the matrix a,l x+yt x+yxt x+y+yt x+y a,l xt xt l y+ + a,l T xyt l y a,l T + x+t xt l y 7 + a,l T xt l+ y+t l y a,l T + xt l y+ a,l T xt l+ y+ a,l T xt l y a,l T xt l y. By replacig the summatio idexes i a followig way p +,q l, p,q l, p,q l+ ad p,q l, we obtai a ew set of equatios + x+yt x+y p q + a p,q T p xt q y+ + a p,q T p xt q y+ p q p q a p+,q T p xt q y p q a p,q+ T p xt q y. 39

6 Neural Networ World 4/, Comparig the coefficiets with the same degree of polyomials T p xt q y, we arrive at a compact formula for the matrix a p,q a p,q ++a p,q +δ p, a p,q +a p+,q ++δ,q a p,q +a p,q+. 8 The Kroecer delta δ p, appears here due to the fact that T x T x ad it cotributes to the idex p twice. The algorithm produces computatio of T x+y as a biliear form, for example T T x T y T x T y T 6 x+y T 3 x 3 4 T 3 y, 9 T 4 x 3 3 T 4 y T 5 x T 5 y T 6 x T 6 y where v T deotes traspositio of vector v. It is worth otig that the off-diagoal cotais the biomial coefficiets. I this example 6 ad they are,, 3, ad Coclusio We have developed a polyomial approximatio for the activatio sigmoid fuctio. The algorithm is based o Chebyshev expasio for all costitutig polyomials. Combiig equatios 3 ad 5, we obtai σx+y at x+y a a,l T xt l y. The mai advatage of represetatio over the stadard sigmoid activatio fuctio σy cosists of decouplig of the oliear switchig process, which is ow hidde i coefficiets a, ad the weightig of the various iputs, embedded i coefficiets a,l. This approach eables us to state, that a major objective of future research will cocer of fidig the uiform ad fiite iput/output operatios for a forward eural etwor as a geeralizatio of a biliear form σw x +w x T x A,l T l x. We have also arrived at a rather compact form for the derivative of the sigmoid fuctio which mimics eq. d dw σw wx+y αu x+y α α,l U xu l y. 39

7 Vlče M.: Chebyshev polyomial approximatio for activatio sigmoid fuctio Acowledgemet The author wishes to tha the Grat Agecy of the Czech Republic for supportig this research through project GAP//795: Novel selective trasforms for ostatioary sigal processig. Refereces [] Abramowitz M., Stegu I.: Hadboo of Mathematical Fuctios, Dover Publicatios, New Yor Ic., 97. [] Meiardus G.: Approximatio vo Futioe ud ihre umerische Behadlug, Spriger- Verlag, Berli, Goettige, Heidelberg, New Yor, 964. [3] Armato A., Faucci L., Pioggia G., De Rossi D.: Low-error approximatio of artificial euro sigmoid fuctio ad its derivative, Electroic Letters, 45,, October 9, pp [4] Basterretxea K., Tarela J. M., del Campo I.: Approximatio of sigmoid fuctio ad the derivative for hardware implemetatio of artificial euros, IEE Proc.-Circuits Devices Syst., 5,, February 4, pp [5] Lee T. T., Jeg J. T.: The Chebyshev-Polyomial-Based-Uified Model Neural Networ for Fuctio Approximatio, IEEE Trasactios o Systems, Ma, ad Cyberetics, 8, 6, December 998, pp [6] Duch W., Jaowsi N.: Survey of Neural Trasfer Fuctios, Neural Computig Surveys,, 999, pp. 63. [7] Vlce M., Hoa P. B.: McClella Trasformatio through the Additio ad Multiplicatio Theorem for Chebyshev Polyomials, Proc. MICROCOLL 86, Budapest 986, pp [8] Vlce M., Zahradi P., Ubehaue R.: Aalytic Desig of FIR Filters, IEEE Trasactios o Sigal Processig, 48, 9, September, pp