New activation functions for complex-valued neural network

Transcription

1 Iteratioal Joural of the Physical Scieces Vol. 6(7), pp , 4 April, 0 Available olie at DOI: /IJPS.05 ISSN Acadeic Jourals Full Legth Research Paper New activatio fuctios for coplex-valued eural etwork Haid A. Jalab * ad Rabha W. Ibrahi Departet of Coputer Syste ad Techology, Faculty of Coputer Sciece ad Iforatio Techology, Uiversity Malaya, Kuala Lupur, Malaysia. School of Matheatical Scieces, Faculty of Sciece ad Techology, Uiversiti Kebagsaa Malaysia, Bagi 43600, Selagor Darul Ehsa, Malaysia. Accepted 4 March, 0 This paper presets a ew types of coplex-valued sigoid fuctio for a fully ulti-layered coplexvalued eural etwork (CVNN). By usig the cocept of the subordiatio betwee aalytic fuctios i ope disc, we able to study the reducibility of CVNN. A real-world proble exaple has bee used as a classifier. The siulatios results reveal that the proposed fully coplex-valued etwork, bee better traied reduces the testig tie by 54% copared to the choice of usig the traditioal sigoid activatio fuctio. Key words: Coplex valued eural etwork, activatio fuctios, reducibility, irreducibility, subordiatio. INTRODUCTION The study o theory ad applicatios of artificial eural etwork had icreased because of their outstadig capability of fittig oliear odels. Neural etwork had successfully bee applied across a extraordiary rage of proble doais, i areas as diverse as fiace, edicie, egieerig, geology ad physics due to their strog capacity to hadle coplex probles ad to iprove syste perforace (Subraaia et al., 00; Taqa ad Jalab, 00). Artificial eural etwork is a atheatical odel which eulates the activity of biological eural etworks i the hua brai. Each euro i the ANN (Artificial eural etwork) has a uber of iputs ad oe output (Sivaada, 006). The coplex valued eural etwork are those eural etwork whose weights, threshold values, iput, output sigals all are coplex ubers ad the activatio fuctio ad its derivatives have to be well behaved every where i the coplex plae (Ki ad Adali, 00). However, the coplex valued eural etwork is extedig its field both i theories ad applicatios. They are used to express real-world pheoea like tie series aalysis, sigal aplitude ad phase, ad to aalyze various atheatical ad geoetrical *Correspodig author. E-ail: haidalab@u.edu.y relatioships. Also the coplex valued eural etwork (CVNN) had show ore powerful capability tha realvalued eural etwork i processig real-valued sigals (Nitta, 004a). I coplex-valued eural etwork, oe of the ai probles is selectig of odes activatio fuctio (Ki ad Adali, 00). I real case, the ode activatio fuctio is usually chose to be a cotiuous, bouded ad ocostat fuctio. These coditios o the activatio fuctio are very ild ad there is o proble i selectig a real fuctio that satisfies these requireets ad that is also sooth (derivative exists). I CVNN, ay regular aalytic fuctio caot be bouded uless it reduces to a costat. This is kow as the Liouvilles s theore. I coplex case, the ai costraits that the activatio fuctio should satisfy ca be foud i literatures (Georgi ad Koutsougeras, 00; Hayki, 008; Gaesh ad Balasubraaia, 009). Nitta (004b) studied the reducibility of ultilayer coplex-valued eural etwork, i which the reducibility is expressed by π rotatio equivalece istead of sig equivalece which is a extesio to Sussa s work of the real-valued eural etwork. Sussa (99) preseted ecessary ad sufficiet coditios to reduce the uber of hidde euros for

2 Jalab ad Ibrahi 767 real-valued eural etworks, ad usig the iportat otio reducibility/ irreducibility of the real valued eural etwork he devised. He proved that the 3-layered realvalued eural etwork was uiquely deteried by its iput-output ap, up to a obvious fiite group, provided that the real-valued eural etwork was irreducible (Uiqueess Theore). Thus, the reducibility is closely related to the redudacy of the real-valued eural etwork, ad is eeded for provig the uiqueess theore. The uiqueess theore is iportat for ivestigatig the properties based o the hierarchical structure of the real-valued eural etwork. COMPLEX-VALUED ACTIVATION FUNCTIONS Coplex plae is two diesioal with respect to real ubers ad is oe diesioal with respect to coplex uber. The coplex ubers have a agitude associated with the ad a phase that locates the coplex uber uiquely o the plae. Here, we cosider the proposed activatio fuctio which aps coplex-values ito coplex ad has the for of F : C C Figure. Activatio fuctio F ad G. F ( z ) = f ( x ) + if ( y ), ( z C ) R R () where i geeral tah ( a ) f ( a ) = ( a 3) e a This arrageet esures that the agitude of real ad iagiary part of F ( is bouded betwee - ad. But ow the fuctio F ( is o loger holoorphic, because the Cauchy-Riea equatio does ot hold (Gooda, 983), that is, F ( z ) F ( z ) + i 0. x y So, effectively, the holoorphy is coproised for boudedess of the activatio fuctio. Our cosideratio of F ( is held betwee the iput layer ad the hidde layer while the fuctio G ( = tah( is cosidered betwee the hidde ad the output (Figures ad ). Here, we cosider the followig 3-layers coplexvalued euros, there is oe hidde layer betwee the iput ad output layers. The iput sigals, weights, thresholds ad output sigals are all coplex ubers. The et iput defied as: U = W X + T, U to a coplex-valued euro, is () Figure. Subordiatio betwee F ad G where W is the (coplex-valued) weight coectig the coplex-valued euros ad, T is the (coplex valued) threshold value of the coplex-valued euro, ad X is the (coplex-valued) iput sigal fro the coplex-valued euro. To obtai the (coplex-valued) output sigal, covert the et output U ito its real ad iagiary parts as follows: U = x + iy = z. The (coplex-valued) output sigal of the hidde ad the output euros are defied as respectively. Σ tah ( x ) tah ( y ) ( z ) = + i, (3) x y ( x 3) e ( y 3) e σ ( z ) = tah( x) + i tah( y) (4) Assue that w i C is the weight betwee the iput euro i ad the hidde euro ; c C the weight

3 768 It. J. Phys. Sci. betwee the hidde euro ad the output euro; s ( deote the output values of the euro ; ( deotes the output euro for the iput patter t z = [ z,..., ], ad let v ( ad u k ( deote the et z iputs to the hidde euro ad the output euro for the iput patter z C, respectively. That is v = wi zi + t, where t is the threshold i= of the hidde euro, u ( = c s ( + c, k g k = k where c k is the threshold of the output euro, s ( = ( v ( ) g k ( = σ ( u( z Deoted by Σ ad )). N the set of all coplex-valued eural,, etwork described previously is the obect of this work. To illustrate our ai results, we eed the followig cocept. Give two fuctios, F ( ad G (, which are aalytic i ope disc, the fuctio F ( is said to be subordiate to G ( deoted by F( if there exists a fuctio h (, aalytic i ope disc with h (0) = 0 ad h ( < such that F ( = h( ) (Miller ad Mocau, 000). More applicatios of this cocept ca be foud i Ibrahi ad Darus (008). REDUCIBILITY OF THE CVNN Here, we show the reducibility of the coplex-valued eural etwork described i Coplex-valued activatio fuctios. First, we eed the followig preliiaries i the sequel (Nitta, 004b):. For a fixed, two coplex-valued eural etwork N N, ad N N, are called I O equivalet if their correspodig coplex-valued fuctios are the sae. It is ot essetial that the uber of euros or paraeters i the layers are equal.. Two coplex-valued liear affie fuctios α : C C ad β : C C are called π/ rotatioequivalet if oe of the followig coditios holds: (5) zc ; α( = β ( i0]. β ( ) zc ; α( = β ( iπ ]. β ( ) (6) π zc ; α ( = iβ ( i ]. β ( ) (7) 3π zc ; α ( = iβ ( i ]. β ( ) (8) 3. A coplex-valued eural etwork N N, is called reducible if oe of the followig three coditios holds: a) Oe of the weights betwee the hidde layer ad the output euro is zero: ; c = 0. b) There exist two hidde euros such that the et iputs to the are π/ rotatio-equivalet: ; ; v ad v are π/ rotatioequivalet. c) There exists a hidde euro such that the et iput to it is a costat: ; v is a costat. I the ext result, we show how a 3-layered coplexvalued eural etwork preserves the reducibility for sadwich (subordiatio ad superordiatio) sigoid activatio fuctios, that is, F(. Theore If a 3-layered coplex-valued eural etwork N N, of two subordiatio activatio fuctios ( F( ) is reducible, the it is I-O equivalet to aother 3-layered coplex-valued eural etwork with the activatio fuctio G ( ad fewer hidde euros. Proof Assue that a three-layered coplex-valued eural etwork is reducible. N N, Case I: Cosider that such that c = 0. I this case by the subordiatio of F ad G, that is, F (0) = 0). This iplies that the hidde euro does ot have effect o the output of the coplex-valued eural etwork, the hidde euro ca be deleted. Hece their correspodig coplex-valued fuctios are the sae. Case II: Let such that, π/ rotatio equivalet. Let v ad v are c be the weight betwee the hidde euro ad the output euro, ad c the weight betwee the hidde euro ad the output euro. I this case, by the subordiatio of F ad G,that is, the iage of F is a proper subset of the iage of G ( F( U ) U )), where U is a ope disc i C, we obtai the sae iput-output ap by reovig the

4 Jalab ad Ibrahi 769 Figure 3. Three-layer feed-forward CVNN ipleetig the back propagatio algorith. hidde euro ad chagig the weight c +, where r {,, i, i}. rc c to Thus their correspodig coplex-valued fuctios are still the sae. Case III: Fially, if for ; v δ, where δ is a costat. The sae iput-output ap ca be obtaied by reovigthe hidde euro ad chagig the threshold of the output euro c to c + s = c + Σ ( v ( z )) = c + Σ ( δ ) because the output Σ( v ( ) of the hidde euro is a costat which leads that σ ( u( ) is a costat uder the subordiatio betwee Σ ad σ, that is, their correspodig coplexvalued fuctios reai the sae. This copletes the proof. Corollary N N, If a 3-layered coplex-valued eural etwork is reducible, the it is I-O equivalet to aother 3-layered coplex-valued eural etwork with fewer hidde euros (Nitta, 004b). Corollary N N, If a 3-layered coplex-valued eural etwork w = 0 has weight i, the it is I-O equivalet to aother 3-layered coplex-valued eural etwork with fewer hidde euros. IMPLEMENTATION The eural etwork used i the siulatio process is a 3- layer feed-forward CVNN ipleetig the back propagatio algorith as show i Figure 3. I this etwork, all the iputs, outputs, weights, ad biases are coplex values. To ipleet our CVNN, we used Matlab R009b o Itel(R) Core TMDuo processor with 3.00 GHz ad 3 GB RAM. At the begiig of the learig process, the weight atrices betwee iput ad hidde layer ad betwee hidde ad output layer are iitialized with the rado coplex values. Vectors for hidde euro biases b ad output euro biases b are also iitialized with rado coplex values. The goal of back propagatio (BP) learig algorith is to iiize the error-eergy at the output layer. The eural etwork lears the relatioships aog sets of iput-output data (traiig sets) that are characteristic of the copoet uder cosideratio. First, iput data are preseted to the iput euros, ad the output data are coputed. The output data are copared with the desired value ad the errors are coputed. Further, error derivatives are calculated ad sued up for each weight ad bias util whole traiig set has bee preseted to the etwork. These error derivatives are the used to update the weights ad biases for euros i the odel. The traiig process proceeds util errors lower tha the prescribed values is reached. Oce traied, the etwork provides a fast respose for differet iput data. We have tested the behavior of the eural etwork by usig fully coplex activatio fuctios, verifyig the

5 770 It. J. Phys. Sci. Table. Weights (coplex values) betwee iput layer ad hidde layer. Node Node H Node H Node H3 Node H4 Iput i i i i Iput i i i i Iput i i i i Iput i i i i Iput i i i i Iput i i i i Table. Weights(coplex values) betwee hidde layer ad output layer. Node output Node output Node H i i Node H i i Node H i i Node H i i correctess ad aalyzig the iproveet of these fuctios over traditioal artificial eural etwork (ANN) solutios to specific real-world probles. These steps are discussed as follows. For the first odel, i the hidde ad output layers, the sigoid activatio fuctio have bee used as a trasfer fuctio. While, i the secod odel, for the sae etwork, we used a pair of coplex activatio fuctios represetig the real ad the iagiary copoet of z :. For the trasfer fuctio of hidde layer: y = tah( /( ( z 3)* exp( ). For the trasfer fuctio of output layer : y = tah( 3. The perforace criterio used is the su of square due to error(sse). A real-world proble illustrates usig CVNN eural etwork as a classifier to idetify the sex of crabs fro its physical diesios (data were take fro Mathworks). I this exaple, we built a classifier that ca idetify the sex of a crab fro its physical easureets. Six physical characteristics of a crab are cosidered: Species, frotallip, rear width, legth, width ad depth. The six physical characteristics will act as iputs to a eural etwork ad the sex of the crab will be target. The classificatio process cosists of two phases: Traiig phase ad testig phase.a traiig set is used i supervised traiig to preset the proper etwork behavior, where each six iputs observed values for the physical characteristics of a crab is itroduced with its correspodig correct target. As these iputs are applied to the etwork, the etwork outputs are copared to the targets. The eural etwork is expected to idetify if the crab is ale or feale. A -hidde layer feed forward etwork is created with 4 euros. The values of weights (coplex values) betwee iput layer ad hidde layer obtaied durig traiig phase, are show i Table, where the rows correspod to iput odes ad colus correspod to hidde odes, while Table shows the values of weights (coplex values) betwee hidde layer ad output layer obtaied durig traiig phase. The rows correspod to hidde odes ad colus correspod to output odes.

6 Jalab ad Ibrahi 77 Table 3. Bias (coplex values) at the hidde odes. Node H i Node H i Node H i Node H i Table 4. Bias (coplex values) at the output odes. Node output Node output i i Table 5. Copariso of testig tie. Activatio fuctio Testig tie (secods) Traditioal sigoid activatio fuctio Proposed coplex activatio fuctio Table 6. Irreducible CVNN. Activatio fuctio Iteratio uber Traditioal sigoid activatio fuctio 54 Proposed coplex activatio fuctio 306 Table 7. Reducible CVNN. Activatio fuctio Iteratio uber Traditioal sigoid activatio fuctio 379 Proposed coplex activatio fuctio 67 The bias (coplex values) at the hidde odes, ad at the output odes are show i Table 3 ad 4, respectively. RESULTS AND DISCUSSION The eural etwork has bee tested with the testig saples. This will give us a sese of how well the etwork will do whe applied to data fro the real world. Table 5 shows the perforace CVNN based classifier i ter of classificatio tie oly, while the classificatio accuracy was ot affected by proposed activatio fuctio. As see fro Table 5, the tie value for CVNN with the proposed activatio fuctio is less tha that of traditioal activatio fuctio usig the sae traiig ad testig data. I the first odel, the eural etwork works reliably whe usig the proposed coplex activatio fuctios. Less errors are foud i the outputs, with respect to the low iteratio ubers, while i the secod odel, the etwork's perforace is droppig whe usig traditioal sigoid activatio fuctio, because of high iteratio ubers. This idicates how slowly a euro adusts its weight ad bias values accordig to the error. Tests by usig fully coplex activatio fuctios, reduces the testig tie by 54 % copared to the choice of usig the traditioal sigoid activatio fuctio, this iproves the testig tie of the etwork ad prevet the etwork fro startig oscillatio as show i Table 5. I our siulatio exaple, for copariso, aother test has bee perfored for both odels to ivestigate the effect of the reducibility o the odel's covergece. The results of this test for the sae su of squared error (SSE) are show i Table 6 (Irreducibility) ad i Table 7 (Reducibility). Irreducibility results i Table 6, show that the secod odel has better ability of quick learig ad global covergece tha the first odel. Table 7 shows

7 77 It. J. Phys. Sci. that for reducibility, the iteratio uber of the secod odel decreased 56 % copared with the easured iteratio uber of the first odel which decreased by 70 %. The icreasig of the traiig rate of the CVNN for reducibility is due to the tightly iitial distributio of coplex rado weights ad coplex activatio fuctios which ted to slow covergece ad iproves stability of CVNN. Coclusio I this paper, we have show the efficiecy of a coplex valued etwork based o the study of the history of artificial eural etworks, ad the siulatio of the CVNNs is discussed. A live exaple illustrates usig CVNN eural etwork as a classifier to idetify the sex of crabs fro physical diesios based o a fully coplex back propagatio eural etwork. The siulatio results of this paper shows acceptable results for the reducible ad ot the irreducible perforace. The perforace of a fully coplex back propagatio eural etwork has bee substatially iproved by the proposed approach. As previously disccused usig the subordiatio relatio, we defied ad studied the reducibility of 3- layers CVNN with two differet activatio fuctios which satisfied F(. This idea leads to N-layers CVNN uder the coditio: REFERENCES Gaesh A, Balasubraaia G (009). Novel Coplex Valued Neural Networks. It. J. Coput. Appl. Math., 4: Georgi GM, Koutsougeras C (00). Coplex doai backpropagatio. Circuits ad Systes II: Aalog ad Digital Sigal Processig. IEEE Tras., 39: Gooda A (983). Uivalet Fuctios. Marier Publ. Co., Tapa, Fl, Vol. I. Hayki S (008). Adaptive filter theory: Pearso Educatio Idia. Ibrahi RW, Darus M (008). Subordiatio ad superordiatio for uivalet solutios for fractioal differetial equatios. J. Math. Aal. Appl., 345: Ki T, Adali T (00). Fully coplex ulti-layer perceptro etwork for oliear sigal processig. J. VLSI Sig. Process., 3: Miller SS, Mocau P (000). Differetial subordiatios: Theory ad applicatios: CRC. Nitta T (004a). Orthogoality of decisio boudaries i coplex-valued eural etworks. Neural Coput., 6: Nitta T (004b). Reducibility of the Coplex-valued Neural Network. Neural If. Process.-Lett. Rev., : Sivaada S (006) Itroductio to eural etworks usig MATLAB 6.0: Tata McGraw-Hill. Subraaia T, Jalab HA, Taqa AY (00). Overview of textual atispa filterig techiques. Sussa HJ (99). Uiqueess of the weights for iial feedforward ets with a give iput-output ap. Neural Netw., 5: Taqa AY, Jalab HA (00). Icreasig the reliability of ski detectors. Sci. Res. Essays, 5: F ( F (... FN, I this case, we used the subordiatio relatio to get the sadwich assertio. Further, the variety of usig subordiate activatio fuctios i oe CVNN does ot chage the reducibility of the etwork. This leads to the questios: Do Theore hold for o subordiate activatio fuctios? More specifically, is there aother relatio o F ad G such that Theore satisfies? ACKNOWLEDGEMENTS The authors would like to thak the reviewers for their coets which helped to iprove the presetatio of the paper.