Electromagnetic Algorithm for tuning the structure and parameters of Neural Networks

Transcription

1 Electromagnetc Algorthm for tunng the structure and parameters of Neural Networks Ayad Mashaan Turky, Salwan Abdullah and Nasser R. Sabar Abstract Electromagnetc algorthm s a populaton based meta-heurstc whch mtates the attracton and repulson of sample ponts. In ths paper, we propose an electromagnetc algorthm to smultaneously tune the structure and parameter of the feed forward neural network. Each soluton n the electromagnetc algorthm contans both the desgn structure and the parameters values of the neural network. Ths soluton later wll be used by the neural network to represents ts confguraton. The classfcaton accuracy returned by the neural network represents the qualty of the soluton. The performance of the proposed method s verfed by usng the well-known classfcaton benchmarks and compared aganst the latest methodologes n the lterature. Emprcal results demonstrate that the proposed algorthm s able to obtan compettve results, when compared to the best-known results n the lterature. I. INTRODUCTION Neural networks (NNs are computng technques nspred by nature, whch has been successfully used to solve a wde varety of problems such as pattern recognton [], sgnal processng [] and optmsaton []. The use of neural network to solve real world problems requres some crtcal decsons, whch may has negatve effect on solvng certan problem. or example, the optmum network structure and parameters are the most mportant attrbutes of the network networks, whch have drect effect on the soluton qualty, to be determned [3]. In multdmensonal space, the tunng of the neural network structure and parameters can be consdered as a complex optmsaton problem, snce each pont represents a potental neural network wth dfferent network structures and lnk weghts [3]. Hence, the use of fxed parameters and structure at the overall connectvty between the neurons may not produce good results. A network that has small neurons may not acheve good performance due to ts lmted nformaton processng power. On the other hand, a network that conssts large number of neurons may contan many redundant connectons and also computatonally expensve [4, 5]. Durng the last decade, several populaton-based methods have been developed to generate the approprate structure and parameters of a neural network. Ilonen et al. Ayad Mashaan Turky and Salwan Abdullah are wth Unverst Kebangsaan Malaysa, UKM Bang, Selangor, Malaysa (e-mal: ayadalrashd@gmal.com, salwan@ftsm.ukm.my. Ayad s also afflated wth Swnburne Unversty of Technology, Vctora, Australa. Nasser R. Sabar s wth The Unversty of Nottngham Malaysa Campus, Jalan Broga, Semenyh, Selangor, Malaysa (e-mal: Nasser.Sabar@nottngham.edu.my. [6] employed a dfferental evoluton for tranng the feed forward neural network. The proposed algorthm produced promsng results when tested on pattern classfcaton and functon approxmaton. Leung et al. [4] proposed a genetc algorthm to tune the structure and parameter values of a neural network. In ths approach, a fully connected three-layer feed forward neural network wth swtches are used and the number of hdden nodes s manually determned, startng wth a small number and teratvely ncreased untl the learnng performance s acheved. The algorthm s tested on sunspots and assocatve memory. Tsa et al. [7] employed a hybrd Taguch-genetc algorthm to tune the structure and the parameter values of a neural network. In ths approach, the same model whch s proposed by Leung et al. [4] s used. The proposed algorthm has shown excellent results when used to estmate the number of sunspots and to realze the assocatve memory. Da et al. [8] ntroduced a new populaton-based heurstc search algorthm called seeker optmsaton algorthm, whch s used to tune the structure and the parameter values of a neural network. The proposed algorthm has shown good results when tested on pattern classfcaton and functon approxmaton. Zhao et al. [9] appled a cooperatve bnary-real partcle swarm optmsaton to tune the structure and parameter values of a neural network. In ths method, bnary partcle swarm optmsaton (PSO s used to tackle the swtches set, where each swtch has ether 0 or value, whlst the basc partcle swarm optmsaton s used to optmse the weght values. The proposed bnary-real PSO algorthm has been able to acheve the state-of-the-art results when tested to estmate the number of sunspots. The successes of the mentoned populaton-based methods are the man motvatng factors for proposng a new populaton-based method based on Electromagnetc algorthm (EM for tunng both structure and parameter values of a feed forward neural network. EM s a populaton based meta-heurstc method that mtates the attracton and repulson of the sample ponts and moves them towards a hgh qualty soluton whle avodng the local optma [0]. It has been successfully employed to solve several optmsaton problems such as examnaton tmetablng problems [], vehcle routng problems [] and ob shop schedulng [3], whch made t a worthy canddate to consder for solvng real world problems. In addton to our knowledge, t has not been comprehensvely studed n the context of tunng both the structure and parameter values of a feed forward neural network [4-6]. Ths paper s organsed as follows. Secton presents the proposed algorthm. Expermental results are dscussed

2 n Secton 3. nally, some bref concludng comments are provded n Secton 4. 0 f α 0.5 (3 f α0.5 II. THE PROPOSED ALGORITHM In ths secton, the NN wth lnk swtches, followed by the descrpton of the EM algorthm and ts applcatons to tune the structure and parameters of the neural network are presented. A. The NN wth lnk swtches In general, the structure of a neural network starts wth a fxed number of nputs, hdden and outputs nodes. These nclude the set of the parameters and network structure. Hence, the use of fxed parameters and structure may not yeld good results wthn a gven tranng perod. Small networks get nto the local mnma too easly, and may not acheve good results due to ts lmted nformaton processng power. On the other hand, large networks may take a long tme to learn the characterstcs of the data and computatonally too expensve [4] [7] [8]. In ths study, we used a fully connected three layer neural network wth lnk swtches that was proposed by Leung et al. [4]. In order to choose an optmal number of hdden nodes, ther numbers are frst fxed between three and seven, n order to test the learnng performance. The nput-output relatonshp can be defned as follows: B. Electromagnetc Algorthm (EM for tunng the neural networks Nature nspred algorthms have proven to be an effectve soluton method for varous optmzaton problems [7, 8]. Electromagnetc algorthm (EM s a recent nature nspred populaton based metaheurstc algorthm ntroduced by Brbl and ang [0]. EM smulates an attracton repulson mechansm of electromagnetc theory n explorng the mult-dmensonal soluton space of a gven problem. Each pont represents a soluton and each soluton s assocated wth a charge, whch represents the qualty of the soluton. The solutons wll exert some force (attracton or repulson on other solutons. The attracton-repulson force s used to explore a soluton search space. The man dea behnd the attracton-repulson s that, a bad qualty soluton repels other solutons from movng towards ts drecton. On the other hand, a good qualty soluton wll attract other solutons to move towards ts drecton. EM has four steps as follows (see g. :, k=,,, no. ( Start Intalsaton where z, z,,z n and y, y n o are the nputs and outputs of neural network, respectvely; n represents the number of nputs; n o represents the number of outputs; n h represents the number of hdden nodes; v denotes the weght of the lnk between the th hdden node and the th nput node; w k denotes the weght of the lnk between the kth output node and the th hdden node; b and b denote the bases for the hdden and output nodes, respectvely; s represents the swtch lnk between the th nput node and the th hdden node; t k represents the swtch lnk between the th hdden node and the kth output node; and denote the bas swtches of hdden and output nodes, respectvely. In case the swtch value s equal to, there s a lnk between two nodes from dfferent layers, otherwse there s no lnk between these two nodes. Logsg (. s a sgmod functon that s defned n Eq. (: logsg( = e. ( or each dmenson, the connecton weght values are tuned to be wthn [-, ] and the lnk swtch bt s ether 0 or as n [4]. Along wth [4], a unt step functon s ntroduced to each lnk that s defned as n Eq. (3: - Set the populaton sze, M - Set the number of teratons as a stoppng crteron, MAXITER - Set the number of teratons for the local search, LSITER Generate populaton of solutons at random Populaton evluaon Evaluate the soluton by callng NN to solve a gven problem dataset usng the current structure and parameter values of current soluton Local search At each teraton, evaluate the modfed soluton by callng NN to solve a gven problem dataset usng the current structure and parameter values of current soluton Calculate the force of each soluton Movement Evaluate the modfed soluton by callng NN to solve a gven problem dataset usng the current structure and parameter values of current soluton No Stoppng crteron satsfed? g. lowchart of the EM algorthm for tunng neural networks (EM-NN. Yes Return the best soluton

3 Intalsaton: In ths step, EM parameters are ntalsed.e., the populaton sze (M, number of teratons (MAX ITER and number of teratons for local search (LS ITER. A populaton of soluton s randomly created. In ths study, a one-dmensonal vector represents the soluton. The sze of the vector s equal to the number of decson varables n the gven problem. Each cell n the vector represents one decson varable. In ths study, the soluton s dvded nto two parts: network structure, and connected weghts. Two types of representatons (bnary and real are used. Bnary representaton represents the network structure, whlst the real representaton represents the regularzaton parameter and connected weghts. or bnary representaton, the soluton s generated by randomly assgnng ether zero or one for each varable. or real representaton, the soluton s randomly generated by assgnng a random value to each decson varable wthn ts upper and lower range as calculated n Eq. (4. x = L x + rand [0,] ( U x - L x (4 where rand return a random number between [0,], U x and Lx are the upper and lower bounds of the decson parameter, respectvely. The proposed method s employed to learn the neural network model for approxmatng the gven nput-output relatonshps: y d (t= g(z d (t, t=,,,n d (5 where z d (t= [z d (t z d (t z d n (t] and yd (t=[ y d (t y d (t y d (t] represent the nputs and the desred outputs of no an unknown nonlnear functon g(., respectvely. n d s the number of the nput-output pars. The qualty of each soluton s calculated by creatng the neural network usng current structure and the weght values encoded n the current soluton usng Eq. (6. The obectve s to maxmse the ftness value as defned n Eq. (6 and to mnmse the error rate as n Eq. (7. f= err wth where the err represents mean absolute error (MAE. The sample of the soluton representaton s gven n g.. Weght values Swtches (6 (7 g.the representaton of the soluton showng weght and swtches values. Local search: In ths step, a local search procedure s conducted to mprove the qualty of the solutons n the populaton that are generated n the ntalsaton step. The local search procedure has two parameters.e., the number of teraton (LS ITER and the multpler for the neghborhood search (δ, whch s referred to as a changng amount when the soluton s updated by addng or reducng by δ amount. At each teraton, the local search procedure generates a neghbourhood soluton by addng or subtractng δ from the current soluton for the real code representaton. Both addng and subtractng have the same probablty rate whch s fxed to 0.5,.e., f the generated random number s less than 0.5, then δ s subtracted from the current soluton, otherwse, δ s added to the current soluton (Brbl and ang [0]. The neghbourhood soluton s generated by flppng-floppng the decson varable value from zero to one or one to zero n the bnary representaton. If the neghbourhood soluton s better than the current one, t wll be accepted and becomes the current soluton for the next teraton. Otherwse, t wll be reected. Ths process s repeated for a pre-defned number of teratons (LS ITER. 3 Total force calculaton: In ths step, the charge of each soluton based on the obectve functon s calculated. Based on the charge value, the soluton wll ether follow attracton or repulson. The charge of each soluton s calculated as stated n Eq. (8: q exp n m k f ( x f ( x ( f ( x f ( x,. best Ths mples that the good qualty solutons have a hgher charge and consequently wll have a strong attracton. Thus, the solutons wll be attracted to the good qualty solutons and wll be repel from the bad qualty solutons. Based on the calculated charge, the total force,, exerted on each soluton s calculated as shown n Eq. (9: m ( x ( x J q q x x x x x q q x rom Eq. (9, the calculated total force between the solutons s proportonal to the product of the charges and s nversely proportonal to the dstance between the solutons. k best f f ( x f ( x f f ( x f ( x (8 (9

4 4 Movement: Based on the calculated total force n Eq. (9, the solutons moved n the drecton of the total force by a random step length usng Eq. (0. The random step length takes a random value between zero and one. A postve force wll move the soluton towards the upper bound, whlst a negatve force wll move the soluton towards a lower bound. Note that, the best qualty soluton wll not move and wll only attract other solutons. or the real code representaton, the movement s calculated based on Eq. (0. x = x + λ (RNG =,,,m (0 or the bnary representaton, the soluton s moved based on the gven probablty as follows: or the attracton case: a soluton s moved accordng to the attracton probablty (PA, whch s fxed to a large number between zero and one. or the repulson case: a soluton s repelled based on the repulson probablty (PR, whch s fxed to small number between zero and one. or example, assume that PA=0.8 and PR =0.0. If the state s attractng, the soluton wll be moved as follows: generate a random number PR between zero and one. If PR s less than PA, flp-flop the current decson varable. Otherwse, keep t unchanged. III. EXPERIMENTAL RESULTS In ths secton, the proposed algorthm EM-NN performance s analysed usng fve datasets obtaned from Unversty of Calforna at Irvne (UCI Machne Learnng Repostory ( These datasets are chosen because they represent a varety of mportant real world problems as well as many researchers have used them to evaluate the performance of ther algorthms. The characterstcs of these datasets are presented n Table. TABLE CHARACTERISTIC O THE DATASETS USED Dataset attrbutes nstances classes Australan Breast cancer German Irs Pma class In order to determne the approprate values for the EM parameters (.e., populaton sze, stoppng crteron, and number of teratons for the local search, some prelmnary experments were conducted. The obtaned prelmnary results n terms of the classfcaton accuracy are presented n Table, where the algorthm performs the best wth the populaton sze = 00 (presented n bold on two datasets. TABLE RESULTS O USING DIERENT POPULATION SIZE Dataset Populaton sze, M Pma Breast cancer We then tuned the number of teratons for the stoppng crteron (MAX ITER and the local search (LS ITER by fxng the value of the populaton sze as 00. We examned MAX ITER wth three dfferent values.e., 50, 00 and 50, and LS ITER wth 5, 0 and 5 as shown n Table 3. TABLE 3 CLASSIICATION ACCURACY WITH DIERENT VALUES O MAX ITER AND LS ITER Datasets MAX ITER LS ITER Pma Breast cancer Table 3 presents the results of the classfcaton accuracy. The best results are presented n bold. rom Table 3, t s clearly shown that the best classfcaton accuracy are obtaned when MAX ITER = 00 and LS ITER = 0. The fnal parameter settngs for the EM algorthm are presented n Table 4. TABLE 4 PARAMETER SETTING Parameter Value Populaton sze, M 00 No. of teratons for the local search (LS ITER 0 EM stoppng condton (MAX ITER 00 In our experments, each dataset has been dvded nto tranng data (75% and test set (5% as n [8]. The experments were executed for 50 tmes wth dfferent seed numbers on a PC wth.4 GHz speed and 4 GB RAM under Wndows 7 Operatng System. In ths study, the performance of the proposed algorthm s compared wth the state-of-the-art approaches, whch are summarsed n Table 5. Note that, these approaches are chosen based on ther ablty to produce the best-known results n the lterature. TABLE 5 ACRONYMS O STATE-O-THE-ART APPROACHES IN COMPARISONS Symbol Refer ences Descrpton PSO+SV M [9] GA+SVM [0] Partcle swarm optmzaton for parameter determnaton and feature selecton of support vector machnes. A GA-based feature selecton and parameters optmzaton for support vector machnes.

5 SS-based ensemble [] SA-SVM [] Ensembles [3] Enhancng the classfcaton accuracy by scatter-search-based ensemble approach. Parameter determnaton of support vector machne and feature selecton usng smulated annealng approach. Creatng dversty n ensembles usng artfcal data. The results of the comparson are presented n Table 6, where the best results are presented n bold. The comparson shows that our proposed approach (EM-NN s able to obtan three new best results out of fve tested datasets. The dea of attracton-repulson mechansms wthn the EM algorthm that ams to move the solutons toward the hgh qualty solutons and avod the soluton from beng trapped n a local optmum helps n achevng hgher classfcaton accuracy on the tested datasets. Ths shows that EM s one of the approprate methods to smultaneously tune the structure and parameters of the NN for the classfcaton problems. TABLE 6 CLASSIICATION ACCURACY OR EM-NN AND OTHER METHODS Algorthms Datasets EM-NN PSO+ GA+ SS-based SA-SVM Ensembles SVM SVM ensemble Australan Breast cancer German Irs Pma ndcates that datasets have not been attempted. The results obtaned are further analysed by conductng a redman s mult comparson statstcal tests wth a sgnfcant nterval of 95% (α = 0.05 to see f there s any sgnfcant dfference between EM-NN and the compared methods (PSO+SVM, GA+SVM and SS-based ensemble [4]. Note that, only those methods that have been tested on all datasets are consdered n ths statstcal test. If sgnfcant dfferences are detected (based on redman s test, post hoc methods (Holm s and Hochberg s tests are conducted to obtan the adusted p-values for each comparson between the control algorthm (the best-performng one n the comparson s the one that has st rank accordng to redman s test and the rest of algorthms. The p-value computed by the redman s test s 0.000, whch s below the sgnfcant nterval of 95% (α = Ths ndcates that there s a sgnfcant dfference among the observed results. Table 7 summarses the rankng obtaned by the redman s test where EM-NN s ranked frst. 3 GA+SVM SS-based ensemble.6 A post hoc method s conducted to obtan the adusted p-values for each comparson between EM-NN (as the controllng method and PSO+SVM, GA+SVM and SS-based ensemble algorthms. Table 8 shows the adusted p-values whch reveals that EM-NN s better than (PSO+SVM and GA+SVM wth α = However, there s no sgnfcant dfference between EM-NN and SS-based ensemble (adusted p-value s hgher than Nevertheless, the results reported n Table 6 clearly show that EM-NN s able to obtan three new best results out of fve tested datasets as compared to SS-based ensemble that s only margnally better on two datasets. TABLE 8 ADJUSTED P-VALUE O THE COMPARED METHODS Algorthm Unadusted P P Holm P Hochberg PSO + SVM GA + SVM SS-based ensemble IV. CONCLUSION In ths paper, we have presented a methodology that handles the problem of tunng the structure and parameters of the three layers fully connected feed forward neural network wth lnk swtches based on the prncple of the electromagnetc-lke mechansm. The performance of the approach s tested on classfcaton of benchmark datasets. We have made comparson wth a set of state-of-the-art approaches from the lterature. Ths approach produces three best results and s consstently good across the all the benchmark problems n comparson wth other approaches studed n the lterature. Wth the help of the electromagnet that moves sample ponts (solutons towards a hgh qualty soluton whle avodng the local optma by utlsng a calculated force value, our approach s capable of fndng better solutons for the classfcaton problems. We are confdent that a sgnfcant contrbuton n producng hgh qualty solutons to the classfcaton problems has been made. REERENCES TABLE 7 AVERAGE RANKING O RIEDMAN S TEST. # Algorthm Rankng EM-NN.4 PSO+SVM 3.5