A PARALLEL APPROACH FOR BACKPROPAGATION LEARNING OF NEURAL NETWORKS CRESPO, M., PICCOLI F.,PRINTISTA M. 1, GALLARD R.

Transcription

1 A PARALLEL APPROACH FOR BACKPROPAGATION LEARNING OF NEURAL NETWORKS CRESPO, M., PICCOLI F.,PRINTISTA M. 1, GALLARD R. Grup de Interés en Sistemas de Cmputación 3 Departament de Infrmática Universidad Nacinal de San Luis Ejércit de ls Andes Lcal San Luis Argentina {mcresp,mpiccli,mprinti,rgallard}@inter2.unsl.edu.ar {mcresp,mpiccli,mprinti,rgallard}@unsl.edu.ar Phne: Fax : , ABSTRACT Learning algrithms fr neural netwrks invlve CPU intensive prcessing and cnsequently great effrt has been dne t develp parallel implemetatins intended fr a reductin f learning time. This wrk briefly describes parallel schemes fr a backprpagatin algrithm and prpses a distributed system architecture fr develping parallel training with a partitin pattern scheme. Under this apprach, weight changes are cmputed cncurrently, exchanged between system cmpnents and adjusted accrdingly until the whle parallel learning prcess is cmpleted. Sme cmparative results are als shwn. KEYWORDS: Neutral netwrks, parallelised backprpagatin, partitining schemes, pattern partitining, system architecture. 1 Members f the Research Grup. 2 Full Prfessr at the Infrmatics Department. Head f the Research Grup. 3 The Research Grup is supprted by the Universidad Nacinal de San Luis and the CONICET (Science and Technlgy Research Cuncil). Departament de Infrmática - Facultad de Ciencias Exactas 1

2 1. INTRODUCTION Neural nets learn by training, nt by being prgrammed. Learning is the prcess f adjustment f the neural netwrk t external stimuli. After learning it is expected that the netwrk will shw recall and generalizatin abilities. By recall we mean the capability t recgnise inputs frm the training set, that is t say, thse patterns presented t the netwrk during the learning prcess. By generalizatin we mean the ability t prduce reasnable utputs assciated with new inputs f the same ttal pattern space. These prperties are attained during the slw prcess f learning. Many appraches t speedup the training prcess has been devised by means f parallelism. The backprpagatin algrithm (BP) is ne f the mst ppular learning algrithms and many appraches t parallel implementatins has been studied [7], [12], [13]. T parallelise BP either the netwrk r the training pattern space is partitined. In netwrk partitining, the ndes and weights f the neural netwrk are distributed amng diverse prcessrs. Hence the cmputatins due t nde activatins, nde errrs and weight changes are parallelised. In pattern partitining the whle neural net is replicated in different prcessrs and the weight changes due t distinct training patterns are parallelised. This paper shws the design f a distributed supprt fr parallel learning f neural netwrks using a pattern partitining apprach. Results n speedup in learning and its impact n recall and generalizatin are shwn. 2. BACKPROPAGATION NETWORKS A backprpagatin neural netwrk is cmpsed f at least three unit layers; an input, an utput and ne re mre hidden (intermediate) layers. Figure 1 shws a three layer BP netwrk. y 1 y 2 y 3 y m... Output layer... 1 Hidden layer... 1 Input layer x 1 x 2 x 3 x n Fig. 1 - A Backprpagatin Netwrk Given a set f p 2-tuples f vectrs (x 1 y 1 ), (x 2 y 2 ),..., (x p y p ), which are samples f a functinal mapping y = φ (x) : x R N, y R M, the gal is t train the netwrk in rder it learns an apprximatin = y' = φ' (x). The learning prcess is carried ut by using a tw phase cycle; prpagatin r frward phase and adaptatin r backward phase [6], [11], [12]. Once an input pattern is applied as an excitatin t the input layer, it is prpagated thrughut the remaining layers up t the utput layer where the current utput f the netwrk is generated. This utput is cntrasted against the desired utput and an errr value is cmputed as a functin f the utcme errr in each utput unit. This makes up the frward phase. The learning prcess include adjusting f weights (adaptatin) in the netwrk in rder Departament de Infrmática - Facultad de Ciencias Exactas 2

3 t minimise the errr functin. Fr this reasn, the errr btained is prpagated back frm each nde f the utput layer t the crrespnding (cntributrs) ndes f the intermediate layer. Hwever each f these intermediate units receives nly a prtin f the ttal errr accrding t its relative cntributin t the current utput. This prcess is repeated frm ne layer t the previus ne until each nde in the netwrk receives an errr signal describing its relative cntributin t the ttal errr. Based n this signal, weights are crrected in a prprtin directly related t the errr in the cnnected units. This makes up the backward phase. During this prcess, as the training is prgressing, ndes in the intermediate levels are rganised in such a way that different ndes recgnise different features f the ttal training space. After training, when a new input pattern is supplied, units in the hidden layers will generate active utputs if such an input pattern preserves the features they (individually) learnt. Cnversely, if such an input pattern d nt cntain thse knwn features then these units will be inclined t inhibit their utputs. It has been shwn that during training, backprpagatin neural nets tend t develp internal relatinships between units, as t categrise training data int pattern classes. This assciatin can be either evident r nt t the bserver. The pint here is that, firstly, the net finds an internal representatin which enables it t generate apprpriate respnses when thse patterns used during training are subsequently submitted t the netwrk and secndly, the netwrk will classify thse patterns never seen befre accrding t the features they share (resemblance) with the training patterns. 3. PATTERN PARTITIONING FOR PARALLEL BACKPROPAGATION Pattern partitining replicates the neural net structure (units, edges and assciated weights) at each prcessr and the training set is equally distributed amng prcessrs. Each prcessr perfrms the prpagatin and the adaptatin phases fr the lcal set f patterns. Als, each prcessr accumulates the weight changes prduced by the lcal patterns which afterward are exchanged with ther prcessrs t update weight values. This scheme is suitable fr prblems with a large set f training patterns and fit prperly t run n lcal memry architectures [1], [8], [10]. Different training techniques, r regimes, can be implemented in a backprpagatin algrithm t implement the learning prcess [6], [11], [12]. In this wrk, because it is apprpriated fr a distributed envirnment, the chice was the set-training regime. Under this technique weight changes are accumulated fr all training patterns befre updating any f them, with a subsequent update all stage each time all patterns in the training set were presented. Nde J Nde I Nde F Nde K 4. THE PARALLEL LEARNING ALGORITHM Fig. 2. A Pattern Partitining Scheme Departament de Infrmática - Facultad de Ciencias Exactas 3

4 T implement a pattern partitining scheme, the whle neural netwrk is replicated amng P prcessrs and each prcessr carries ut the learning prcess using Ts/P patterns where Ts is the size f the training set. As shwn in figure 2, weight changes are perfrmed in parallel and then the crrespnding accumulated weight changes vectrs, awc, are exchanged between prcessrs. Nw we describe the basic steps f a backprpagatin learning algrithm under the set-training regime, als knwn as per-epch training 4. The superindexes h and identify hidden and utput units respectively. Batch-Training Algrithm Repeat 1. Fr each pattern x p = (x p1, x p2,..., x pn ) 1.1 Cmpute the utput f units in the hidden layer: h net pj N h = w x i= 1 ji pi + Θ j h i pj = f (net h pj ) 1.2 Cmpute the utput f units in the utput: net pk L = w i j= 1 kj pj + Θ k pk = f (net pk ) 1.3 Cmpute errr terms fr the units in the utput layer: δ pk = (y pk - pk ) f ( net pk ) 1.4 Cmpute errr terms fr the units in the hidden layer: h δ pj = f (net h pj ) δ pk k 1.5 Cmpute weight changes in the utput layer: c kj = c kj + η δ pk ipj 1.6 Cmpute weight changes in the hidden layer: c h ji = c h ji + η δ h pj xi h 2. Send lcal c and c and Receive remte c h y c. 3. Update weights changes in the utput layer: w kj = w kj + ( c kj (lcal)+ c kj (remte)) 4. Update weights changes in the hidden layer: h = w h ji +( c h ji (lcal)+ c h ji (remte)) T M w ji Until (E = ( 1 (y pk - pk ) 2 ) < maximum accepted errr) r p= 1 2 k = 1 (number f iteratins < maximum number f iteratins) The subindexes p, i, j and k identify the p th input pattern, the i th input unit, the j th hidden unit and the k th utput unit respectively. w ij is the weight crrespnding t the cnnectin between unit j and unit i, c ij is the accumulated change fr the weight 4 During an epch, r batch, the submissin f all patterns in the partitin, the crrespnding cmputatins and the accumulatin f weight changes must be perfrmed befre weights update takes place, then the next epch begin. w kj Departament de Infrmática - Facultad de Ciencias Exactas 4

5 crrespnding t the cnnectin between unit j and unit i, Θ is the bias and N, L and M identify the number f input, hidden and utput units respectively. f dentes the sigmid functin which is used t cmpute the activatin f each nde. 5. PARALLEL LEARNING SUPPORT In this sectin we present a brief descriptin f the underlying system architecture and prcedures supprting the parallel learning prcess, running n the prcessrs distributed in a LAN f wrkstatins [3]. Each prcessr is allcated fr a replicated neural netwrk. The parallel learning supprt presented here is independent f the neural netwrk structure. 5.1 System Architecture Figure 3 shws the supprt system architecture, prcesses and interactins. The parallel learning algrithm is independently initiated by a prcess at each LAN nde. The PrcW prcess, will be respnsible f lcal learning prcessing and when needed will request a service t exchange vectrs awc. PrcW frks twice t create child prcesses HI and HJ fr cmmunicatin with ther LAN ndes. Lcal Weight Vectr awc NN Cmpute New weight Vectr Current Weight Vectr AWC PrcW Remte Weight Vectr Lcal Weight IPC1 IPC2 PrcHI Receives Remte Weight Vectrs Sends Envia Lcal Vectr Weight Pes Vectrs Lcal N E T W O R K PrcHJ The tasks perfrmed by each prcess are: Fig Supprt System Architecture Main prcess PrcW runs the learning algrithm fr the neural net NN. After each epch PrcW request the fllwing services: Sending f the lcal awc vectr. Receiving f remte awc vectrs. T update the weights fr every pattern in the partitin it creates a current AWC vectr by gathering the lcal and remte awc vectrs. Prcess PrcHI receives the awc vectrs sent by remte PrcHJ prcesses. Prcess PrcHJ sends the lcal awc vectrs generated during ne re mre epchs t the remte PrcHI prcesses. Departament de Infrmática - Facultad de Ciencias Exactas 5

6 These three prcesses execute independently but they cmmunicate fr exchanging infrmatin. In rder t allw the cncurrent executin f learning and the external cmmunicatin with remte prcesses, a set f nn blcking mechanisms where used [4], [9], [14], [15]. This cmmunicatin is handled via interprcess cmmunicatin mechanisms (IPCs). The interactin between prcesses PrcW and PrcHI is established via IPC1 when after an epch the learning prcess request t distribute its awc vectr. The interactin between prcesses PrcHJ and PrcW is established via IPC2 when after an epch the learning prcess request remte awc vectrs t subsequently determine the current AWC vectr. 6. THE PARALLEL VERSION OF THE NND SUPERVISOR In ur research, Neural Netwrk Devices (NND) are used t supervise lad distributin in Distributed Systems. In particular, fr parallelising BP, the test case was chsen as a reduced versin f the ne discussed elsewhere [2]. The neural netwrk was built as a Feed Frward Neural Net (FFNN) f 14 input ndes, 2 utput ndes and ne hidden layer f 30 ndes with a ttal number f 527 intercnnectins. The pattern space cntains a ttal number f 2,376 patterns. Fr the experiments discussed later nly ne third (abut 800 samples) f the pattern space was enugh t btain satisfactry results. In previus wrks by creating prttypes we crrbrate that claimed neural netwrk features such as fast respnse, strage efficiency, fault tlerance and graceful degradatin in face f scarce r spurius inputs make them apprpriate tls fr Intelligent Cmputer Systems. Nw, parallel learning f prttypes reflecting different allcatin plicies are being implemented and in a testing stage by means f a pattern partitining scheme with a settraining regime as explained in previus sectins. 7. EXPERIMENTS DESCRIPTION Our initial experiments in parellelising neural nets were divided int tw stages; 1. T establish if strng crrelatin exists between data dependencies and gdness f results expressed as the recall and generalisatin abilities. 2. T cmpare gdness f results and speedup fr diverse sizes f the training set. In bth analysis, t cmpare results, the neural net was trained sequentially and in parallel. Under pattern partitining, the pattern space was divided int three sets and each set was assigned t ne prcessr. Many series were perfrmed and each ne cnsisted f many runs fr η values f 0.05 and 0.1. The fllwing relevant perfrmance variables were examined: T: is the running time f the learning algrithm. R = r/size. Is the recall abitily f the neural net. Were Size is the size f the submitted training set and r is the ttal number f patterns recgnised when nly thse patterns f the training set are presented, after learning, t the netwrk. G = g/s. Is the cmbined recall and generalizatin ability. Were S is the size f the ttal pattern space and g is the ttal number f patterns recgnised when all the patterns are presented, after learning, t the netwrk. Sp =T seq / T par is the rati between the sequential learning and the parallel learning times. Departament de Infrmática - Facultad de Ciencias Exactas 6

7 Rec B/C = Sp/(R seq - R par ) is the benefit-cst rati fr recall. It indicates the benefit f speeding up the learning prcess which is paid by the cst f lsing recall ability. GenB/C = Sp/(G seq - G par ) is the benefit-cst rati fr generalizatin. It indicates the benefit f speeding up the learning prcess which is paid by the cst f lsing generalizatin ability. Partitining schemes were applied as fllws (See figures 4.a and 4.b). In the first analysis stage, fr sequential backprpagatin (SBP) the training set T s was built by unifrm selectin f 30% f the pattern space S. As we decided t dedicate nly three separate prcessrs, fr parallel training (PBP) three randm disjint subsets were selected accrding t the fllwing criteria: One third f Ts fr each subset. (Experiment PBP-10-D) 10% f S fr each subset. (Experiment PBP-10-Ι) In the secnd analysis stage, fr sequential backprpagatin (SBP) the training set T s was built by unifrm selectin f X% f the pattern space S. Fr parallel training (PBP) three disjint subsets f X/3% f S were selected. X was chsen as 30, 45 and 60. (Experiments PBP-X/3). Ttal Patterns Set 30% f Patterns SBP 10% 10% 10% PBP PBP PBP Fig. 4.a - Dependent-Data Partitining Apprach Departament de Infrmática - Facultad de Ciencias Exactas 7

8 X % f Patterns Ttal Patterns Set SBP X/3% X/3% X/3% PBP PBP PBP Fig. 4.b - Independent-Data Partitining Apprach 8. RESULTS We shw nw sme results frm bth investigatin stages. 8.1 Stage I T determine the pssible existence f a strng crrelatin between data dependencies and gdness f results, experiments SBP, PBP-10-D and PBP-10-Ι were cnducted. The issues studied thrugh a large number f runs were; learning time, recall, generalizatin and number f iteratins needed t reach an acceptable errr value while training. The fllwing figures and tables shw the crrespnding mean values. Experiment Learning Time Recall Generalizatin SBP PBP-10-D PBP-10-Ι Table 1. Summary f T, R and G results in Stage I Secnds Learning Time SBP PBP D PBP I Experiments % Recall and Generalizatin R 100 G SBP PBP D PBP I Experiments Fig. 5 - Values f T fr Stage I under sequential and parallel prcessing. Fig. 6 - Values f R and G fr Stage I under sequential and parallel prcessing. Departament de Infrmática - Facultad de Ciencias Exactas 8

9 As we can bserve in the abve table and figures imprtant learning time reductin is achieved withut significant lss f recall and generalizatin capabilities f the neural netwrk. This results are attained by using either parallel partitining apprach. Hwever the independent data criterin (PBP-10-Ι) seems t prvide better perfrmance values, pssibly due t a mre unifrm distributin f data gathered. Learning Prcess Trace 100 Errr PBP-10-D PBP-10-I SBP Number f iteratins Fig. 7 Prgress f the learning prcess under sequential and parallel prcessing As it is shwn in figure 7, the substantial time reductin btained under either parallel apprach arise frm a lesser number f iteratins t arrive t an acceptable errr value. When the learning prcess is perfrmed in parallel, each replicatin f the neural net is trained with fewer number f patterns and it learns nt nly due t its individual wrk but als fr the experience shared with ther replicas. 8.2 Stage II T cmpare gdness f results and speedup fr diverse sizes f the training set many tests were perfrmed in experiments SBP-X and SBP-X/3 addressed t learning time, recall, generalizatin, speed-up, and benefit-cst rati. Here, parallel prcessing was implemented under the independent data apprach. Experiment Learning time Recall Generalizatin SBP PBP SBP PBP SBP PBP Table 2. Summary f T, R and G results in Stage II Departament de Infrmática - Facultad de Ciencias Exactas 9

10 Learning Time Recall Secnds SBP Experiments PBP 30 vs vs vs % SBP Experiments PBP 30 vs vs vs 20 Fig. 8 - Values f T fr Stage II, under sequential and parallel prcessing Fig. 9 - Values f R fr Stage II under sequential and parallel prcessing % SBP Generalizatin Experiments PBP 10 vs vs vs 60 Fig Values f G fr Stage II under sequential and parallel prcessing. As we can bserve in Table 2 and Figure 8 cnsiderable learning time reductin is achieved als fr diverse sizes f the training set thrughut experiments f Stage II.This results are attained by using the independet data apprach. Figures 9 and 10 shw the assciated lss in recall and generalizatin f the neural netwrk fr different sizes f the training set. As anticipated, the amunt f capabiliy detriment decreases as the training set size is incremented, and its values range frm 2.9% t 5.6% fr recall and frm 2.1% t 5.3% fr generalizatin. Speed-up SBP-30 vs. PBP-10 SBP-45 vs. PBP-15 SBP-60 vs. PBP Speed-up Sp SBP-30 vs PBP-10 SBP-45 vs PBP-15 SBP-60 vs PBP-20 Experiments Fig Speed-up values achieved thrugh parallel prcessing. The abve table and figure indicate the Speed-up attained thrugh parallel prcessing when different sizes f the prtins f the pattern space S are selected fr training the neural netwrk. As the size f T s increases then an effect f slwing the Speed-up arises but this cnsequence is cmpensated by an imprvement in quality f results. Departament de Infrmática - Facultad de Ciencias Exactas 10

11 Benefit/Cst Rati SBP-30 vs. PBP-10 SBP-45 vs. PBP-15 SBP-60 vs. PBP-20 Recall B/C Generalizatin B/C Benefit-Cst Rati fr Recall Rec B/C SBP-30 vs PBP-10 SBP-45 vs PBP-15 SBP-60 vs PBP-20 Fig Benefit-Cst Rati fr Recall thrugh parallel prcessing. 3 Benefit-Cst Rati fr Generalizatin Gen B/C 2 1 SBP-30 vs PBP-10 SBP-45 vs PBP-15 SBP-60 vs PBP-20 The Benefit-Cst Rati gives an indicatin f a benefit (speed-up) btained by paying a cst (detriment in recall r generalizatin). This perfrmance variable culd be f great help when designing a parallel apprach fr training neural nets. It is clear that when larger sizes fr the training set are used the benefit decreases and cnsequently the cst decreases. In any case, decisin will depend n applicatin requirements. 9. CONCLUSIONS Fig Benefit-Cst Rati fr Generalizatin thrugh parallel prcessing. The time that it takes t train a neural netwrk has lng been an issue f research. The length f training time depends, essentially, upn the number f iteratins required. This number depends n several interrelated factrs. Sme f them are; size and tplgy f the netwrk, initialisatin f weights and the amunt f training data used. A means f training acceleratin based in the last mentined factr is a parallel apprach knwn as pattern partitining. Departament de Infrmática - Facultad de Ciencias Exactas 11

12 In this paper we presented a feasible architecture fr a system supprting parallel learning f backprpagatin neural netwrks using a pattern partitining scheme with a set-training regime. A preliminary set f experiments in ur investigatin, revealed that the beneficial effects f parallel prcessing can be achieved with minr capability lss. Fr the small neural net selected fr the testing case a substantial acceleratin ranging frm 4 t mre than ten times was attained by using as few as three prcessrs. As the distributed apprach t parallelise the learning prcess shwed its effectiveness, at the present time, tests with larger number f prcessrs, different training set sizes and variable cmmunicatin intervals are being perfrmed fr different neural netwrks. 9. ACKNOWLEDGEMENTS We acknwledge the cperatin f the prject grup fr prviding new ideas and cnstructive criticisms. Als t the Universidad Nacinal de San Luis and the CONICET frm which we receive cntinuus supprt. 10. REFERENCES [1] Berman F., Snyder L. - On mapping parallel algrithms int parallel architectures- Parallel and Distributed Cmputing, pp , [2] Cena M., Cresp M. L., Gallard R. Transparent Remte Executin in LAHNOS by Means f a Neural Netwrk Device. Operating System Reviews, Vl. 29, Nr 1, ACM Press, [3] Cluris G., Dllimre J., Kindberg T. -Distributed Systems: Cncept and Design - Addisn-Wesley, [1] Cmer, D. E., Stevens, D. L. - Internetwrking with TCP/IP - Vl. III - Prentice Hall. [1] Fster, Ian T. : Designing and Building Parallel Prgrams - Addisn Wesley, [2] Freeman, J., Skapura, D. Neural Netwrks. Algrithms, Applicatins and Prgramming Techniques. Addisn-Wesley, Reading, MA, [3] Girau, B. - Mapping Neural Netwrk Back-Prpagatin nt Parallel Cmputers with Cmputatin/Cmmunicatin Overlapping. [4] Kumar, V., Shekhar, S., Amin, M.- A Scalable Parallel Frmulatin f the Backprpagatin Algrithm fr Hypercubes and Related Architectures. IEEE transactins n Parallel and Distributed Systems, Vl. 5. Nr.10, Octber pp [5] McEntire, P. L., O Reilly, J. G., Larsn, R. E. (Editrs) : Distributed Cmputing: Cncepts and Implementatins - Addisn Wesley, [6] Petrwski, A., Dreyfus, G., Girault, C.- Perfrmance Analysis f a Pipelined Backprpagatin Parallel Algrithm. IEEE. Trasactins Netwrks, Vl. 4, Nvember pp [7] Plaut, D., Nwlan, S., Hintn, G. Experiments n Learning by Backprpagatin. Tech. Reprt, CMU-CS , Carniege Melln University, Pittsburg, PA, [8] Rumelhart, D., Hintn, G., Willams, R. Learning Internal Representatins by Errr Prpagatin. MIT Press, Cambridge, MA, [9] Rumelhart, D., McClelland, J. Parallel Distributed Prcessing, vl. 1 y 2. MIT Press, Cambridge, MA, [10] Stevens, R.W.- Advanced Prgramming in the UNIX Envinment- AdDisn- Weslwy Publishing Cmpany [11] Stevens, R.W.- UNIX Netwrk Prgraming. Prentice Hall-Englewd Cliff Departament de Infrmática - Facultad de Ciencias Exactas 12