Hand Written Digit Recognition Using Backpropagation Neural Network on Master-Slave Architecture

Transcription

1 J.V.S.Srinivas et al./ International Journal of Coputer Science & Engineering Technology (IJCSET) Hand Written Digit Recognition Using Backpropagation Neural Network on Master-Slave Architecture J.V.S.SRINIVAS DVR College of Engineering & Technology, Hyderabad. India N. SHIRISHA DRK Institute of Science & Technology, Hyderabad, India Abstract:- The objective of this work is to identify the hand written digits represented by rectangular box of 6x6 in a gray scale of 56 values. Backpropagation neural network (BPN) is one of the siplest odels of supervised training ulti layer neural networks. In this paper we design an BPN and train it with a set of hand written data. We also ipleent BPN on Master Slave architecture to iniize the learning tie. The perforance paraeters are evaluated for both sequential ipleentation and parallel ipleentation on Master Slave architecture. Keywords:- Backpropagation, Master-Slave architecture, Learning rate and Seeion Handwritten Digit Data Set. I. INTRODUCTION In the recent years handwriting nuber recognition is one of the challenging research proble. Many approaches such as iniu distance, decision tree and statistics have been developed to deal with handwriting nuber recognition probles. Alceu de Britto et al.[] proposed an approach for recognizing the handwritten nueral strings that relies on the two-stage HMM-based ethod. The ain objective for our syste was to recognize isolated Arabic digits exist in different applications. The studies were conducted on Seeion Handwritten Digit Data Set taken fro UCI achine learning repository. The dataset contains 593 handwritten digits fro around 80 persons that were scanned, stretched in a rectangular box 6x6 in a gray scale of 56 values. Then each pixel of each iage was scaled into a Boolean (/0) value using a fixed threshold. Each person wrote on a paper all the digits fro 0 to 9, twice. In general, parallel processing can be achieved in two different ways data set parallelis and algorithic parallelis. Algorithic parallelis offers two possible schees for the MLP: (i) apping each layer of the network to a processor and (ii) apping a block of neurons to a processor. A special case of second schee is to ap each neuron of the network to a processor so that the parallel coputer becoes a physical odel of the network. The proposed parallel ipleentation avoids recoputation of weights and requires less counication cycle per pattern. The counication of data aong the processors in the coputing network is also less. We obtain the perforance paraeters like speed-up, optial nuber of processors and processing tie for both sequential ipleentation and parallel ipleentation on Master Slave architecture. The analytical and experiental results show that the proposed parallel ipleentation perfors better than the sequential ipleentation. II. MATHEMATICAL MODEL In this paper, we consider a fully connected MLP network trained using BP algorith with oentu. We consider MLP with three layers (l=0,,) having N l neurons in each layer. The different phases in the learning algorith and corresponding processing tie are discussed in the following sections. A. Sequential Back Propagation Algorith The BP algorith is a supervised learning algorith, and is used to find suitable weights, such that for a given input pattern (X 0 ), the network output (Y i) should atch with the desired output (d i ). The algorith is divided into three phases, naely, forward phase, error BP phase, and weight update phase. The details on each of these phases and the tie taken to process are discussed below. Forward phase: For a given input pattern X 0, the activation values of hidden and output layer are coputed as follows: ISSN : Vol. 3 No. Nov 0 549

2 J.V.S.Srinivas et al./ International Journal of Coputer Science & Engineering Technology (IJCSET) Y Nl l l l i = f ( w ji X i ), i = 0,,,3,... N l ; l =,; X = j= l where w ji is the weight of the synapse fro the i th neuron in the (l-) layer to the j th neuron in the l th layer, y l j is the output of the j th neuron that belongs to the l th layer, and f(.) is the activation function (bipolar sigoid function). Let t, t a, and t ac be tie taken for one floating point ultiplication, addition, and calculation of activation value respectively. The tie taken to coplete the forward phase (T ) is given by T = N (N 0 + N ) M + (N + N ) t ac where M = t a + t Error back propagation phase: In this phase, the deviation between the network output and the desired value is back propagated to all the neurons through output weights. The δ i ter for i th neuron in output layer is given by δ i = (Y i d i ) f`(.) i=,,3,..n where f (.) is the first derivative of the activation function. The ter (δ i) for i th hidden neuron is given by N δ i = f (.) wjiδ i, i =,,3,... N j= The tie taken to coplete the error back propagation phase is represented by T and is calculated as T = ( + N ) N M + (N + N ) t ac Weight update phase: In this phase, the network weights are updated and the updation process of any w l j depends on the value of δ l j and y l- j. Δw l ji = η δ l iy l- j + μδ w l- ji l =, Where η the learning is rate and μ is the oentu factor. The tie taken to update the weight atrix between the three layers is represented by T 3 and it is equal to T 3 = N (N 0 + N ) M + (N 0 + N ) N t ac Let t ac = β M. The total processing tie (T seq ) for training a single pattern in one iteration is the su of the tie taken to process the three phases and is given as T seq = T + T + T 3 = (N K + N γ) M + N (N 0 + N ) t ac Where k = N 0 + 3N + β and γ = + β B. Parallel Ipleentation: In this ipleentation, the hidden layer is partitioned using vertical parallelis and weight connections are partitioned on the basis of synaptic level parallelis. The Master Slave architecture of the MLP network used in the proposed schee is shown in the fig.. In this architecture there is aster and slave processors. The output layer is placed on front end processor, hidden layer is partitioned into N / neurons and the partitions are placed on slave processors. The input neurons are placed on all the slave processors. In this architecture the counication is only between the aster and Slave processors. Each of the Slave processor with aster executes three phases of BP training algorith. Parallel execution of the three phases and the corresponding processing tie for each phases are calculated. Y ISSN : Vol. 3 No. Nov 0 550

3 J.V.S.Srinivas et al./ International Journal of Coputer Science & Engineering Technology (IJCSET) Fig: Forward phase: In this phase, the activation values of the neurons are calculated. The activation value for the hidden neurons is calculated at the slave processors. But, the activation value for the output neurons is calculated at the aster. The activation values and the weight connections of neurons present in slave processors are send to aster and then activation values of output neurons are calculated. The approxiate tie taken to process the forward phase (T ) is given by. T = [ + N N 0 ] N M + [ N +N ] t ac + N t a + t co where M = t a + t Error propagation phase: In this phase, each processor calculates the error ters. The approxiate tie taken for processing error back-propagation phase (T M ) includes the error calculation at the FEP and at the slave proc4essors and is T = (N + ) N M/ + {N / + N } t ac + t co Weight update phase: In this phase, the weight connections between the neurons of the output layer and hidden layer are updated (we also update the weight connections between hidden and input layers). To update the weight connection between the i th hidden neuron and j th input neuron, we need activation values at the jth input neuron and δ i ter in the ith hidden neuron. Since both the activation value as well as the error ter are present in the processor, no counication is required. The processing tie to update the weight connections (T M 3) is given by T 3 = (N o + N ) N M/ + ((N o + N ) N / ) t Let T M be the tie taken to train a single pattern in proposed HP schee, and it is equal to the algebraic su of tie taken to process all the three phases. T P = M[(N 0 + N ) N + (N + ) N ]/ + [ N +N ] t ac + (N o + N ) N t / + M log {σ + µ[ (N /) +N ]} Here t co = log p [ T ini + T c D] where p is the nuber of processors, D is data size( nuber of words), T ini is startup tie and T c is per word transfer tie.[] C. Analytical Perforance Coparison: Maxiu nuber of processors: In general nuber of processors in parallel ipleentation in the worst case is equal to nuber of hidden neurons. Increase in the nuber of processors does not iprove the perforance after a certain stage. Speed up analysis: Speed-up for -processor syste is the ratio between the tie taken by uniprocessor to the tie taken by parallel algorith in -processor network. The speed-up ratio for parallel ipleentation can be forulated as S = T seq / T P = [(N k + N β)m + N (N 0 + N ) t ac ] {M[(N 0 + N ) N + (N + ) N ]/ + [ N +N ] t ac + (N o + N ) N t / + M log {σ + µ[ (N /) +N ]}} - ISSN : Vol. 3 No. Nov 0 55

4 J.V.S.Srinivas et al./ International Journal of Coputer Science & Engineering Technology (IJCSET) If the network size is extreely larger than the nuber of processors, then the speed up ratio will approach. This is due to extra coputation required in weight update phase and extra counication in exchanging the hidden neurons activation values. Optial nuber of processors: Fro the expression of T P, it is clear that if we increase the nuber of processors, the tie taken to counicate will also increase and the tie taken for coputation will decrease. The total processing tie will decrease first and then increase after a certain nuber of processors. So, there exists an optial uber of processors * for which processing tie is iniu. We calculate the closed for expression for optial nuber of processors by partially differentiating the training tie expression T P with respect to and then equating it to zero. Difference between processing ties: Fro the expressions of T seq and T P the difference in processing tie is calculated as follows. T seq T P = [(N k + N β)m + N (N 0 + N ) t ac ] {M[(N 0 + N ) N + (N + ) N ]/ + [ N +N ] t ac + (N o + N ) N t / + M log {σ + µ[ (N /) +N ]}} Assue that N = N = N 3 = = N. Then T seq T P [5N 5N(σ + μ) 7 + β(n + 4N 4)]t a + [5N + N 5N(σ + μ) 0 + β(n + 4N 4)] t > 0 Fro the above equation it is clear that the tie taken by parallel ipleentation is less than the tie taken by the sequential algorith. III. EXPERIMENTAL RESULTS In this section, we present the perforance of the proposed parallel algorith on aster-salve architecture and copare the results with the sequential approach. Both the algoriths are ipleented in Sparc-II processor based sun workstations connected through Ethernet. There was a substantial decrease in the learning tie of the networks in parallel ipleentation. The results are shown graphically in fig-. Learning Rate:0.. Moentu Factor:0.5 Tie in Sec 8 6 Tie Coparison Basic Sequential Parallel 0 Algoriths Fig- IV. CONCLUSION In this paper, an ipleentation of distributed BP algorith to train MLP network with single hidden layer on a cluster of coputers connected over Ethernet LAN is presented. Hybrid partitioning technique is proposed to partition the network. The partitioned network is apped onto Master-Slave architecture. The analytical perforance of the proposed algorith is copared with the vertical partitioning. Using the hybridpartitioning schee, recoputation of weights is avoided and the counication tie is reduced. As one ISSN : Vol. 3 No. Nov 0 55

5 J.V.S.Srinivas et al./ International Journal of Coputer Science & Engineering Technology (IJCSET) hidden layer is adequate for a large nuber of applications, in the present project work the algorith is developed for neural nets with one hidden layer ACKNOWLEDGEMENTS We acknowledge UCI achine learning repository for giving the opportunity to use the Seeion Handwritten Digit Data Set to train and test our odel. REFERENCES [] Christopher M. Bishop, Neural Networks for pattern recognition, 995. [] Ji Torresen and Shinji Toit, A Review of Parallel Ipleentations of Back propagation Neural Networks, citeseer.ist.psu.edu/ htl. [3] R. Greenalw, H. Hoover, and W. Ruzzo. Liits to Parallel Coputation, Oxford university press, 995. [4] V. Kuar, S. Shekar, and M.B. Ain, A Scalable Parallel Forulation of the Back-Propagation Algorith for Hypercubes and Related Architectures, IEEE Trans. Parallel and Distributed Systes, vol.5, no.0,pp , Oct [5] Czauderna, T.; Seiffert, U.: Ipleentation of MLP networks running backpropagation on various parallel coputer hardware using MPI. Proceedings of the 5 th International Conference on Recent Advances in Soft Coputing (RASC), Nottingha, United Kingdo, Deceber 6-8, 004, pp.6-. [ISBN ] [6] Han-wook Lee and Chan-ik, An Efficeint Parallel Block Backpropagationn Learning Algorith in Transputer- Based Mesh- Connected Parallel Coputers, IEICE Trans. Inf. & Syst., vol. E83-D, No.8, August, 000. [7] Faraarz valafar and Okan K. Ersoy, Parallel Ipleentation of Back-Propagation Neural Network on MASPAR MP- [8] Laurence Fausett, Fundaentals of neural networks; Architectures, algoriths and applications, PEA, 005 [9] S. Haykins, Neural Networks_A Coprehensive Foundation. PEA, 004. [0] Jacek. M.Zurada, Introduction to Artificial Neural systes, 004. [] Alceu de Britto, S., R. Sabourin, F. Bortolozzi and Y. Ching Suen, 003. The recognition of handwritten nueral strings using a twostage HMM-based ethod. Int. J. Docu. Anal. Recog., 5: 0-7. DOI: 0.007/s [] Annanth Graa, Anshul Gupta, George Karypis, Vipin Kuar Introduction to parallel coputing, PEA, 004. [3] W. Richard Stevens.Unix Network Prograing Prentice Hall. [4] N. Sundararajan and P. Saratchandran, Parallel Architecture for Artificial Neural Networks, IEEE CS Press, 998. [5] V. Sudhakar, C. Siva, and R. Murthy, Efficient Mapping of Back-Propagation Algorith onto a Network of Workstations, IEEE Trans. Man, Machine, and Cybernetics-Part B: Cybernetics, vol.8, no.6, pp , 998. ISSN : Vol. 3 No. Nov 0 553