CHARACTER RECOGNITION USING A SELF-ADAPTIVE TRAINING

Transcription

1 CHARACTER RECOGNITION USING A SELF-ADAPTIVE TRAINING Dr. Eng. Shasuddin Ahed $ College of Business and Econoics (AACSB Accredited) United Arab Eirates University, P O Box 7555, Al Ain, UAE. and $ Edith Cowan University School of Engineering and Matheatics 00-Joondalup drive, Joondalup, Perth, WA-607, Australia. Eail: Dr_Shasuddin_Ahed@Yahoo.Co ABSTRACT A robust algorith to train neural networ (NN) with individual character presented in pixel forat is described. The output of the trained NN is related with the pixel binary forat. The trained NN perfors the classification based on the original pixel forat code presented to the NN. A apping schee is proposed to deterine the self-adaptive variable learning rates for rapid converge. The proposed character recognition ethod trains at a faster rate than that of the standard bac propagation training ethod and prevents oscillations during training. Key Words: Self-adaptive training, ANN, Dynaic variable learning rates, Character recognition.. Introduction A self-adaptive ANN training schee that identifies variable learning rates is discussed in this study. Consider an ANN error function containing training weights. The actual error function in higher diension is decoposed into lower diensional error function (Ahed et al., 000). The transfored error function retains true convex characteristics of the original error function. The training schee selects the learning rate for a weight paraeter say w j, j=,,.., by a sall factor, w j, such that iproveent in training is noted. Once this phase is over, all the networ weights are then updated and the error function is evaluated.. Character Recognition Proble A training experient is considered to recognize the letters L and T. An ANN with nine inputs, two hidden units and a single output unit is considered for this proble. Each input pattern consists of a 3x3 pixel binary iage of the letter. The training set is fored by four orientations of each letter as shown in Figure and Figure. A 9-- ANN configuration is chosen to copare the results given in Kaarthi and Pittner (999). Figure 3 shows the variable learning rates per epoch. As the training continues the variable learning rates as seen in Figure 4 converges to lower agnitude. Consequently the training converges rapidly. The error function value reduces of the order 0-3 at the early stage of training and further trained at 0-6 tolerance liit for accuracy. Notice that the function converges as the self-adaptive training paraeters gradually reduce to sall agnitude. Figure 5 shows training

2 IACIS 00 CHARACTER RECOGNITION USING A SELF-ADAPTIVE TRAINING perforance with standard bac propagation (BP) training. The training perforance is copared in Figure 6 with the standard BP training. The binary input is passed through the 9-- ANN (Figure 7) input neurons and output neuron provides the necessary signal to recognize the L-T input pattern. It is evident fro Figure 3 that considerable reduction in error function is achieved within epochs to train the proble. The standard BP taes on average 78 epochs to train the character recognition proble, while the proposed ethod taes on average 79 epochs. (a) (b) (c) (d) Figure Four Orientations of the Letter L (a) (b) (c) (d) Figure Four Orientations of the Letter T Self-Adaptive Learning Rates 5.00E E E E E-0.50E-0.00E-0.50E-0.00E E E+00 Figure 3 Convergence with self-adative bac propagation training ethod ( L-T letter recognition) Figure 4 Self-adaptive paraeter generation with self-adaptive bac propagation training ethod 4.00E E E-0.50E-0.00E-0.50E-0.00E E E Self-adaptive BP Standard BP Figure 6 Function evalutions and epoch easure coparison Figure 5 Convergence with standard BP training ethod (L-T letter recognition proble). Identifying self-adaptive learning rates The standard BP training ethod and its variant are not self-adaptive but ad-hoc in training ethod (Fahlan, 988; Hinton, 987; Jacobs, 988; Wier, 99; Vogl, 986 and Kaarthi et al., 999). General references on BP neural networs include Ruelhart, McClelland (989); Muller and Reinhardt (990); and Hertz, Krogh and Paler (99). More discussions on neural networs as statistical odel can be found in Cheng and Titterington (994), Werbos (994), Kuan and White (994), Bishop (995).

3 CHARACTER RECOGNITION USING A SELF-ADAPTIVE TRAINING IACIS 00 Input Layer Hidden Layer Outer Layer Input Pattern i=9 n= o= w no w in Figure 7 Three layered feed forward 9-- ANN To describe the self-adaptive iterative training ethod, an ANN error function f (w) with log activation function is defined in hidden layer. When the algorith is applied to the error function with an initial arbitrary weight vector w, at epoch, the algorith generates a sequence of vectors w +, w +, during epoch +, +,.,.., and so on. The iterative algorith is globally convergent if the sequence of vector converges to a feasible solution set Ω. Consider for exaple the following training proble, where w is defined over the diension E : iniize f (w) () subject to: w E. Let, Ω E be the solution set, and the application of an algorithic interpolation ap, B, generates the sequence w +, w+,..., starting with weight vector w such that ( w, w+, w+,...) Ω, then the algorith converges globally and the algorithic ap is closed over Ω. Consider further that the error function is a iniization proble, since it is possible to train such function (Hayin, 994). Let Ω be a non-epty copact feasible subset of E, and if the algorithic ap B generates a sequence: { w } Ω such that f(w) decreases at each iteration while satisfying f( w ) > f( w + ) > f( w + ),., then the error function f( w ) is a descent function. The expression w defines a sequence (w, w, w 3,.., w n ). In ANN coputation f (w) is assued to posses descent properties, which iplies it is convex in nature (Hyin, 994). 3

4 IACIS 00 CHARACTER RECOGNITION USING A SELF-ADAPTIVE TRAINING Therefore, it is possible to define a descent direction along which the error function can be trained. Following sections deonstrate how the self-adaptive variable learning rates are generated, such that the training direction is always correct. Property Suppose that f : E fi E and the gradient of the error function, f (w), is defined then there is a T directional vector d such that f ( w) d < 0, and f ( w + hd ) < f ( w) :{ h (0, d ), d > 0}, then the vector d is a descent direction of f (w), where d is assued arbitrary positive scalar. A vector defined as directional derivative conceptualizes the direction along which the error function converges to iniu. Also, assue that the error function is sooth and continuous. Therefore the negative gradient of the error function points towards the iniu ANN error space. Next the directional derivative is defined using the following property. Let, f : E + fi E, w E and d is a non-zero vector satisfying ( w + h d) E, h > 0 and fi 0 The directional derivative at location w along the descent direction d is given by: hfi 0 + f ( w+ hd) - f ( w) h f ( w; d) = li. () h. Property Let f : E fi E is a descent function. Consider any training weight w E and d E : d 0. Then the directional derivative f ( w; d) of the error function f (w) in direction d always exists. The central thee behind the self-adaptive BP training is the coputation of the directional search vector d and the learning rate paraeter h in weight space. Since the exact location of the iniu valley is not nown iediately, there is an uncertainty in identifying the boundary or region in weight space over which the training ay explore. The uncertainty can be reduced if it is possible to eliinate the sections of the error function (Kiefer, 953), which do not contain the iniu. An interpolation-training ap can do this operating over the error function in constrained interval. Hence a closed apping is established. Therefore, what is needed is a training schee that explores the constrained region, L, of the error surface. Consider the interpolation ap, B, saples the error surface with discrete length in a given direction. Few definitions are needed to describe the interpolation-training ap. For convenience define the following expression to update the ANN connection weights: u w + = w h d + or u = w + hd. (3) Now consider an ANN training proble expressed as: h arg{ in f ( w + hd )} (4) subject to : h L where, L defines a closed interval L = { h : h E } over the error function. 4

5 CHARACTER RECOGNITION USING A SELF-ADAPTIVE TRAINING IACIS 00 Property 3 The interpolation ap that coputes learning rates is defined in restricted error space written as: A : E E fi E such that: A( w, d) = { u : u = ( w + h d ) h} L (5) f ( u) = in f ( w + h d) (6) Subject to : h h L. Suppose that the algorith B operating on f(w) produces descent directions. The correct learning rate is obtained during the training phase, when the ap is closed as set value apping (Luenberger, 984). It is, therefore, now possible by any standard technique; though difficult; to identify learning rate, η, for each epoch while descending along the iniu trajectory and exploring all the diensions. The error function as a result onotonically converges to a iniu value. The algorith gradually converges to the acceptable iniu liit in few epochs.. Training One of the ost difficult coputational steps in identifying the trained weights in ANN is that the ANN error function contains several local inius within the reasonable range of w j. The reduced ANN error function can be considered a continuous function of training weights w j, j=,,3,.,, describing a hyper surface in diensional space (Moller, 997). Suitable reduction of the error surface constitutes a parabolic hyper surface, and the learning rates are deterined fro the reduced function with initial rando weight estiated at the beginning of training. As a result, few epochs are needed to converge to the iniu trajectory of the error function. 3. Character recognition proble Ten siulation experients are carried out to test the proposed training ethod against the standard BP training ethod (Ruelhart et al., (989) with sall rando starting weights. Siulation results are shown in Table and Table. The sall rando starting weights are used to initiate the training. Table shows the result against the standard BP training ethod. The average nuber of epoch with the self-adaptive bac propagation ethod is The axiu and iniu values in nubers of epoch are 33 and 53 respectively. The average terinal function value is.8x0-9, which indicates that the ANN is trained to produce accurate results. The average nuber of epoch is 77.7 with the standard BP training. The axiu and iniu nubers of epoch are 365 and 80 respectively. The relative efficiency of the self- 5

6 IACIS 00 CHARACTER RECOGNITION USING A SELF-ADAPTIVE TRAINING adaptive bac propagation-training algorith over the standard BP-training algorith in nuber of epoch is (77.7/78.8).8. Table shows the coparison with different training ethods including the result given in Kaarthi et al. (999). The proposed ethod perfors better than the weight extrapolation ethod suggested by Kaarthi et al. (999). They train L-T letter recognition proble with 8 epoch and the terinal function value is at the end of training. The self-adaptive training ethod finds terinal function value, which is coparatively less than the weight extrapolation and standard BP training ethod. L-T Letter Recognition Expt # Self-adaptive bac propagation Standard bac propagation Mean E Median E Standard E Deviation Range E Miniu E E-05 Maxiu E L-T Letter Recognition Measure Perforance Measures Self-adaptive BP Standard Bac Propagation Kaarthi et al. (999) Weight extrapolation Average S. Deviation Max Min Speed up Table Training perforance with 9- - ANN: L-T letter recognition proble Table Coparison with standard bac propagation and self-adaptive training ethod (9-- ANN: L-T letter recognition proble) CONCLUSION A strictly self-adaptive training schee to train forward ANN is shown. The iportant feature of this ANN training is that the learning rates are dynaically coputed each epoch by an interpolation ap. The ANN error function is transfored into a lower diensional error space and the reduced error function is eployed to identify the variable learning rates. As the training progresses the geoetry of the ANN error function constantly changes and therefore the interpolation ap always identifies variable learning rates that gradually reduces to a lower agnitude. As a result the error function also reduces to a saller terinal function value. It iproves over the BP training ethod and the weight interpolation ethod deonstrated by Kaarti et al., (999) in character recognition proble. The proposed self-adaptive training ethod needs 79 epochs, while the standard BP ethod needs 78 epoch and the weight extrapolation ethod needs 8 epoch to train the L-T character recognition proble with 9-- ANN configurations. REFERENCES Ahed, S., and Cross, J., (000). Convergence in Artificial Neural Networ without Learning Paraeter, Second International Coputer Science Conventions on Neural 6

7 CHARACTER RECOGNITION USING A SELF-ADAPTIVE TRAINING IACIS 00 Coputations, 000, Berlin, Gerany Ahed, S., Cross, J., and Bouzerdou, A. (000). Perforance Analysis of a new ulti-directional training algorith for Feed-Forward Neural Networ, World Neural Networ Journal, Bishop, Christopher M. (995). Neural Networs for Pattern Recognition. Oxford, UK: Clarendon. 4 Cheng, Bing and D. M. Titterington. (994). Neural Networs: A Review Fro a Statistical Per-spective. Statistical Science 9 (): Fahlan, S.E., (988). Faster-learning variations on bac-propagation: an epirical study, in Touretzy, D., Hinton, G. and Sejnowsi, T. (Eds), Proceedings of the Connectionist Models Suer School, Morgan Kaufann, San Mateo, CA, Hayin, Sion., (994). Neural Networs: A Coprehensive Foundation., Macillan College Publishing, Hertz, John, Anders Krogh, and Richard G. Paler. (99). Introduction to the Theory of Neural Coputation. Reading, MA: Addison-Wesley 8 Hinton, G. E., (987). Learning Translation Invariant Recognition in Massively Parallel Networs. In De Baer, J.W., Nijan, A.J., and Treleaven, P.C. (Eds.), Proceedings PARLE Conference on Parallel Architectures and Languages Europe, - 3. Berlin: Springer-Verlag. 9 Jacobs, R.A. (988). Increased Rate of Convergence Through Learning Rate Adaptation. Neural Networs,, Jang, J.-S.R., Sun, C.-T., Mizutani, E., (997). Neuro-Fuzzy and Soft Coputing, Prentice Hall International, Inc, NJ, USA. Kaarthi, S.V., and Pittner, S., (999). Accelerating Neural Networ Training Using Weight Extrapolations, Neural Networs,, Kiefer, J., (953). Sequential Miniax search for a Maxiu. Proceedings of the Aerican atheatical Society, 4, Kuan, Chung-Ming and Halbert White. (994). Artificial Neural Networs: An Econoetric Per-spective. Econoetric Reviews 3 (): Luenberger, D. G., (984). Introduction to Linear and Nonlinear Prograing., nd ed. Addision-Wesley, Reading, Mass., USA. 5 Moller, Martin., (997). Efficient training of feed forward neural networs, in Brown Anthony, editor, Neural Networ: Analysis, Architectures and Applications,: Institute of Physics Publishing, Bristol and Philadelphia. 6 Muller, Berndt and Joachi Reinhardt. (990). Neural Networs : An Introduction. Berlin: Springer. 7 Ruelhart, D.E. and McClelland, J.L. (989), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. I, MIT Press, Cabridge, MA. 8 Vogl, T. P., Mangis, J.K., Rigler, A.K., Zin, W.T., and Alon, D.L., (988). Accelerating the Convergence of the Bac-Propagation Method. Biological Cybernetics, 59, Weir, M.K., (99). A Method for Self-Deterination of Adaptive Learning Rates in Bac Propagation. Neural Networs, 4, Werbos, Paul. (994). The Roots of Bac propagation: Fro Ordered Derivatives to Neural Net-wors and Political Forecasting. New Yor: John Wiley. 7