COMPARISON OF FUZZY ARTMAP AND MLP NEURAL NETWORKS FOR HAND-WRITTEN CHARACTER RECOGNITION

Transcription

1 COMPARISON OF FUZZY ARTMAP AND MLP NEURAL NETWORKS FOR HAND-WRITTEN CHARACTER RECOGNITION Yong Hur Ty nd Mrzuki Khlid* Centre for Artificil Intelligence nd Rootics (CAIRO) Fculty of Electricl Engineering, Universiti Teknologi Mlysi, Jln Semrk, 5400 Kul Lumpur e-mil: (* ll correspondence should e sent to) Astrct --- Fuzzy ARTMAP is one of the recently proposed neurl network prdigm where the fuzzy logic is incorported. In this pper, we compre the Fuzzy ARTMAP neurl network nd the well-known ck-propgtion sed Multi-lyer perceptron (MLP), in the context of hnd-written chrcter recognition prolem. The results presented in this pper shows tht the Fuzzy ARTMAP out-performs its counterprt, oth in lerning convergence nd recognition ccurcy. Keywords --- Neurl network, Pttern recognition, Chrcter recognition, Adptive resonnce theory, Fuzzy ART, Fuzzy ARTMAP, Bck-propgtion, Multilyer perceptron. INTRODUCTION Chrcter recognition prolem hs long een identified s the prolem tht computer cn hrdly solved however humn-eings tke this for grnted. There hs lredy een sustntil reserch undertken to solve this prolem. Neurl network is one of the techniques tht hs een used for pttern recognition since 950s s it hs etter performnce thn other sttisticl nd rtificil intelligence techniques[3]. Neurl networks re preferred for pttern recognition prolems ecuse of their prllel processing cpilities s well s lerning nd decision-mking ilities. Fukushim [4] hs developed Neocognitron tht could recognize shift, size, rottionl ptterns. Shustotovish nd Thrsher [7] developed system tht works with fields of chrcters. The rnge of pplictions for chrcter recognition includes postl code reding, utomtic dt entry, recognition of print nd script, utomted crtogrphy, nk, nd reding services for the lind. This pper hs een orgnised s follows. In the next section, we develop the ckground informtion needed for the two rchitectures. Next, we discuss the ckpropgtion (BP) lgorithm used in the MLP nd the Fuzzy ARTMAP lgorithm. As the Fuzzy ARTMAP hs just een recently proposed[], it would e interesting to mesure its performnce s compre the well-estlished BP lgorithm in the context of chrcter recognition prolem. We lso discuss the hnd-written chrcter feture sets which we used in the experiments. The next section descries the experiments tht were performed nd the results. The finl section summrises our results nd conclusions. 2. MULTI-LAYER PERCEPTRON (MLP) The MLP is supervised neurl network. It cn hve multiple inputs nd multiple outputs nd multiple lyer of nodes/neurons. Figure shows n rchitecture of threelyered perceptron. The leftmost lyer is the lyer to which the input dt is supplied; the rightmost lyer is the output lyer nd the middle lyer is the lyer to interconnect input nd output lyers. Ech lyer of the network is fully

2 interconnected to its susequent higher lyer. The links etween ech neuron re clled weights, where the knowledge is eing stored. I i O i θ j O j θk O k where normlly f(x) is sigmoidl function s follows: f(x) = exp( x) (5) The output vector generted from the feed-forwrd process is then compred with the desired output vector. The cost function used here is squred error function ξ, which is given y Input lyer w kj w ji net j Hidden lyer net k Output lyer ξ = ξ p (6) p Figure. Exmple of MLP. From Figure, O k, O j nd O i re output vlues of output, hidden nd input lyers, respectively. w kj denotes the weights etween output nd hidden lyers, w ji denotes the weights etween hidden nd input lyers. The MLP utilises the ck-propgtion (BP) lgorithm for trining[8]. The lgorithm consists of 2 phses: feedforwrd process nd ck-propgtion process. For the initil stge, the weights of the network re rndomly selected. The lerning rte h nd momentum is pre-set efore the lerning phse. During the lerning phse, n input vector is presented to the network nd the vector propgtes from the input lyer to the output lyer. Thus, the output of the hidden lyer hs the following nottion, O j = f( net j ) () net j = w jioi θ j O i = I i (2) i 2 ξ p = ( τ ) 2 pk O pk (7) k where τ pk is the desired output for the kth component of the output pttern for pttern p nd O pk is the corresponding ctul output. Bck-propgtion process is then using the error to djust weights ccordingly, sed on the steepest descent method, s follows: where nd w kj (t) = ηδ k O j α w kj (t) (8) δ k = O k ( - O k ) (τ k - O k ) (9) w ji (t) = ηδ j O i α w ji (t) (0) similrly, the output lyer ecomes where O k = f( net k ) (3) δ j = O j ( - O j ) (τ j - O j ) δ k w kj () k net k = w O kj j θ k (4) j Configurtion for the MLP neurl network used in the experiment is (4-lyer neurl network with 64 inputs nd 8 outputs, 2 intermedite lyers ech with 5

3 neurons). The lerning rte, h, is 0.7 wheres momentum,, is 0.4. fetures of the Fuzzy ART rchitecture. The two mjor susystems re the ttentionl susystem nd the orienting susystem. 3. ADAPTIVE RESONANCE THEORY (ART) Adptive Resonnce Theory (ART) ws developed y Crpenter nd Grosserg for pttern recognition purpose[5]. There re lot of vrition of ART, ART, ART2, ART3, Fuzzy ART, Fusion ART, Fuzzy ARTMAP, ART-EMAP, just to nme few. The mjor distinct dvntges of ART neurl networks re :- The ttentionl susystem is used to recognise nd clssify previously lerned ptterns. The orienting susystem is invoked when ART encounters new pttern. Under these circumstnces, the orienting susystem shuts down ny ttempt to continue mtching with stored ptterns, nd sets up new ctegory to respond to the new type of pttern. Thus, the ttentionl susystem performs the ctul pttern recognition, nd the orienting susystem cts on the ttentionl susystem to enle it to respond to novel ptterns. It retins knowledge of previously lerned ptterns or ptterns ctegories (STABILITY). It cn lso lern new ptterns (PLASTICITY). ART hs shown tht it cn solve the stility-plsticity dilemm which other feed-forwrd neurl networks cnnot solve. In other words, ART is le to lern mny new things without forgetting things lerned in the pst. Wheres MLP cnnot lern new informtion incrementlly without forgetting old informtion unless it is retrined with the old informtion long with the new. ART is type of competitive lerning network, nd suitle for oth ctegories (pttern) formtion nd recll (recognition). When n input pttern is dequtely similr to one stored in the ART s long term memory (LTM), ART recognises the ptterns s elonging to the ctegory, nd modifies (generlise) the stored ctegory to ccommodte new fetures of the current input pttern. When n input pttern is not dequtely similr to ny stored ctegory, pttern formtion occurs. ART selects n uncommitted (new) ctegory to store the current input. If no more uncommitted nodes re left, the current input hs no response. Hence, stored ctegories remin stle to irrelevnt inputs nd yet re sensitive to novel fetures of inputs. In such cse, ART mkes stisfctory trde-off etween stility nd plsticity. 3.. Fuzzy ART Architecture Fuzzy ART is vrition of ART system derived from the first genertion of ART, nmely ART. It is synthesis of ART nd fuzzy logic. Unlike ART which cn only ccept inry input ptterns, Fuzzy ART llows oth inry nd continuous input ptterns. Figure 2 shows the sic Ptterns of ctivity develop over the neurons in the two lyers of the ttentionl susystem which re clled short term memory (STM) ecuse they exist only in ssocition with single ppliction of n input pttern. The weights tht interconnect etween F nd F 2 lyers re clled long term memory (LTM) trces ecuse they encode informtion tht remins prt of the network for n extended period. Gin Control Gin Control F 2 F Attentionl Susystem y j - xi Input wji Figure 2. Fuzzy ART Architecture Fuzzy ART Dynmics - Orienting Susystem Reset Signl The input pttern with the length of M is denoted s I = (I,..., I M ) where I i [0, ]. The F ctivity vector is denoted s x = ( x,..., x M ) nd the F 2 ctivity vector with N numer of ctegory is denoted s y = ( y,..., y N ). M nd N is ritrry. Associted with ech F 2 ctegory node j (j =,...,N) is vector w j ( x j,..., x jm ) of dptive weights, or LTM trces. Initilly, ech ctegory is uncommitted, w j (0) =... = w jm (0) = ; (2)

4 After ctegory is chosen for coding it ecomes committed. Next, the Fuzzy ART does top-down templte mtching, when the F ctivity vector x oeys the following eqution Fuzzy ART dynmics re determined y choice prmeter > 0; lerning prmeter [0, ]; nd vigilnce prmeter r [0, ]. x= I if F 2 is inctive I w if the Jth F node is chosen J 2 (8) When Fuzzy ART receives n input pttern I, I is copied immeditely over into x. The next step is to pss the informtion in lyer F up to F 2, with the choice function, T j, is defined s The top-down templte is compred with the initil input pttern. If the pttern is mtched to within specified vigilnce criterion r, then resonnce occurs. T j (I) = I wj α wj, (3) I w I J ρ ; (9) Mismtch reset occurs when where the fuzzy AND opertor is defined y (p q) I min(p i, q i ) (4) I w I J < ρ (20) nd the norm is defined y M p pi i= (5) Next the F 2 winning nodes, T J is inhiited for the durtion of the input representtion to prevent it from competing further. A new index J is then chosen, y (6). The serch process continues until the chosen J stisfies (9). Once eqution (9) is fulfilled, the weight vector w J is modified to encode the pttern for ny M-dimensionl vectors p nd q. For nottionl simplicity, T j (I) is written s T j. w J (new) = β ( I w J (old) ) ( - β ) w J (old) (2) These vlues of T j in F 2 undergo competitive process. Only one vlue of T j will win. The ctegory choice is indexed y J, where T J = mx{t j : j =... N }. (6) If more thn one T j is mximl, the smllest index is chosen. The ctivity vector y in F 2 is 3.2. Fuzzy ARTMAP Architecture ARTMAP (Figure 3) is clss of neurl network rchitectures tht perform incrementl supervised lerning of recognition nd multidimensionl mps in response to inry input vectors presented ritrrily[]. In turn, the Fuzzy ARTMAP extends the ARTMAP y integrting fuzzy logic into the system, which llows it to ccept input vector vlues etween 0 nd. The generlistion is done y replcing oth ART modules (ART nd ART ) of the inry ARTMAP system with fuzzy ART modules. for i = J y i = i =... N (7) 0 for i J The winning node y J is wht the Fuzzy ART thinks is its est mtch for the input pttern. Fuzzy ARTMAP consists of two fuzzy ART modules (ART nd ART ) which re linked together vi n inter-art module, F. During the lerning phse, the input vector I 0, is presented to the ART nd the desired output vector O 0, is presented to the ART. The ART nd ART modules clssify the input nd desired output vector into ctegories,

5 then the mp field (inter-art module) mkes ssocitions from ART ctegory to ART ctegory. If I 0 predicts n incorrect O 0, then mechnism clled mtch trcking is triggered. This mechnism increses the vigilnce prmeter of ART, r, y minimum vlue nd, hence, force the ART module to serch for nother ctegory suitle to e ssocited with the desired output vector. r is then set ck to the seline vigilnce prmeter, ρ, for every step of lerning tril...., y N ). M nd N re lso ritrry. For the inter-art module, F output vector is denoted y x = ( x,..., x N ), nd the weight vector from the jth F 2 node to F is denoted y w j = ( x j,..., x jn ). Vectors x, y, x, y, x re initilised to 0 etween input presenttions. Initilly, the weight vectors w re set to (uncommitted). The inter-art module F is ctivted whenever ny of the ART or ART ctegories is ctive Fuzzy ARTMAP Dynmics The inputs to ART nd ART re in complement code form, it is necessry for the successful opertion of Fuzzy ARTMAP[]. For ART, if the input is I 0 = (,..., M ), then x = y w wj y 0 J oth the ART nd ART is ctive ART is ctive, ART is inctive ART is inctive, nd ART is ctive oth the ART nd ART is inctive (24) where I = (, c ) = (,..., M, c,..., M c ) (22) i c = - i i M (23) During lerning phse, ART vigilnce prmeter r, equls the seline vigilnce, ρ t the eginning of every input presenttion. When Fuzzy ARTMAP receives n input/output pir (I 0 /O 0 ), ART chooses the Jth node of F 2 z nd ART chooses the Kth node of F 2. When oth ART nd ART re ctive nd x 0, then the input/output pirs re ssocited with the eqution The F ctivity vector is denoted y x = ( x,..., x 2M ) nd the F 2 ctivity vector with N numer of ctegory is denoted y y = ( y,..., y N ). M nd N re ritrry. For ART, O 0 = (,..., M ), O = (, c ). The F ctivity vector is denoted y x = ( x,..., x 2M ) nd the F 2 ctivity vector with N numer of ctegory is denoted y y = ( y, w jk = 0 j = J nd k = K otherwise (25) Inter-ART module ART module F ART module F 2 reset ρ F 2 reset F F I = (I0, I0 c ) F 0 ρ mtch trcking O = (O 0, O0 c ) F 0 ρ I 0 O 0 Figure 3. Simplified Fuzzy ARTMAP Architecture.

6 If x = 0, then there is mismtch. Inter-ART module triggers mtch trcking mechnism. This mechnism increses r, y minimum vlue nd hence, forces the ART module to serch for nother ctegory suitle to e ssocited with the desired output vector. 4. CHARACTER FEATURES The input dt of hnd-written chrcter hs resolution of 32 x 32 pixels with lck/white colour. In order to reduce the numer of inputs to the neurl network, we develop simple pre-processing technique. We segmentise the itmp into n 8 x 8 region (Figure 4) with ech region tking up 4 x 4 pixels. Figure 5. 8x8 itmp fter the compression The vlue of ech region is then pssed on to the neurl network for clssifiction. The pre-processing significntly reduced (from 32x32=024 down to 8x8=64) the input vector to the neurl network nd yet retined most of the fetures of the chrcter. Some smples of the hnd-written chrcter used in the experiments re given in Figure 6. Figure 4. A 32x32 itmp segmented into 8 x 8 region. Ech region hs vlue etween 0 nd. The vlue of the region, v oeys the eqution where v = M * N M N x= 0 y= 0 f ( x, y) (26) M denotes the numer of pixels in one row = 4 Figure 6. Smples of hnd-written chrcter used in this experiment. 5. EXPERIMENTAL SET-UP The dtse consists of more thn 500 smples of lck nd white hnd-written chrcters. The resolution of ech chrcter is 32 x 32 pixels. The smples re seprted into 2 sets of trining nd testing dt. Trining dt set is sugrouped into 6 wheres testing dt set is su-grouped into 3 ctegories. N denotes the numer of pixels in one column = 4 f(x,y) denotes the inry vlue of pixel t position (x, y) The result of the pre-processing is shown s follows :- 5.. Convergence Rte In this experiment, the convergence rtes of oth types of neurl networks re studied to mke comprison etween the two neurl networks.

7 Ech neurl network is trined with 6 groups of dt from the trining dt set, nd the time for the networks to converge to stle stte is logged. One full presenttion of the trining ptterns to the networks is considered s one epoch. 6. CONCLUSIONS In this pper, we hve shown tht the Fuzzy ARTMAP neurl network rchitecture offers distinct dvntges over the MLP neurl network. These dvntges re: For the Fuzzy ARTMAP, stle stte mens the network does not select new/uncommitted ctegory, while for the MLP, it simply mens tht the men squre error (m.s.e) hs chieved minimum vlue of 0.. Results re shown s in Tle 3. Fster convergence rte. Our results show tht the Fuzzy ARTMAP trining lgorithm requires much less trining itertion s compred to the MLP neurl network Recognition Accurcy All the trined neurl networks re then tested with the dt stored in testing dt sets. The neurl networks trined with groups nd 2 re tested with test group. The networks trined with groups 3 nd 4 re tested with test group 2. The networks trined with groups 5 nd 6 re tested with test group 3. The ccurcy of the network recognition is then logged. Better recognition ccurcy. The Fuzzy ARTMAP cn recognise the chrcters more ccurtely when compred to the MLP. Online lerning cpility. This cpility hs mde Fuzzy ARTMAP to lern new pttern without hving to retrin the network. Tle 4 shows the result of the recognition ccurcy test. The result shows tht the Fuzzy ARTMAP cn recognise more ccurtely when compred to the MLP. It lso shows tht oth networks hve etter performnces if presented with more trining ptterns (compring etween trining groups nd 2, 3 nd 4, 4 nd 5). The networks performed worse if the clsses of chrcter re more (compring etween trining group,3, nd 5, 2,4 nd 6) Online Lerning For the MLP, the network hs to e retrined if new clss of pttern is required to lern[3]. The Fuzzy ARTMAP cn lern new pttern without retrining. In the experiment, the network is initilly trined with trining group 2 tht hs 5 clsses of chrcters. The recognition ccurcy of the network to test group 2 tht hs 0 clsses of chrcters re then recorded. The sme network is then trined with trining group 7 tht hs nother 5 clsses of chrcter, nd the recognition ccurcy of the network to test group 2 re then recorded. The result of the experiment is shown in Tle 5. It cn e oserved tht for the network tht is trined with only group 2 hs the ccurcy of 48.8%. The ccurcy incresed to 88.8% if the sme network is eing trined with group 7 which hs nother 5 clsses of chrcters.

8 Tle. Groups in Trining Dt Set. Group Clsses of chr. Occur. / Clss Totl chr Tle 2. Groups in Testing Dt Set. Group Clsses of chr. Occur. / Clss Totl chr Tle 3. Comprison of the convergence rte etween Fuzzy ARTMAP nd MLP. The tle shows tht convergence rte for the Fuzzy ARTMAP neurl network is much fster thn the MLP. Trining Group Fuzzy ARTMAP MLP epoch Time (ms) epoch Time (ms) Tle 4. Comprison of recognition ccurcy etween the Fuzzy ARTMAP nd MLP. Trin with group Test with group Fuzzy ARTMAP MLP Totl numer of dt Mtch Accurcy (%) Mtch Accurcy (%) Tle 5. Result of the online lerning of Fuzzy ARTMAP. Trin with group Test with group Totl numer of dt Mtch Accurcy (%) ,then

9 REFERENCES [] G. A. Crpenter, S. Grosserg, N. Mrkuzon, J. H. Reynolds, nd D. B. Rosen, Fuzzy ARTMAP: A Neurl Network Architecture for Incrementl Supervised Lerning of Anlog Multidimensionl Mp, Neurl Networks, vol. 3, 992, pp [2] G. A. Crpenter nd S. Grosserg, A Mssively Prllel Architecture for Self-Orgnizing Neurl Pttern Recognition Mchine, Computer Vision, Grphs, nd Imge Processing, vol. 37, 987, pp [3] Clifford R. Prten... et l., Hndook of Neurl Computing Applictions, Acdemic Press, Inc. SD, CA, USA. [4] K. Fukushim, Neocognitron: A hierrchicl neurl network cple of pttern recognition, Neurl Networks, vol., 988, [5] J.A. Freemn, Simulting Neurl Networks with Mthemtic, Addison-Wesley, USA. [6] KulKrni, Arun D, Artificil neurl networks for imge understnding, ITP, USA. [7] A. Shustorovich nd C.W. Thrsher, Neurl Network Positioning nd Clssifiction of Hndwritten Chrcters, Neurl Networks, Vol. 9, No. 4, 996, pp [8] D.E. Rumelhrt et l., Prllel Distriuted Processing: Explortions in the Micro Structure of Cognition Vol. I: Foundtions, MIT Press, Cmridge, MA, 986.