Alphanumeric Character Recognition

Transcription

1 Amerian Journa of Signa Proessing. 0; (): DOI: /j.ajsp Training Tangent Simiarities with N-SVM for Aphanumeri Charater Reognition Hassiba Nemmour *, Youef Chibani Signa Proessing Laboratory, Fauty of Eetroni, University of Houari Boumediene, Agiers, Ageria Abstrat This paper proposes a fast and robust system for handwritten aphanumeri harater reognition. Speifiay, a neura SVM (N-SVM) ombination is adopted for the assifiation stage in order to aeerate the running time of SVM assifiers. In addition, we investigate the use of tangent simiarities to dea with data variabiity. Experimenta anaysis is onduted on a database obtained by ombining the we known USPS database with C-Cube upperase etters where the N-SVM ombination is evauated in omparison with the One-Against-A impementation. The resuts indiate that the N-SVM system gives the best performane in terms of training time and error rate. Keywords Handwritten Aphanumeri Charaters, Svms, Tangent Vetors. Introdution In various appiations of doument anaysis, aphanumeri harater reognition onstitutes a very important step. For instane, in bank hek proessing this task is required for date reognition whie in automati mai sorting it is used to reognize addresses. However, the fat that haraters are written in various manners by different sripts and different toos onduts to high simiarity between some upperase etters and digits suh as Z and or O and 0. Due to this ambiguity, assifiation systems fai ommony in disriminating upperase etters from digits. Besides, in handwriting reognition robust assifiers shoud be used to ahieve a satisfatory performane sine onventiona ones provide systematiay insuffiient reognition rates[]. In the past reent years, attentions were foused on earning mahines suh as neura networks and hidden Markov modes[]. Currenty, Support Vetor Mahines or SVMs are the best andidate for soving a handwriting reognition tasks with medium number of asses[3]. Speifiay, SVMs were used for handwritten digit reognition in many researh works[4,5]. In this work, we investigate their use for soving an aphanumeri harater reognition whih aims to disriminate upperase Latin etters from digits. However, it is obviousy known that suh appiation handes ommony arge sae databases whereas the SVM training time is quadrati to the number of data. For this reason, we investigate aso, the appiabiity of N-SVM ombination for aphanumeri assifiation. N-SVM is a neura-svms ombination * Corresponding author: hnemmour@yos.om(hassiba Nemmour) Pubished onine at Copyright 0 Sientifi & Aademi Pubishing. A Rights Reserved whih was introdued for handwritten digit reognition in order to redue the runtime and improve auray[6]. Furthermore, due to the arge variabiity of aphanumeri haraters the use of feature extration shemes is a prerequisite to reah suffiient reognition auraies. Speifiay, feature extration an aeviate some imitations reated to sript variations as we as to some segmentation errors. Many handwriting reognition researh works have shown that earier desriptors whih were extensivey used for pattern reognition suh as geometri moments and generi Fourier desriptors annot dea with haraters ambiguity[7,8]. Thereby, more effiient desriptors were deveoped to aow invariane with respet to some variations of data. Reenty, some features based on urvature or ontour information were introdued for haraters reognition. Among them, we note the ridgeet transform whih showed high disrimination abiity for printed Chinese haraters[9]; urvature features for handwritten digit reognition[0] and waveet pakets for handwritten Arabi word reognition[]. In fat, a good desriptor shoud be invariant with respet to some affine transformations suh as sma deformations, transations and rotations whie being variabe from a ass to other. In this framework, tangent vetors whih are based on invariane earning onstitute one of the best desriptors for handwritten digit reognition[,3]. Tangent vetors are omputed by performing a set of affine transformations to eah pattern. Then, assifiers are trained on the generated tangent sampes to produe invariant deision funtions. Sine the resuts for handwritten digits were very promising, presenty, we try to inorporate tangent vetors into the proposed aphanumeri harater reognition system. Speifiay, tangent simiarities based on ass-speifi tangent vetors are used for both desribing variabiity and reduing the size of data[4]. The rest of this paper is arranged as foows. Setion

2 Amerian Journa of Signa Proessing. 0; (): gives a brief review of SVM assifiers. Setion 3 desribes the N-SVM arhiteture whie setion 4 presents the tangent vetor based simiarities. The ast setions summarize the experimenta resuts and give the main onusions of the paper.. Support Vetor Mahines Support Vetor Mahines (SVMs) were designed to onstrut binary assifiers from a set of abeed training sampes defined by: ( x i, yi ) R N { ±}, where i =, ( is the number of training data). SVMs seek the inear separating hyperpane with the argest margin by soving the foowing optimization probem[3,5]: T Minimize w w () Subjet to yi ( xi w + b) i () T denotes the transpose, b is a bias whie w is the norma to the hyperpane. When inequaities in (3) do not hod for some data, the SVM is non-ineary separabe. Then, the margin of separation is said to be soft and non-separabe data are handed by introduing a set of nonnegative sak variabes { ξ i } into the deision surfae[3]. Then, the goa is to find a hyperpane whih minimizes misassifiations whie maximizing the margin of separation suh that: T Minimize ww+ C ξi (3) i= Subjet to yi ( xi w + b) ξi (4) C is a user-defined parameter that ontros the tradeoff between the mahine ompexity and the number of non-separabe data[4]. Commony, a dua Lagrangian formuation of the probem in whih data appear in the form of dot produts, is used: Maximize LD = αi αα i jyyx i j i xj (5) Subjet to i i= i, j α iyi = 0 (6) where α i are Lagrange mutipiers. The dua probem is usefu when data are not ineary separabe in input spae. In suh a ase, they are mapped into a feature spae via a kerne funtion suh that: K( xi, x) = φ( xi ), φ( x). The dua probem beomes: LD = αi αα i jyyk i j ( xi, xj) (7) i Thereby, the deision funtion is expressed in terms of kerne expansion as: Sv i= i, j f( x) = αiyk i ( xi, x) + b (8) Sv is the number of support vetors whih are training data for whih 0 < α j C. The optima hyperpane orresponds to ( x) = 0 f whie test data are assified aording to: ( x) ( x) positive ass if f > 0 x (9) negative ass if f < 0 A mathematia funtions whih satisfy Merer s onditions are eigibe SVM-kernes. The simiarity between data is expressed either in the form of a dot produt or distane measure. The most ommony used kernes in pattern reognition are isted in Tabe (). Noyau Poynomia: Po Sigmoïde: Sig RBF Distane Négative: ND pp, ss 0, ss, σσ, γγ: are user defined. Tabe. Some SVM kernes KK xx ii, xx jj xx ii xx jj + pp tttttth ss 0 xx ii xx jj + ss eeeeee xx ii xx jj σσ xx ii xx jj γγ Furthermore, various approahes were proposed to extend SVMs for muti-ass assifiation probems[6]. The One-Against-A (OAA) is the eariest and the most ommony used impementation. For a C -ass probem, It performs M SVMs eah of whih is designed to separate a ass from a the others. The th SVM is trained with a of the exampes in the th ass with positive abes, and a other exampes with negative abes whih eads to a omputationa time approximatey about C. Then, data are assigned to the positive ass with the highest output as: C arg max f x (0) = ( ) Unfortunatey, the time required for training the OAA-based SVM is a imitation for arge sae databases. Rea that the training of a SVM is quadrati to the number of data[7]. This means that for OAA this time is mutipied by the number of asses. Thereby, for aphanumeri assifiation one must dea with this time ompexity. Presenty, we adopt a Neura-SVM (N-SVM) ombination whih aims to aeerate the runtime of SVMs. Sine this ombination was introdued for handwritten digit reognition we try to extend its use for aphanumeri harater reognition. The N-SVM is briefy presented in the foowing setion. 3. N-SVM Combination The goa of N-SVM ombination onsists of reduing the runtime required by standard SVMs. This arhiteture was proposed for handwritten digit reognition where 5 dihotomies are randomy seeted from the 45 possibe dihotomies between numera asses. Then, outputs of the SVMs trained over the seeted dihotomies are handed by a neura network to produe an automati deision about asses. The resuts obtained for USPS database highighted the vaidity of this approah to redue the runtime and improve the reognition auray[6]. Thus, for aphanumeri

3 36 Hassiba Nemmour et a.: Training Tangent Simiarities with N-SVM for Aphanumeri Charater Reognition harater reognition, whih is a probem of 36 asses (0 digits and 6 upperase etters), we deveop 8 SVMs designed to independenty separate pairs of asses. In summary, the N-SVM ombination is omposed of the foowing steps: From the C (C-)/ possibe dihotomies (where C=36): a) Seet randomy a dihotomy and assoiate its asses Ci and C j to a SVM. b) Deete asses C i and C j from the set of asses. ) Return to ) if the number of seeted pairs is ess than C/. d) Train SVM assifiers over the seeted pairs of asses. e) Train a feed-forward neura network over SVM outputs. The neura network is MutiLayer Pereptron (MLP) with a singe hidden ayer. The input ayer ontains 8 nodes, whih reeive SVM outputs whereas eah node in the output ayer orresponds to a numera ass from 0 to Z. On the other hand, the number of hidden nodes is fixed experimentay. The MLP is trained by the standard bak propagation agorithm inuding the momentum fator[8]. In addition, N-SVM system reeives feature vetors whih are onstituted by tangent simiarities with respet to a asses. The omputation of suh simiarities is detaied in the foowing setion. 4. Tangent Vetor Based Simiarities for SVMs A priori knowedge was initiay introdued to SVMs through the VSV approah whih generates virtua sampes from support vetors to enfore invariane around the deision boundary[9]. Indeed, invariane earning is based on the idea of earning sma or oa transformations, whih eave the ass of data unhangeabe. To define these transformations, et ~ x( β ) denotes a transformation of a pattern x that depends on a set of parameters L β = { β,, β L} R. The inear approximation of ~ x ( β ) using a Tayor expansion around β = 0 an be written as[4]: L L ( ) = x + β + ο( ) = v ~ x β β () = v are Tangent Vetors (TV) orresponding to partia derivatives of the transformation ~ x with respet to β = ( =, L) so that : ~ x( β ) v = β = 0 () β Sine terms of seond order and higher in () are ommony negeted, ~ x ( β ) = x for β = 0 whie for sma vaues the transformation does not hange the ass membership of x. Hene, the inear approximation desribes transformations suh as transation, rotation and axis deformations by one prototype and its orresponding tangent vetor. The first use of the TV was introdued through the so-aed Tangent distane with the K-NN assifier[]. This distane gave very good preisions for handwritten digit reognition but its time ompexity was very important. Aways for handwritten digit reognition, TV were used with SVM assifiers by introduing the Tangent distane into distane-based kernes suh as RBF and negative distane[0]. However, the runtime of SVMs was signifianty extended. More reenty, a simiarity and dissimiarity measures were introdued in order to aow using tangent vetors with a SVM kernes[]. These measures are based on ass-speifi tangent vetors whose auation is faster ompared to the onventiona sheme. Then, the time ompexity is redued but the runtime is sti extended. In the present work, we aim to introdue the TV onept for aphanumeri harater reognition without extending the runtime of SVMs and N-SVM. Speifiay, we empoy tangent simiarities whih are based on ass-speifi TV, as data features. The ass-speifi TV are omputed as foows[4]. For a set of asses { =, C} with training data x n { n =, N C }, tangent vetors maximizing the dissimiarity between asses are hosen suh that are the eigenvetors with argest eigenvaues { Σ } v of the matrix : Σ (3) S ( Σ ) T Σ : Covariane matrix omputed from the fu dataset. S : Cass dependent satter matrix given by: N S = x n= ( )( ) T x µ n µ (4) µ : mean of the ass. It is straightforward to not that the number of TV shoud be obviousy fixed by the user. In addition, eah tangent vetor refers to a transformation or speifi variabiity knowedge whose number shoud be a priori hosen by the user. So, one estimated tangent vetors are inorporated into the ovariane matrix speifi to their asses suh that[4]: ~ Σ = Σ Σ Σ L T v v (5) + = γ γ : user defined parameter. Σ ~ : TV-based ovariane matrix for the ass. Furthermore, the tangent simiarity orresponds to the Tangent Vetor-based Mahaanobis (TVM) distane of a pattern x with respet to a ass beomes as: n T ~ ( x ) = ( x µ ) Σ ( x ) TVM µ (6) In[4], these simiarities were used to improve the ikeihood funtion of Bayesian assifiers. For our part, we investigate their use as data features for training SVMs. The idea onsists of substituting eah pattern by a feature vetor P x = TVM x, TVM x C onstituted by its TVM with ( ) { ( ) ( )} respet to a asses. Then, to evauate kerne funtions the distane x P x i P x whie the x i is repaed by ( ) ( )

4 Amerian Journa of Signa Proessing. 0; (): dot produt is substituted by ( x ) P( x) P i. Note that benefits behind the use of suh feature vetors are not ony the impiit inorporation of invariane knowedge into SVMs but aso the redution of the data size. For aphanumeri harater reognition the data size dereases from the harater image size to a vetor of 36 omponents (whih is the number of asses). This makes the kerne evauation as we as the training stage faster. 5. Experimenta Resuts Experiments are onduted on a dataset obtained by ombining the we-known USPS handwritten digits with a set of ursive upperase etters extrated from the C-Cube (Cursive Charater Chaenge) database. The USPS is omposed of 79 training data and 007 test data distributed on the 0 digits. These images are normaized to a size of 6 6 pixes yieding 56 dimensiona datum vetor. For this reason, ursive upperase etters of C-Cube database whih ome with varying sizes were normaized to the same size. This database ontains 5793 ursive haraters manuay extrated from ursive handwritten words, inuding both upper and ower ase versions of eah etter. A set of 608 upperase etters were extrated and normaized to sae with the USPS data. The resuting database inudes 734 training sampes and 358 test sampes. Figure shows some exampes whih highight the simiarity between upperase etters and digits S O G H Figure. Training sampes from the USPS-C-Cube database. We first performed a series of OAA and N-SVM runs to test out a possibe onfigurations. Based on the orresponding error rates the best parameter vaues were seeted as foows. The reguarization parameter is fixed at 0 whie kerne parameters are as: the sigma is 5 for RBF, d= for poynomia kerne, γγ = for negative distane whie sigmoid parameters are tuned to: ss 0= 0.009, ss =.. The neura network of N-SVM is a mutiayer pereptron with 90 hidden nodes, the step size equas 0.08, the momentum is about and the network is trained for 5000 iterations eah of whih orresponds to one proessing of a training data through the network. In order to assess the behavior of tangent simiarities with kernes based on a dot produt and distane measure, we evauated the Error Rate (ER) of both OAA and N-SVM by using kernes isted in Tabe (). The resuts in terms of error rate are summarized in Tabe. In this test, the number of tangent vetors was arbitrariy fixed at 0. Tabe. Error rates (%) provided by the training of tangent simiarities using 0 tangent vetors RBF ND Po Sig N-SVM 8,5 5, 7,4 30, OAA 3,37 5,9 9,78 9,7 We an note that N-SVM outperforms the OAA with three kernes whie it is sighty ess aurate when using the sigmoid kerne. In addition, with both approahes, distane-based kernes dea better than dot produt-based kernes where the RBF gives the best resuts. This kerne aows a gain whih equas at east 7%. In the seond test, we investigated the behavior of OAA and N-SVM with respet to variations of tangent vetors number. This evauation was arried out by using the RBF kerne with the preedent parametri seetion. So, for both approahes, variations of error rate for different hoies of the TV number are potted in Figure. Roughy speaking, whatever the number of TV, OAA and N-SVM behave simiary but with different error rates. More preisey, the N-SVM ombination gives a smaer error rate with a differene of 0.5%. We note aso, the first TV improve the error rate whih dereases with a fator of more than 4% with N-SVM and 6% with OAA. However, arger numbers of TV have a ompounding effet sine the error rate grows to more than 3% with both approahes. This outome an be expained by the fat that TV are obtained by eigenvetor deomposition. So, simiary to the prinipe omponent anaysis the ast eigenvetors are noisy and annot bring any disriminating knowedge about data. Error rate (%) Error rate variations aording to the number of tangent ve- Figure. tors N-SVM OAA Number of Tangent Vetors Finay, the best error rates are obtained for 30 TV. The resuts orresponding to this hoie are reported in Tabe 3. In addition to error rates, this tabe reports the number of support vetors (#SV) per binary node (SVM), the Training Time (TT) expressed in hours as we as the reognition speed whih is the Number of Reognized Charaters (NRC) per seond. As an be seen, N-SVM outperforms the OAA by 0.8% in error rate whie being muh faster. In fat, N-SVM training is 5 times faster beause it empoys 8 SVMs trained on different dihotomies whie the neura network takes about 5 minutes. On the ontrary, eah SVM

5 38 Hassiba Nemmour et a.: Training Tangent Simiarities with N-SVM for Aphanumeri Charater Reognition in the OAA is trained over the fu dataset whih yieds a arger runtime. In onsequent, N-SVM requires a smaer number of SV per SVM node the reason for whih it has a higher reognition speed. Hene, roughy speaking N-SVM provides a signifiant aeeration of the runtime whie being sighty more aurate. Tabe 3. Performane evauation of N-SVM and OAA using 30 TV OAA N-SVM Error rate (%) #SV TT (Hours) NRC (seond) Furthermore, Figure 3 exhibits the error rates ahieved in eah ass of interest by OAA and N-SVM. We remark that error rates respetive to numera asses are ommony smaer than those ahieved for severa upperase etters whih an reah 00%. This is the ase of etters I, J and Z whih are ompetey onfused with other asses when using the OAA approah. This outome is reated to the number of training sampes whih is smaer (ess than 0 training sampes) ompared to those of the other asses. In fat, in the onsidered dataset etter asses have smaer training sets ompared to numera asses. In addition, the normaization of etter images aording to the size of USPS data atered the shape desription of some etters whih ome with a very arge initia size (images of pixes). Nevertheess, N-SVM performs gobay better than OAA sine it gives ower error rates in 3 asses. Besides, it redues signifianty the error rates of asses whih are ompetey onfused by OAA (I,J, and Z). Error rate (%) Figure 3. Error rates respetive to eah ass of interest. 6. Conusions OAA N-SVM A C E G I K M O Q S U W Y Casses In this paper, a fast and robust system for handwritten aphanumeri harater reognition is proposed. Tangent simiarities based on ass speifi tangent vetors are used to dea with data variabiity whie N-SVM ombination was used to aeerate the training stage of muti-ass impementation of SVM (OAA). The N-SVM ombination takes advantage from the high separation abiity of SVM to separate the pairs of asses and the earning abiity of neura network to onstrut an automati deision about aphanumeri asses. The experimenta anaysis was onduted on a dataset obtained by ombining two benhmark datasets whih are the USPS for handwritten digits and the C-Cube for upperase etters. The resuts indiate that tangent simiarities aow a signifiant improvement in error rate when using 30 tangent vetors. Besides, it has been shown that N-SVM outperforms the OAA in both runtime and reognition auray. REFERENCES [] Kim, K. I., Jung, K., Park, S. H., and Kim, H. J., 00. Support vetor mahines for texture assifiation, IEEE Transations on Pattern Anaysis and Mahine Inteigene, Vo. 4, [] Pamondon, R., and Srihari, S. N., 000. On-ine and off-ine handwriting reognition: A omprehensive survey, IEEE Transations on Pattern Anaysis and Mahine Inteigene, Vo., [3] Burges, C. J. C., 998. A Tutoria on support vetor mahines for pattern reognition. Data Mining and Knowedge Disovery, Edited by Ussama Fayyad, Vo., -67 [4] Shökopf, B., 997. Support vetor earning, Thèse de PhD: Université de Berin, 73 pages [5] Keysers, D., Paredes, R., Ney, H., and Vida E., 00. Combination of tangent vetors and oa representations for handwritten digit reognition, Leture Notes in Computer Siene, Vo. 396, [6] Nemmour, H., and Chibani, Y., 00. Handwritten digit reognition based on a Neura-SVM ombination, to appear in Internationa journa of omputers and appiations, Vo, 3, [7] Trier, O. D., Jain, A. K., and Taxt. T., 996. Feature extration methods for harater reognition A survey, Pattern reognition, Vo. 9, [8] Nemmour, H., and Chibani, Y., 009. Handwritten aphanumeri harater reognition based on support vetor mahines and ombination of desriptors, ACM Internationa Conferene on Inteigent Computing and Information Systems ICICIS 09, 9- Marh, Cairo [9] Chen, G. Y., Bui, T. D., and Krzyzak, A., 006. Rotation invariant feature extration using ridgeet and Fourier transforms; Pattern Anaysis and Appiation Journa, Vo. 9, [0] Yang, L., Suen, C. Y., Bui, T. D., et Zhang, P., 005. Disrimination of simiar handwritten numeras based on invariant urvature features, Pattern Reognition Journa, Vo. 38, [] Broumandnia, A., Shanbehzadeh, J., and Varnoosfaderani, M. R., 008. Persian/arabi handwritten word reognition using M-band paket waveet transform, Image Vision and Computing Journa, Vo.6, [] Simard, P., Le Cun, Y., Denker, J., and Vitorri, B., 993. Effiient pattern reognition using a new transformation distane, Advanes in Neura Information Proessing Systems, Vo. 5, 50-58

6 Amerian Journa of Signa Proessing. 0; (): [3] Simard, P., Le Cun, Y., Denker, J., et Vitorri, B., 998. Transformation invariane in pattern reognition tangent distane and tangent propagation, Leture Notes on Computer Siene, Vo. 54, [4] Keysers, D., Mherey, W., and Ney, H., 004. Adaptation in statistia pattern reognition using tangent vetors, IEEE Transations on Pattern Anaysis and Mahine Inteigene, Vo. 6, [5] Vapnik, V., 995. The nature of statistia earning theory, Springer-Verag, New York [6] Hsu, C-W., and Lin, C-J., 00. A omparison of Methods for Muti-ass Support Vetor Mahines, IEEE Transations on Neura Networks, Vo. 3, [7] Joahims, T., 998. Making arge sae SVM Learning pratia, MIT Press: Advanes in Kerne Methods Support Vetor Learning, B. Sh okopf, C. J. C. Burges, and A. J. Smoa Editors, [8] D. E. Rumehart, G. E. Hinton, & R. J. Wiiams, Learning interna representations by error propagation, MIT Press (Parae Distributed Proessing: Exporations in the mirostruture of Cognition),, 986, [9] DeCoste, D., and Shökopf, B., 00. Training invariant support vetor mahines, Mahine Learning Journa, Vo. 46, 6-90 [0] Haasdonk, B., and Keysers, D., 00. Tangent distane kernes for support vetor mahines, In Pro. Of the 6 th Internationa Conferene on Pattern Reognition (ICPR), vo., pp , 00 [] Nemmour, H., and Chibani, Y., 008. Inorporating Tangent Vetors in SVM Kernes for Handwritten Digit Reognition, the th Internationa Conferene on Frontiers in Handwriting Reognition, ICFHR 08, 9- Août, Montréa, Canada