Modelng of Pen-Coordnae Informaon n SCPR-based HMM for On-lne Recognon of Handwren Japanese Characers Junko Tokuno, Ypng Yang, Gledson Pegore da Slva, Akho Kada and Masak Nakagawa Tokyo Unversy of Agrculure and Technology, 2-24-16, Naka-cho, Kogane, Tokyo, Japan {j-okuno, yangypng, pegore, ak}@hands.e.ua.ac.jp, nakagawa@cc.ua.ac.jp Absrac Ths paper descrbes sochasc modelng of pencoordnae nformaon n HMMs wh srucured characer paern represenaon (SCPR for on-lne recognon of handwren Japanese characers. SCPR allows HMMs for Kanj characer paerns o share common s. Alhough SCPR-based HMMs have been successfully appled o Kanj characer recognon, he pen-coordnae feaure has no been modeled snce s an unque feaure for each characer paern. In hs paper, we employ mappng from a common o each occurrence n Kanj paerns n he esmaon sep of SCPR-based HMMs. We also employ adapaon of sae parameers o each characer paern n generang characer HMMs by composng SCPR-based HMMs. Expermenal resuls showed ha he correc recognon rae mproved from 83.6% o 92.3% for Japanese Kanj characer paerns by modelng he pen-coordnae nformaon n he SCPRbased HMMs. Keywords: On-lne handwrng recognon, Srucured characer paern represenaon, HMM, Pen-coordnae feaure 1. Inroducon As he populary of PDA, able PC, and oher penbased or paper-based sysems, he demands for mprovemen of on-lne handwrng recognon accuracy have ncreased. The hdden Markov model (HMM has been successfully appled no only o Wesern handwrng recognon [1] bu also o on-lne handwrng recognon of Chnese [2], Kanj characers of Chnese orgn [3]-[6] and Hangul characers n Korea [7] because of s promsng ably o model deformaons of srokes and varaons of he number of sample pons. In case of alphanumerc on-lne handwrng recognon, characer HMM has been wdely employed, where each whole leer of he alphabe s modeled ypcally by one HMM and all words are represened by employng only some dozens of HMMs a mos. On he oher hand, here are housands of characers n Orenal characers of Chnese orgn, so ha modelng each characer by an HMM leads o an nfeasble characer recognon sysem requrng huge amoun of memory and ranng daa. To ackle hs problem, srucured characer paern represenaon (SCPR-based HMM [2] has been proposed where each Kanj characer s represened as a compose of consuen s, whch are shared among several Kanj characers. The SCPR-based HMM provdes such advanages as reducng he oal sze of he models, makng he recognon sysem robus agans deformaon of common s and so on. In he SCPR-based HMMs, he pen-drecon feaure exraced from consecuve pen-p posons has almos always been employed [2][6]. In conras, he pencoordnae feaure has no been employed, hough s no less mporan han he pen-drecon feaure. Ths s due o ha he pen-coordnae nformaon s more subjec o change when s are composed no each characer paern. The proposed SCPR-based HMMs model boh he pen-drecon feaure and pen-coordnae feaure by employng mappng from a common o each occurrence n Kanj characer paern n he esmaon sep of SCPR-based HMMs. Moreover, we adap SCPRbased HMM parameers o each characer paern accordng o he nformaon of he sze and he poson of raw paerns n he recognon sep. Ths paper s organzed as follows. Secon 2 descrbes our recognon sysem whch employs HMM and SCPR as key componens. Secon 3 presens he proposed approach for sochasc modelng of he pencoordnae feaure by SCPR-based HMMs. Secon 4 shows expermenal resuls and Secon 5 concludes hs work. 2. Recognon Sysem Snce HMM-based on-lne handwrng recognon for housands of Kanj characers requres an enormous compuaon me, s dffcul for compuers wh low performance such as PDAs o work praccally. Therefore, we frs carry ou coarse classfcaon and reduce recognon canddaes [8] and hen apply HMMbased recognon for hose canddaes. In our research, we se he number of maxmum recognon canddaes o 200. The proposed HMM-based recognon sysem bascally consss of a feaure exracon module, SCPR, SCPR-based HMMs, and a decoder. In hs secon, we show he oulne of our recognon sysem.
Srucural nformaon ( s 1x, s 1y ( s 2x, s 2y ( s 3x, s 3y h 1 ( s 4x, s 4y h 2,h 3 w 4 h 4 Subpaerns SCPR Fgure 1. Srucured characer paern represenaon. C 3 D B c 4 2 d b E A e a 5 0 1 f h F H g 6 8 G 7 Fgure 2. Sub-sroke caegores: A-H (a-h are long (shor sub-srokes and 0-8 are he drecon of offsrokes. 2.1. Feaure Exracon Characer paerns We use a sequence of pen-p posons (x,y, =1,,T, sampled a a ceran nerval from a pen able as he pen-coordnae feaures. Moreover, we exrac he dsplacemen ( x, y =(x -x -1,y -y -1 from wo consecuve pen-p posons and use (r,θ as he pendrecon feaure, where r = x + y and θ denoe 2 2 he velocy and he drecon of he pen movemen, respecvely. We hen denoe by O=o 1,o 2,,o T, o =(x,y,r,θ he feaure sequence represenng each characer paern n case ha we employ boh he pencoordnae feaures and he pen-drecon feaures. 2.2. Srucured Characer Paern Represenaon (SCPR Kanj characer paerns, deographc characer paerns of Chnese orgn, are mosly composed of mulple s. Very ofen s are shared among several Kanj characer paerns as shown n Fgure 1. Each s composed by a sequence of srokes and each sroke s made of a sequence of subsrokes. In hs paper, a sroke denoes a sequence of pen-p coordnaes sampled from pen-down o pen-up and an off-sroke denoes a vecor from pen-up o he nex pen-down. A sroke s dvded no a sequence of sragh-lne sub-srokes [6]. Sub-srokes are classfed no egh drecons of wo-quanzed lengh whle offsrokes are classfed no egh drecons and anoher of very shor dsance wh arbrary drecon as shown n Fgure 2. Srucured characer paern represenaon (SCPR represens a characer paern as a compose of s and her srucures, where common s are shared among several characer paerns ha nclude he s s her shapes. SCPR s suable for paerns ha have srucures lke Chnese w 1 Basc w 2 w 3 Lnear mappng ( s 5x, s 5y Mapped basc 64 Fgure 4. Generang mapped basc by lnear mappng. characers and provdes advanages such as he sze reducon of a dconary (a se of prooype paerns and greaer recognon robusness agans deformaon of common s. Japanese Kanj characers are of Chnese orgn and are almos he same as Chnese characers. Korean Hangul characers, alhough hey are phonec characers, also have a srucural composon smlar o ha used for Chnese characers, so srucured approaches have also been appled. In our on-lne recognon sysem, each characer paern prooype s represened by a SCPR ha s a compose of s and off-srokes. Each SCPR has consuen s and he srucural nformaon on where n he characer paern each s placed n erms of he boundng box (s x,s y <w,h> for he, where (s x,s y denoes he oplef corner and <w,h> denoes <wdh, hegh> of he boundng box as shown n Fgure 3. All he basc s (s whch are no decomposed furher no smaller s as well as he characer paerns are represened by a square shape of resoluon, and are reduced o boundng boxes n srucural nformaon hrough lnear mappng when hey are ncluded n larger s or characer paerns (Fgure 4. In hs paper, we call a resul of he lnear mappng a mapped basc, even f he mappng s dencal. Snce he recognon mehod s sensve o sroke order varaons, here are mulple prooypes for each and mulple SCPRs for each characer paern. Before he sar of hs sudy, all he prooypes had already been clusered by employng a smple cluserng algorhm based on he LBG algorhm [11]. Though he number of defnons s dfferen among s, he average s 4.53 for each and he sandard devaon s 5.86. w 5 Fgure 3. Srucural nformaon of SCPR. h 5
a a11 a 22 33 Basc Mapped basc Learnng paern π 1 a 12 a 23 S1 S2 3 a S 34 π 1 a12 S 1 Mappng b ( o 1 b ( o b ( o b ( o 2 3 1 Inversely mapped learnng paern Fgure 5. Sub-sroke HMMs : (Lef pen down model, (Rgh pen up model. 2.3. SCPR-based HMMs We model each by HMMs. In hs paper, we call our HMMs as SCPR-based HMMs snce hey are SCPRs. In he prevous sudy [6], each sub-sroke s represened by a lef-o-rgh HMM of hree saes whle each off-sroke s represened by a sngle sae (Fgure 5. On he oher hand, each sroke s represened by concaenaon of sub-sroke HMMs and each s represened by ha of sroke HMMs nerleaved wh off-sroke HMMs n hs work. Therefore, he oal number of saes n each HMM s deermned by connecng hree saes subsroke HMMs and sngle-sae off-sroke HMMs. Here, le λ (k =(A (k,b (k,π (k be he se of HMM parameers of a or off-sroke k, wh he followng noaons: A (k ={a (k j } : se of all he sae-ranson probably dsrbuons from sae S o S j, B (k ={b (k ( o } : se of all he probably dsrbuons of observng symbols o a sae S, π (k ={π (k } : nal sae probably dsrbuons. The observaon probably dsrbuons are represened by mxures of M Gaussan dsrbuons gven by b ( o = M c 1 exp(- ( o 2 - µ Σ -1 m m n m= 1 (2π Σm m ( o - µ m (1 wh he mean vecors µ, he covarance marces Σ (n s he dmenson of he observaon feaure vecor o. and weghng coeffcens c m. Here, he drecon feaure θ has a connuous probably dsrbuon wh 2π cycle. In hs sudy, each sae of HMM has a sngle Gaussan dsrbuon wh dagonal covarance. These parameers can be raned hrough he Verb ranng or he Baum- Welch algorhm. 2.4. Decoder Accordng o a SCPR and SCPR-based HMMs, he decoder concaenaes he SCPR-based HMMs o generae an HMM of each canddae characer paern, and hen calculaes he probably ha he npu paern s produced from he HMM. Ths operaon s effecvely done by he Verb search algorhm. Inverse mappng Fgure 6. Mappng and nverse mappng. 3. Modelng of Pen-coordnae Feaures The basc dea of modelng pen-coordnae feaures by SCPR-based HMMs s based on he followng mappng and nverse mappng. As menoned before, all s are represened by a square shape of resoluon n our recognon sysem. In hs sudy, we call he square sze of normalzaon sze. When we generae characer paerns by combnng s, we apply a lnear mappng from each o s area n a characer paern. In conras, we apply he nverse of he above mappng when we learn s (Fgure 6. A smple dea s o enlarge he sze of he boundng box of a mapped basc n a learnng paern o he normalzaon sze. By applyng he nverse mappng, we can exclude characer dependency of each (dfference n sze and poson when appears n dfferen characer paerns o model pen-coordnae feaures of he by SCPR-based HMM. 3.1. Inal Parameers of SCPR-based HMMs Before esmang SCPR-based HMM parameers by he Verb ranng or he Baum-Welch mehod, we se nal parameers n each SCPR-based HMM by applyng he nverse mappng. However, handwrng usually has noses due o a hand vbraon ec. and hese noses have no or lle correlaon wh he boundng box sze of a, so ha he nverse mappng may magnfy hese noses and reflec hem no he. Therefore, we se he nal parameers from characer paerns whch are composed of only a sngle. Frs of all, we exrac he feaures: O=o 1,o 2,,o T, o =(x,y,r,θ from each characer paern n ranng daa and hen we assgn hose feaures o each sae of concaenaed SCPR-based HMMs equally by employng pen-up nformaon. We se he nal parameers for each sae by akng he average of hose feaures n he ranng daa. Exceponally, here are some s whch do no appear as characer paerns as hem alone. In hs case, we frs generae characer HMMs and carry ou he Verb segmenaon for he HMMs. Accordng o he resul of he Verb segmenaon, we exrac hose s and apply nverse mappng o hem. Fnally, we assgn he feaures: O=o 1,o 2,,o T, o =(x,y,r,θ o
each sae of SCPR-based HMMs accordng o he resul of he Verb segmenaon and se he nal parameers based on he above procedure. 3.2. Adapaon of SCPR-based HMM Parameers In boh he esmaon sep and he recognon sep, we generae HMMs for characer paerns by connecng one or more han one SCPR-based HMM accordng o SCPRs descrbed n Secon 2.4. Each SCPR-based HMM whch s a composon of generaed characer HMMs corresponds o a whose square sze s smaller han normalzaon sze, hough each SCPRbased HMM parameers are esmaed by normalzng each ranng paerns. In oher words, he parameers for he pen-coordnae feaure (o =(x,y of a common are mapped o dsnc values when s composed no dfferen characer paerns. Then, we need o adap he parameers of SCPR-based HMM o each characer. v Here, le µ = ( µ ( x, µ ( y be he mean vecor of he Gaussan dsrbuon a a sae S of SCPR-based HMMs, an adaped mean vecor s gven by w µ ( x = µ ( x + s x (2 h µ ( y = µ ( y + s y (3 where µ (x and µ (y denoe he mean vecors of he pencoordnae feaure x and y, and w,h,s x,s y are noed n Secon 2.2. Moreover, we assume ha he dagonal covarance marx Σ =(σ 2 xx,σ 2 yy of he Gaussan dsrbuon a each sae S of SCPR-based HMMs s correlaed o he boundng box szes of s and adap Σ o each characer accordng o he correlaons. Then, n order o oban he correlaons beween he average of he covarance and he boundng box sze we analyzed hem usng he daabase HANDS_kuchbue_d_97_06 [9]. The resul s shown n Fgure 7. Accordng o he resul, we conver he dagonal covarance marx Σ =(σ 2 xx,σ 2 yy o ˆ 2 2 Σ = ( ˆ σ, ˆ σ as follows: xx yy σ xx - 0.085 ( - w ( σ xx 9.4707 ˆ σ xx = (4 0.085w + 0.005 ( σ xx < 9.4707 σ - 0.080 ( - ( 8.99614 yy h σ yy ˆ σ yy = (5 0.080h + 0.007 ( σ yy < 8.99614 3.3. Esmaon of SCPR-based HMM In he esmaon, n order o avod nose nfluence, we employ dsplacemen normalzaon proposed n our prevous sudy [10] nsead of he nverse mappng for Sandard devaon of gaussan dsrbuon Basc σ xx average of average of 12 10 8 6 4 2 0 01 32 64 96 Boundng box sze (pxel Mapped basc Dsplacemen 1 ( x1 - x' 1, y1 - y' 1 ( x 1, y 1 Dsplacemen 2 ( x2 - x' 2, y2 - y' 2 ( x 2, y 2 Dsplacemen 1 > Dsplacemen 2 each. Ths approach s based on he assumpon ha here are some correlaons beween he boundng box sze of a and he freedom of movemen of each feaure pon n he because each feaure pon can move n larger area f he boundng box sze of he s larger as shown n Fgure 8. From he analyss of our prevous sudy, he relaonshp beween he boundng box sze: <w,h> and dsplacemen: (D Ax,D Ay s as follows: D Ax ( w = 0.0846w +1.7 (6 D Ay ( h = 0.0539h + 3.5 (7 The dealed procedure of esmang parameers for SCPR-based HMM based on hs formulaon s as follows. 1. We frs exrac boh he pen-drecon feaures and he pen-coordnae feaures: O=o 1,o 2,,o T, o =(x,y,r,θ from each characer paern n ranng daa and carry ou he Verb ranng for concaenaed nal SCPR-based HMMs. Accordng o he resul of he Verb segmenaon, we exrac s whch do no appear as characer paerns. 2. We conver pen-drecon feaures of each as follows: DAx( x ˆ ' = µ ( x + ( x - µ ( x (8 D ( w Ax Learnng paern σ yy Fgure 7. The relaons beween he sandard devaon and he boundng box sze. Fgure 8. Dsplacemen beween he basc and he learnng paern. ( x ' 1, y ' 1 ( x ' 2, y ' 2
DAy ( y ˆ ' = µ ( y + ( y - µ ( y (9 D ( h where (x,y and (x,y are feaure pons of a prenormalzed and a normalzed, r respecvely. And ˆ µ ( ˆ (, ˆ = µ x µ ( y s he mean vecor of a sae S where a feaure pon (x,y s observed. 3. We updae parameers for each sae by akng he average of hose convered feaures n he ranng daa. 4. Expermens We carred ou expermens o show he effec of he pen-coordnae feaure and ha of SCPR-based HMMs for Japanese Kanj recognon. The daabase HANDS_kuchbue_d_97_06 [9] conans 1,435,440 characers wren by 120 wrers. In he expermens, we used 675,840 paerns (only JIS1 Kanj paerns. Paerns from 60 wrers were used for esmang he HMM parameers by he Verb ranng, and hose from he remanng 60 wrers were used for es. 4.1. Expermen 1: Comparson of Feaures In hs expermen, we compared he convenonal pen-drecon feaures wh he pen-coordnae feaures added o hem as modelng feaures for SCPR-based HMMs o show he effec of he pen-coordnae feaures. The recognon raes accordng o he feaure ses are shown n Table 1. We can see from he resul ha he recognon rae wh pen-coordnae feaures s hgher han ha wh only pen-drecon feaures abou 9 pon. Some samples of correcly recognzed characer paerns by modelng he pen-coordnae feaures are shown n Table 2. We can see from he resul ha msclassfcaons beween characer paerns whch have he same drecon of sroke movemen are mosly correced by modelng he pen-coordnae feaures. Moreover, HANDS_kuchbue_d_97_06 s a collecon of characer paerns of usual ex. As shown n Table 3, ncludes many paerns composed of a few sroke and hey are ofen msrecognzed by only he pen-drecon feaures bu recognzed by boh he pen-drecon feaures and he pen-coordnae feaures. These facs are he reasons of he large mprovemen of recognon rae. Conversely, he characer paerns ha have been urned o ncorrec recognon are shown n Table 4. These characer paerns have he smlar SCPR such as 頴 and 穎 hough he oal number of hose msclassfed characer paerns s small compared o he mproved characer paerns. From hese resuls, can be concluded ha he pencoordnae feaures are effecve for he SCPR-based HMMs. Ay Table 1. Comparson of feaures. Feaures N-bes cumulave recognon rae [%] 1 ~2 ~3 ~10 ( r, θ 83.6 88.4 90.0 92.7 ( x, y, r, θ 92.3 95.1 95.9 96.9 Table 2. Examples of characer paerns correcly recognzed by employng pen-coordnae feaures ( x, y, r, θ Inpu Correc recognon rae Msrecognzed characer paerns of (r,θ (x,y,r,θ (r,θ 治 0.2% 90% 活, 沿, 冶 何 1% 84% 佑 偵 2% 95% 順, 悼, 須 床 2% 97% 庇, 居, 呆 操 3% 97% 燥 Table 3. Correc recognon rae of characer paerns whch ofen appear n he HANDS_kuchbue_d_97_06 daabase. Inpu Appea -rance Correc recognon rae Msrecognzed characer rae (r,θ (x,y,r,θ paerns of (r,θ 人 8% 88% 96% 八, 入 日 6% 6% 65% 白, 月, 用 大 5% 51% 80% 丈 子 4% 53% 84% 与, 土 十 4% 87% 99% 七, 土 Table 4. Examples of characer paerns ha have been urned o ncorrec recognon by employng pencoordnae feaures ( x, y, r, θ Inpu Correc recognon rae Msrecognzed characer paerns of (x,y,r,θ (r,θ (x,y,r,θ 頴 70% 92% 穎 未 72% 82% 朱 埋 77% 98% 浬, 哩 卿 78% 90% 郷 鉛 78% 97% 鈷 4.2. Expermen 2: SCPR-based HMMs vs. Characer HMMs We compared he convenonal characer HMMs and he proposed SCPR-based HMMs wh respec o he amoun of ranng paerns o model deformaon of srokes. Fgure 9 shows he recognon raes when varyng he amoun of ranng paerns. Noe ha here are 5,632 characer paerns per wrer. As he resul, he SCPR-based HMMs acheved beer recognon performance wh a smaller amoun of ranng paerns han he characer HMMs. Ths s because a larger number of ranng paerns are
Recognon rae (% 100 95 90 85 80 75 70 65 employed for each SCPR-based HMM han for each characer HMM even when he amoun of ranng paerns are lmed. In hs sudy, here are mulple prooypes for each and mulple SCPRs for each characer paern o recognze characer paerns wh wrong sroke order as menoned n Secon 2.2. SCPR-based HMMs can be raned from a lo of s wh varous sroke orders because s are shared among several characer paerns. On he oher hand, when he sroke orders of he same characer class are dfferen, each characer HMM s raned separaely, even f he sroke order of consuen s are he same. Therefore, here s no an enough amoun of ranng paerns for he characer HMMs o model deformaons of srokes. Ths resul also shows ha here s no apprecable dfference beween he recognon rae of he SCPRbased HMMs and he characer HMMs n case ha here s enough amoun of ranng daa. 5. Concluson Characer HMM SCPR-based HMM 5 10 20 30 60 The number of w rers for ranng Fgure 9. Comparson of characer HMM and SCPRbased HMM abou he amoun of ranng paerns. In hs paper, we have proposed he sochasc modelng of pen-coordnae nformaon by SCPR-based HMMs. Through he expermens, has been shown ha he recognon accuracy s vasly mproved by modelng he pen-coordnae nformaon for SCPR-based HMMs. We also showed ha he proposed SCPR-based HMMs are superor o convenonal characer HMMs n case ha ranng paerns are lmed. Even f here are a large amoun of ranng paerns, he recognon accuracy of he SCPR-based HMMs s no less han ha of he characer HMMs. In he near fuure, we wll consder he sably of pen-coordnae feaures observed a each sae of SCPRbased HMMs [5] n order o mprove recognon accuracy. Acknowledgmen The HMM based approach on whch hs work s exended s due o he research of he frs auhor wh Prof. Shgek Sagayama currenly a Tokyo Unversy and Hrosh Shmodara currenly a Ednburgh Unversy and Msuru Naka a Toyama Prefecural Unversy. The auhors would lke o hank hem. Ths work s parally suppored by Gran-n-Ad for Scenfc Research under he conrac number B:17300031. References [1] J. Hu, M. K. Brown and W. Turn: HMM based on-lne handwrng recognon, IEEE Trans. PAMI, 18, 10, pp.1039-1045, Oc.1996. [2] H. J. Km, K. H. Km, S. K. Km and F. T-P. Lee: Onlne recognon of handwren Chnese characers based on hdden Markov models, Paern Recognon, 30, 9, pp.1489-1499, 1997. [3] H. Io and M. Nakagawa: An on-lne handwren characer recognon mehod based on hdden Markov model (n Japanese, Techncal repor of IEICE, PRMU97-85 pp.95-100, July.1997. [4] K. Takahash, H. Yasuda and T. Masumoo: On-lne handwren characer recognon usng hdden Markov model (n Japanese. Techncal repor of IEICE, PRMU96-211, pp.143-150, Mar.1997. [5] D. Okumura, S. Uchda and H. Sakoe: An HMM mplemenaon for on-lne handwrng recognon based on pen-coordnae feaure and pen-drecon feaure, Proc. ICDAR 05, pp.26-30, Aug.2005. [6] M. Naka, N. Akra, H. Shmodara and S. Sagayama: Sub-sroke approach o HMM-based on-lne Kanj handwrng recognon, Proc. ICDAR 01, pp.491-495, Sep.2001. [7] S-J. Cho and J. H. Km: Bayesan Nework Modelng of Hangul Characers for On-lne Handwrng Recognon, Proc. ICDAR 03, pp.207-211, Aug. 2003. [8] K. Masumoo and M. Nakagawa: Improvemen of a coarse classfcaon mehod for on-lne recognon of handwren Japanese characers (n Japanese, Techncal Repor of IEICE, PRMU2001-272, pp.9-16, Mar. 2002. [9] M. Nakagawa and K. Masumoo: Collecon of on-lne handwren Japanese characer paern daabases and her analyss, IJDAR, 7, 1, pp.69-81, 2004. [10] A. Kada and M. Nakagawa: A learnng algorhm for srucured characer paern represenaon used n on-lne recognon of handwren Japanese characers, Proc. IWFHR 02, pp.163-168, Aug. 2002. [11] Y. Lnde, A. Buzo and R. M. Gray: An algorhm for vecor quanzer desgn, IEEE Trans. on Communcaons COM-28(1: 84-95. 1980.