A Hilbert Warping Method for Camera-based Finger-writing Recognition

Transcription

1 A Hilber Warping Mehod for Camera-based Finger-wriing Recogniion Hiroyuki Ishida 1,2, Tomokazu Takahashi 3,IchiroIde 1 and Hiroshi Murase 1 1 Nagoya Universiy, Furo-cho, Chikusa-ku, Nagoya, Aichi, Japan hishi@murase.m.is.nagoya-u.ac.jp 2 Japan Sociey for he Promoion of Science, Japan 3 Gifu Shooku Gakuen Universiy, Nakauzura 1-38, Gifu-shi, Gifu, Japan Absrac We propose a ime-warping algorihm for recognizing finger acions by a camera. In he proposed mehod, an inpu image sequence is aligned o he reference sequences by phase-synchronizaion of he analyic signals, and hen classified by comparing he cumulaive disances. A major benefi of his mehod is ha overfiing o sequences oncorrec caegories is resriced. The proposed mehod exhibied high recogniion accuracy in finger-wriing characer recogniion. 1. Inroducion Camera-based analysis of human behavior has been sudied for decades [1. One os applicaions is fingerwriing recogniion sysem [2 in which characers wrien in he air are idenified. I has gained aenion as a novel means of man-machine ineracion [3 because: (1) users can operae compuers jus by simple fingeracions, and (2) i does no require exra equipmens excep for a camera. In [2, finger-wriing characers were recognized from rajecories of he finger posiion. Since he rajecories are nonlinearly warped wih respec o he ime axis, he dynamic ime warping (DTW) mehod [4 is employed for he sequence alignmen; an inpu sequence is classified o a reference sequence which gives he minimum cumulaive disance. However, he DTW has a drawback for he classificaion ask. Because he DTW finds he bes alignmen for he reference sequences of all caegories, misclassificaion can occur due o he over-fiing o incorrec caegories. To cope wih his problem, we propose a Hilber warping mehod which finds he proper alignmen only for he correc caegory. In he proposed mehod, he sequences are convered ino he form of analyic signals [5. An imporan propery of he analyic signal is ha is insananeous phase increases consanly. Using his propery, boh of he sequences are aligned by phase-synchronizaion of analyic signals. Undesirable over-fiing o incorrec caegories is avoided if he sequence alignmen is performed by he phase-synchronizaion. In his paper, we apply he proposed mehod o camera-based recogniion of finger-wriing characers. Figure 1 shows he flow of he proposed mehod. Firsly, image sequences are convered o ime-varying feaure vecors by he eigenspace mehod [6, as proposed in a gesure recogniion mehod [7. Secondly, each feaure value is ransformed o an analyic signal. The empirical mode decomposiion (EMD) [8 is inroduced here o ensure ha he phase of he analyic signal becomes monoonic. Finally, he cumulaive disance beween wo sequences are calculaed by synchronizing he phase of analyic signals. This paper is organized as follows: Secion 2 inroduces he propery of analyic signals. In Secion 3, he proposed Hilber warping mehod is described. Resuls are presened in Secion Analyic signal An image sequence is ransformed o analyic signals [5 for sequence alignmen. Le f() be a feaure value obained from he -h image in he sequence. An analyic signal a() is composed of he original signal f() as he real par and is Hilber ransform H [f() = (1/π) f() as he imaginary par [5. I is denoed as a() =f()+jh [f() = a() e jφ(), (1) where φ() is defined as he insananeous phase. In principle, φ() increases monoonically, which means ha a() roaes couner-clockwise in he complex plane as illusraed in Fig /8/$ IEEE

2 [ f () a() fi f i i f ϕ() f () [ [ [ Figure 2. Consrucion of analyic signal. H [f() is he Hilber ransform of f(). e 2 Figure 1. Proposed Hilber warping mehod for finger-wriing recogniion. x(2) x() e 3 g() g(3) g(2) g(1) e 1 x(3) x(1) 3. Hilber warping mehod The mehod for he sequence classificaion is described in his Secion. Alhough a similar approach was proposed in [9, he over-fiing problem in he classificaion was no aken ino consideraion. Furhermore, he performance of he sequence alignmen was no perfec, since neiher EMD nor a feaure vecor was used. The proposed mehod ensures proper alignmen for a correc caegory, bu avoids over-fiing o incorrec caegories. 3.1 Feaure vecor Using he eigenspace mehod, feaure vecors are obained from images. Iniially, he mean vecor μ and an R-dimensional eigenspace {e 1,, e R } are consruced from all reference images [6. Le he -h image in a sequence be represened by a normalized vecor x(). I is projeced on he eigenspace as a poin g() by g() = [e 1 e R (x() μ) (2) = [f 1 () f R (), (3) as shown in Fig. 3. These () (1 i R) are used as he feaure values for sequence alignmen. Figure 3. Feaure vecors in eigenspace. 3.2 Calculaion of phase-shif The feaure vecor g() is convered o an analyic signal vecor (ASV) α() by ransforming each elemen () o an analyic signal a i () using Eq. (1). Thereby, α() is represened by α() = [ a 1 () a R (). (4) Le α (c) () be a reference ASV of caegory c, and α in () be an inpu ASV. Phase-shis evaluaed from he argumen ( ) of he Hermiian inner produc p (c) ( 1, 2 ) given by p (c) ( 1, 2 )= [ α (c) ( 1 ) α in ( 2 ), (5) where he superscrip denoes he complex conjugae ranspose of a vecor. In he alignmen sage, he frame 1 corresponding o he frame 2 is sequenially searched according o he sign of p (c) ( 1, 2 ). 3.3 Calculaion of phase-shif using EMD Equaion (5) is effecive only if he phase is increases monoonically. Unforunaely, such requiremen is no saisfied unless he original () has a zero-crossing poin beween local maxima [1. For example, an analyic signal generaed from () in Fig. 4 (a) has local

3 Value of f() Value (a) f() f() 1s IMF 2nd IMF 3rd IMF Residual Imaginary par Imaginary par.5 a().5 Real par (b) Analyic signal of (a).5 1s IMF 2nd IMF 3rd IMF [1 1 1[2 1[3 1 [4 2 (c) 1 d (, ) Figure 5. Phase-synchronizaion process for sequence alignmen (c)imfsof(a).5 Real par (d) Analyic signals of (c) Table 1. Hilber warping algorihm for calculaing he cumulaive disance D (c) o caegory c. Figure 4. Examples of analyic signal. loops in which he phase decreases (Fig. 4 (b)). In order o eliminae hese loops, we apply he EMD 1 o decompose () o oscillaion funcions called inrinsic mode funcions (IMFs) (Fig. 4 (c), (d)). Some of he IMFs should be excluded during he period where hey are considered o make loops. Suppose ha b i () is a sum of analyic signals of such IMFs and he residual, he following vecor is subraced from α(). β() = [ b 1 () b R () (6) Accordingly, he righ side of Eq. (5) is modified as [ α (c) ( 1 ) β (c) ( 1 ) [ α in ( 2 ) β (c) ( 1 ). (7) Hilber warping algorihm /* Iniializaion */ 1 D (c), 1 [1 1, 2 1, i 1 2 do 3 do /* Search by he sign of he phase-shif */ 4 1 [i +1 1 [i+sgn p (c) ( 1 [i, 2 ) 5 i i +1 6 unil sign of p (c) ( 1 [i, 2 ) changes /* Disance d (c) ( 1, 2 ) is calculaed */ 7 D (c) D (c) + min i d (c) ( 1 [i, 2 ) 8 1 [1 arg min 1[i d (c) ( 1 [i, 2 ) , i 1 1 unil 2 reaches he las frame 11 reurn D (c) 3.4 Hilber warping algorihm The proposed algorihm for he alignmen beween a reference sequence (1 1 T 1 ) and an inpu sequence (1 2 T 2 ) isshownintable1. As illusraed in Fig. 5, his algorihm explores he ime-warping pah by racing he node ( 1, 2 ) where p (c) ( 1, 2 ), and simulaneously compues he cumulaive disance D (c). In his algorihm, he frameo-frame disance d (c) ( 1, 2 ) is defined as an Euclidean disance beween ASVs by d (c) ( 1, 2 )= α (c) ( 1 ) α in ( 2 ) 2. (8) 1 The algorihm is described in [8. We developed a library hh.h for using he EMD and Hilber ransform in MIST libraries [11. Finally, he inpu sequence is classified o ( ĉ = arg min D (c) c 1=1 d(c) ( 1, 1) + T 1 1= 1 +1 d(c) ( 1,T 2 ) ),(9) where 1 and 1 are he frame numbers which are aligned o 2 =1and 2 = T 2, respecively. This mehod avoids he over-fiing o incorrec caegories because he searched pah ( p (c) ( 1, 2 ) ) does no coincide wih he pah giving he minimal D (c) if he wo sequences canno be aligned consisenly.

4 A (se 1) A (se 2) Recogniion rae (%) HW + EMD Simple HW DTW A (se 3) Figure 6. Example omages in daases Dimension of eigenspace 4. Experimenal resul An experimen was conduced using finger-wriing characer daases 2 (Fig. 6) which consised of 1 daases wrien by 1 persons individually. Each daase conained 26 image sequences of finger-wriing leers (uppercase A Z). Recogniion raes were evaluaed by leave-one-ou cross-validaion; all he sequences excep for an inpu daase were used as references. The classificaion was based on he neares neighbor rule (1-NN). The performance of he proposed mehod (HW+EMD) was compared wih he DTW. The cumulaive disance D (c) (T 1,T 2 ) of he DTW was calculaed by D (c) (, ) = (1) D (c) { ( 1, 2 ) = min D (c) ( 1 k, 2 1) } k + d (c) ( 1, 2 ), ( k 2), (11) where d (c) ( 1, 2 ) here is an Euclidean disance in he eigenspace. The proposed mehod was compared also o he simple Hilber warping mehod wihou EMD (simple HW). This simple HW used Eq. (5) insead of Eq. (7). 4.1 Recogniion accuracy Figure 7 shows he recogniion raes. The horizonal axis of he graph represens he dimension R of he eigenspace. According o he resuls, he proposed mehod ouperformed he DTW. For example, caegories H and M were disinguished properly (Table 2). Unlike he DTW, he proposed mehod 2 The daases can be downloaded for evaluaion freely from hp// hishi/finger-wriing.hml. Figure 7. Recogniion raes of fingerwriing characers. Table 2. Confusion marices for caegories H and M (R =5). (a) DTW (b) HW + EMD Resul Inpu H M H 8/1 1/1 M 2/1 8/1 Resul Inpu H M H 9/1 /1 M /1 1/1 avoided he over-fiing o caegory M. Disance marices d (c) ( 1, 2 ) for recognizing caegory H in daase 1 are presened in Fig. 8. From he lower-righ sub-figure of Fig. 8, we can see ha he differen caegory was successfully rejeced. As described in 3.4, he search pah is composed of ASV pairs wih he same insananeous phase. Accordingly, he phase-synchronizaion gave he proper ime-warping pah for he classificaion. The resuls indicae also ha he EMD is necessary especially when he dimension of he feaure vecor is small. 4.2 Compuaional cos The compuaion ime for recognizing one sequence is shown in Table 3, where he resuls of he Hilber warping mehods include also he ime required for he Hilber ransform. The proposed mehod was approximaely hree imes faser han he convenional DTW, since he calculaion of d (c) ( 1, 2 ) was drasically reduced as shown in Fig. 8. The EMD was useful also in erms of speed because he monooniciy of he phase conribues o he

5 DTW HW + EMD Figure 8. Example of disance marices. Values of d (c) ( 1, 2 ) are shown by he inensiy (: black). Nodes filled wih oblique lines were no searched. Pars of his research were suppored by he Grans- In-Aid for JSPS Fellows (19-654) and Scienific Research (16354). References [1 D. Gavrila, The visual analysis of human movemen: A survey, Compuer Vision and Image Undersanding, vol.73, no.1, pp.82 98, January [2 L. Jin, D. Yang, L. Zhen, and J. Huang, A novel vision based finger-wriing characer recogniion sysem, Proc. 18h In. Conf. on Paern Recogniion vol.1, pp , Hong Kong, China, Augus 26. [3 V. Pavlovic, R. Sharma, and T. Huang, Visual inerpreaion of hand gesures for humancompuer ineracion: A review, IEEE Trans. Paern Analysis and Machine Inelligence, vol.19, no.7, pp , July [4 H. Sakoe and S. Chiba, A dynamic programming algorihm opimizaion for spoken word recogniion, IEEE Trans. Acousics, Speech and Signal Processing, vol.26, no.1, pp.43 49, February [5 S. Hahn, Hilber ransforms in signal processing, Arech House, Norwood, Maryland, Table 3. Average compuaion ime for recognizing one sequence. The number of eigenvecors was 5. The experimen was performed on a Penium IV 3 GHz PC. DTW Simple HW HW + EMD Time [ms efficien alignmen of sequences. 5. Conclusion In his paper, a Hilber warping algorihm for sequence classificaion is proposed. The sequence alignmen process is based on he phase-synchronizaion of analyic signals, which is suiable for classificaion. The experimenal resul showed he high classificaion performance of he proposed mehod for finger-wriing characer recogniion. Acknowledgemen [6 H. Murase and S. Nayar, Visual learning and recogniion of 3-d objecs from appearance, In. Journal of Compuer Vision, vol.14, no.1, pp.5 24, January [7 T. Waanabe and M. Yachida Real-ime gesure recogniion using eigenspace from muli-inpu image sequences, Proc. 3rd In. Conf. on Auomaic Face and Gesure Recogniion, pp , Nara, Japan, April [8 N. Huang and S. Shen, Hilber-Huang ransform and is applicaions, World Scienific, Inerdisciplinary Mahemaical Sciences, vol.5, FarrerRoad, Singapore, 25. [9 A. Maheswaran and B. Davis, Analyical signal processing for paern recogniion, IEEE Trans. Acousics, Speech and Signal Processing, vol.38, no.9, pp , Sepember 199. [1 T. Zagajewski, Criicism of he definiion of insananeous frequency, Bull. of he Polish Academy of Sciences, vol.37, no.7 12, pp , November [11 MIST projec, hp://mis.suenaga.m.is.nagoyau.ac.jp/rac-en/.