A NEW APPROACH DEDICATED TO REAL-TIME HAND GESTURE RECOGNITION

Transcription

1 A EW APPROACH DEDICAED O REAL-IME HAD GESURE RECOGIIO guyen Dang Binh, Enoida Shuichi, oshiai Ejia Inelligence Media Laboaoy, Kyushu Insiue of echnology 68-4, Kawazu, Iizua, Fuuoa 8, JAPA {ndbinh, shuichi, oshi}@icey.ai.yuech.ac.jp Absac We inoduce a new Pseudo -D Hidden Maov Model (PDHMM) sucue dedicaed o he ie seies ecogniion (-CoPDHMM). he -PDHMM allows i o do epoal analysis, and o be used in lage se of hand gesues oveen ecogniion syses in unconsained envionens. Addiionally, obus and flexible hand gesue acing using an algoih ha cobines wo poweful sochasic odeling echniques: he fis one is pseudo wo diension hidden Maov odel (PDHMM) and he second echnique is he wellnown Kalan file. Ou wo also pesen a feaue exacion ehod based on he join saisics of a subse of DC coefficiens and hei posiion on he hand. Using feaue exacion ehod along wih he - CoPDHMM sucue was used o develop a coplee vocabulay of 36 gesues including he Aeica Sign Language (ASL) lee spelling alphabe and digis. he esuls ae effeciveness of he appoach. colo, and locaion wih espec o he hand. he appeaance of he hand also depends on is pose; ha is, is posiion and oienaion wih espec o he caea. he basic idea lies in he eal-ie geneaion of gesue odel fo hand gesue ecogniion in he conen analysis of video sequence fo CCD caea. o cope wih all his vaiaion, we cobines wo poweful sochasic odeling echniques: he fis one is pseudo wo diension hidden Maov odel (PDHMM) and he second echnique is he well-nown Kalan file appoach wih hand deecos as descibed in Secion. We use he cobined use of ie spaializaion and PDHMM in he poposed -CoPDHMM odel fo hand gesues ecogniion in Secion 3. he nex secion pesens he esul of expeiens. Finally, he suaize conibuion of his wo in he conclusion secion. Keywods: gesue ecogniion, Pseudo -D HMM, ie seies ecogniion, Kalan file.. IRODUCIO Copue ecogniion of hand gesues povides a oe naual huan-copue ineface. he sign language is undoubedly he os gaaically sucued and coplex se of huan gesues. In Aeican Sign Language (ASL), he use of hand posues [5] is vey ipoan o diffeeniae beween any gesues. hus, a fas and eliable ehod o exac he hand posues changes fo he video sequence is vey ipoan in ASL ecogniion syses. he hand is coplex objec. he ain challenge in hand gesues deecion and ecogniion is he oun of vaiaion in visual appeaance. Fo exaple, hands vay in shape, size, coloing, and in sall deails such as he head lighs, gill, and ies. In addiion o is igid ansfoaion i has 4 joins which ean ha nube of possible configuaion. Visual appeaance also depends on he suounding envionen. Ligh souces will vay in hei inensiy, Fig. he ASL gesue se o be ecognized. HAD RACKIG he appoach suggesed in his aicle fo acing of hand gesue is one of he fis aeps o use a saisical shape ode fo acing. he saisical odel is epesened by a so-called Pseudo -D Hidden Maov Model (PDHMM) []. Addiionally, his P-DHMM is cobined wih a Kalan file fo oion pedicion. As

2 will be shown lae in oe deail, such an appoach has he following advanages: - he saisical shape odel is able o exploi soe a pioi-nowledge abou hand's shape. I eains soe of advanages of his appoach while being able o expand is flexibiliy and obusness. - A he sae ie, he advanage of he odel-fee appoach can be exploied o auoaically lean he feaues, which ae elevan fo he poble. hus, i cobines he advanages of odel-based and odel-fee appoach. - he syse does no ely on any oion infoaion has seveal ipoan advanages. One of he is he capabiliy of acing hands people independen of he fac ha hey ae oving o no. - he advanage ha he acing is possible in pesence of ohe oving objecs in he bacgound. hee, i is also explained ha anohe advanage of he PDHMM appoach is he exploiaion of he auoaic scaling capabiliies of HMM in geneal, ha becoe ipoan in zooing opeaions due o he fac ha he aced objec ay change is size dasically. - he HMM uli-sea echnique, i is easily possible o cobine vaious feaues, such as e. g sin colo o hand shape and o give hose feaues a diffeen weighing. - Using he peviously enioned capabiliy, i is possible o eihe design he syse fo hands pesonindependen odel o fo hands-specific odel. We develop a eal ie hand acing ehod based on he PDHMM and Kalan file, which is obus and eliable on hand acing in unconsained envionens and hen he hand egion exacion fas and accuaely. We need o conside he ade-off beween he copuaion coplexiy and obusness... Basic Hand acing Algoih We popose a novel acing ehod fo poble using wo poweful sochasic odeling echniques, naely PDHMM and Kalan file. he ey feaue of ou algoih is he fac ha i aes use of wo poweful sochasic odeling echniques, naely PDHMM and Kalan file. he inpu of he Kalan file elies on he infoaion povided by a coplex shape odel of he hand's peson of which he sucue has been auoaically leaned and acquied by he PDHMM. he dynaic infoaion need fo acing is solely geneaed by he Kalan file. While he Kalan file obains is inpu infoaion fo he PDHMM, he Kalan file iself feeds is oupu infoaion bac o he PDHMM and ipoves in his way he shape deecion pocedue of he PDHMM. his opial feedbac beween hese wo odules is anohe eason fo he poweful pefoance of he appoach. By leing only Kalan file be esponsible fo he dynaic infoaion of he acing pocess, and elying in he easueen pocess copleely on shape and colo infoaion, he acing pocedue becoes eniely independen of ohe disubing oions in he bacgound. We us fis exac hand egion in each inpu iage fae in eal ie... Measueen Veco Geneaion wih PDHMM PDHMM geneaes a easueen veco ha is uses as inpu o he Kalan file. he coponens of his veco ae he cene of gaviy of he hand peson deeced in he iage and he widh and heigh of he bounding box. he following seps ae caied ou fo ha pupose: fisly, he iage is pocessed wih a DC-based feaue exacion ehod ha is adoped fo []. An ovelap beween adjacen sapling windows ipoves he abiliy of he HMM o odel he neighbohood elaions beween he windows. he esul of he feaue exacion is wo-diensional aay of vecos. his aay is pesened o PDHMM as shown in Fig. Fig. Sochasic odel of a wo-diensional objec use a PDHMM Such a PDHMM can be consideed as a -D sochasic odel of an objec in an iage. I odels he occuence of a feaue veco sequence which can be deived fo ha objec of he objec is pe-pocessed in he sae anne as descibed above [3]. he paaees of he PDHMM consiss of he ansiion and oupu pobabiliies of he vaious HMM saes and can be leaned in ode o odel diffeen objecs. he leaning of hand shape pesons can be accoplished in he following way: Seveal hunded iage of hands wih appopiae pe-pocessing ae pesened o he PDHMM fo leaning he sucue of he hand by applying paaee esiaion ehods. Since he PDHMM can be consideed as an elasic odel, i is capable of odeling he hand in vaious posiions. In ode o be capable of locaing a hand wih a flexible envionen wih coplex bacgound, he following ipoan sep is caied ou: he PDHMM is ained wih saic iage ha show a hand wihin a coplex envionen, and no isolaed o in fon of a unifo bacgound. While he HMM paaees of he syse ae leaned successfully, seveal of he PDHMM saes will be assigned o he bacgound, and ohe saes will be assigned o he hand egions. he acual acing pocedue sas wih he

3 pesenaion of he fis fae of he acing video sequence o he ained PDHMM. If an iage conaining a hand is pesened o he above displayed especially ained PDHMM, he Viebi algoih can be used again in ode o copue he da bacgound saes and blocs assigned o he whie hands saes, hus obaining he hand's shape. he cene of gaviy of he hands ae copued fo he segenaion esul obained fo Viebi algoih by siply calculaing he appopiae oen fo he blocs inside he blac aed aea indicaing he hand (as show in Fig. ). he coodinaes of his cene of gaviy, denoed as x and y, and he size of he bounding box of he segenaion, denoed as w (widh) and h (heigh) seve as he easueen inpu he Kalan file..3. Cobinaion of PDHMM Oupu wih Kalan File and Measuing ajecoies In ode o descibe he oving hands and o epesen he esul of he acing pocedue, we use Kalan file o pedic hand locaion in one iage fae based on is locaion deeced in he pevious fae. Fis, we easue hand locaion and velociy in each iage fae. Hence, we define he sae veco as x : x =(x(), y(), v x (), v y (),w(), h()) () Whee x(), y(), v x (), v y () shows he locaion of hand (x(), y()), he velociy of hand (v x (), v y ()) and he widh and heigh of hand w(), h() in he h iage fae. We define he obsevaion veco y o pesen he locaion of he cene of he hand deeced in he h fae. he easueen veco y consiss of he locaion of he cene of he hand egion. he sae veco x and obsevaion veco y ae elaed as he following basic syse equaion: x Φ x Gw () = + y = Hx + v (3) Whee Φ is he sae ansiion aix, G is he diving aix, Φ is he obsevaion aix, w is syse noise added o he velociy of he sae veco x and v _ is he obsevaion noise ha is eo beween eal and deeced locaion. Hee we assue appoxiaely unifo saigh oion fo hand beween wo successive iage faes because he fae ineval is sho. hen Φ, G, and H ae given as follows: Φ= (4) G = (5) H = (6) he (x, y) coodinaes of he sae veco x coincide wih hose of he obsevaion veco y defined wih espec o he iage coodinae syse. Also, we assue ha boh he syse noise w and he obsevaion noise v ae consan Gaussian noise wih zeo ean. hus he covaiance aix fo w and v becoe σ w I 4 x 4 and σ v I espec, whee I 4 x 4 4x4 epesen a 4x4 ideniy aix. Finally, we foulae a Kalan file as K P H ( HP H + I (7) = x ) = Φ{ x + K ( y Hx ) } σ w = Φ( P K HP ) Φ + Q σ v x (8) P Whee x equal x, he esiaed value of x fo y,.., y -, P equals Σ / σ v, Σ epesens he covaiance aix of esiae eo of x, K is Kalan gain, and Q equals GG. hen he pediced locaion of he hand in he +h iage fae is given as (x(+), y(+)) of x +. If we need a pediced locaion afe oe han one iage fae, we can calculae he pediced locaion as follows: x Φ x + K ( y Hx ) () P { } + = + = Φ ( P KHP ) Φ σ v = x σ w ( Φ ) + Φ Q( ) (9) () Whee + is he esiaed value of x + fo y,..,y, P + equals Σ + / σ v, Σ + epesens he covaiance aix of esiae eo x +. Deeining ajecoies. We obain hand ajecoy by aing coespondences of deeced hand beween successive iage faes. Suppose ha we deec hand cenoid in he h iage fae. We efe o hand's locaion as P ' +. Fis, we pedic he locaion P + of hand in he nex fae (+) h iage fae + wih he ' pediced locaion P +. Finding he bes cobinaion. Fo he inpu infoaion of he PDHMM, conained in he veco y, he syse esiaes he sae veco x and pedics in ha way he infoaion abou bounding box, conained in he las wo diension of x. he hid and fouh diension of x delive he velociy of he hand..4. Ineacion Beween Kalan File and PDHMM he PDHMM appoach allows he elegan incopoaion of addiional feaue, such as e.g. colos o exues in he hand segenaion pocedue, by using uli-sea echniques. In his case, diffeen feaues ae deived fo each fae of he iage sequence (fo insance DC-based feaues, colo feaues and exue feaues).

4 Each diffeen feaue ype leads o a diffeen feaue sea, if all faes of he sequence ae pocessed. he saes of he PDHMM odel he occuence of each feaue wih a diffeen pobabiliy of he cobined feaues in a ceain sae ae copued as he poduc of he pobabiliies geneaed by each feaue's densiy funcion. Weighing facos can be inoduced in ode o adjus he influence of he vaious feaue seas. Consequenly, he syse can even be used o ac hand in pesence of ohe oving hands, if he hand o be aced has been acquied peviously by he PDHMM paaees, which will auoaically, lean he shape cues fo he hand shape and colo. An ipoan poins he fac ha - while veco x is consuced fo he veco y in he Kalan equaions. he updae of he veco x is used in eun as inpu o he PDHMM in ode o ipove esiaion of he veco y, hus esuling ino a coopeaive feedbac beween he Kalan file and he PDHMM. he coplee ineacion pocedue beween PDHMM and Kalan file is illusaed in Fig.3: on lef sie up sie, a oving hand has been segened, and he coodinaes of he cene of gaviy seve as easueen signal fo he Kalan file which pedics a new sae veco fo his easueen inpu and he oion equaion. On he igh uppe side, his leads o a new bounding box, which can be deived fo he updaed sae veco (inne blac ecangle). his aea is enlaged and hus yields an iage facion shown on he igh lowe side (blac-whie bold ecangle), which seves as seach aea fo he PDHMM. Fo hee, he loop is closed by yielding a new segenaion which geneaes he new easueen signal in he uppe lef pa of Fig.3. Kalan File x =(x(), y(), v x(), v y()) y =(v x (), v y (), w(), h()) x = Φ x + K ( y Hx ) { } Fig. 3 Schee of ineacion beween PDHMM and Kalan file 3. HE -COMPDHMM MODEL he pseudo -D Hidden Maov Model (PDHMM) odel [], we have developed o deal wih eal-ie hand gesue ecogniion syse [3]. he advanages of he ipoved PDHMM sucue is siplificaion, and efficien -D odel ha eains all of he useful HMM feaues and inelligen selecion of aining iages of aining sage, educing he nube of local iniu in PDHMM aining. One ha he inpu space is peclassified; a big poble is divided in seveal sall ones. his philosophy allows a poble wih a lage nube of classes o be solved easily, educing he aining ie and/o peiing a vey good soluion o be found, inceasing he ecogniion ae. aing hese advanages and anspoing i o a ie vaying space, we popose he -CoPDHMM shown by he Fig. 4. Se ewo Buffe 3.. Se ewo Inpu Veco (Video Sequence) ie oalizaion Max.. X(), Space: S X o (), Space: S o Fig. 4 he -CoPDHMM sucue X o(), Space: S o X (), Space: S Classified signal PDHMMs Banch he Se newo eceives an inpu veco and selecs a subspace fo he inpu space accoding o a siilaiy cieion. In a new -CoPDHMM sucue he inpu space of he Se newo (S ) is defined as a odified subspace of oiginal inpu space S, and he PDHMM inpu space S is he copleenay subspace. his saegy peis a oe efficien use of he spaial analysis capabiliy of he layes. Given X a ie sequence of diensional vecos x() belonging o he space S, wih ie lengh (X) saples. x o () is defined as sub-veco of x(), x o () S ; x () is a sub-veco of x, x () S and S S and S S S= S x S () As we ae dealing wih epoal seies, we need o ceae a space S o able o descibe copleely he ie vaiaions exising in he chosen subspace S o. he veco x () and he epoal lengh (x) of he inpu signal can be ando vaiable. So we popose he applicaion of a ie oalizaion pocedue on he ie independen space S. his pocedue is needed because a fixed diension veco is due in he inpu of he LVQ ha coposes he Se newo. he diension of he Se inpu laye L is elaed o he oiginal inpu ie sequence X by L = Di(S o ) x ; = and E{(X)} (3) whee Di(S o ) is he diension of he seleced subspace S o and is he fis inege geae han he expeced value of he ie lengh (X). o obain he veco X i o

5 fo he oiginal ie sequence X, we sugges he odeling of he ie sapling seies by Di(S o ) coninuous funcions using cubic spline inepolaion pocedue. Given a o diensional ie seies X i o={x i o( ), x i o( ),..., x i i o( (X o) ) and defining a se of Di(S o ) coninuous associaed funcion f i () = CubicSpline(X i o) fo i=,,..., Di(S o ). Resapling each coninuous funcion f i () in saple poins by i x o ( n) = f i ( n x o ),whee n=,,..., and o is saple peiod in an appopiae ie basis. he syse inpu veco i Di(S ) Di(S ) X =[x (),x (),...,x (),x(),x (),...,x (), Di(S ) x (), x (),...,x ()] (4) he selecion of he subspaces S o and S is vey ipoan in he -CoPDHMM sucue. he feaue selecion can be based on a class sepaabiliy cieion ha is evaluaed fo all of possible cobinaions of he inpu feaues. An Ineclass Disance Measue cieion based on he disance beween DC vecos as follows: j= i j J s = P( wi ) P( w j ) δ( ξ i, ξ jl ) (5) i= j= i j = l= whee is he nube classes, P ( wi ) is he pobabiliy of he ih class, i is he nube of paen vecos belonging o he class w i and δ ( ξ i, ξ jl ) is he DC disance based easue fo he h candidae paen of he class i o he l h candidae paen of he class j defined by d j j δ ( ξ, ξ ) = ξ ξ = ξ ξ ) (6) l l whee d is he diension of candidae space. In he - PDHMM conex, i is needed o opiize he join class sepaabiliy of he Se and PDHMM. Fo he equaion (5), he values of he class pobabiliy P( wi ) ae possible o esiae only fo he PDHMM oupu. he classificaion cieion is based on he pobabiliy of eo obained fo he PDHMM. I can be odeled accoding whee newo and J p= = J δ S + J δb (7) J δ S is he ineclass disance easue fo he Se fo he p h PDHMM, defined as i= J δ B p i j J = S P( i ) P( j ) ( x, x δ ψ ψ δ i ) (8) jl j= i p l j = l= whee P( ψ i ) is he pioi pobabiliy obained fo he i h pseudo class designed by he Se newo aining algoih, x i ae he inpu vecos in he S o space, and is he oal nube of vecos. p p i j Jδ B = P( w ) P( w ) ( x, x ) p i jl ip ψp jp ψp δ (9) i= j= i j = l= whee P( w ip ψ p ) is he condiional pobabiliy obained fo he i h class of he p h PDHMM given he Se. x i ae he inpu vecos allocaed fo he class i in he S space by he Se newo and p is he nube of classes allocaed o he Se newo. he se of feaues of x and x ha axiize he class sepaabiliy cieion given by equaion (7) has chose o define he spaces S o, S, and S o. Doing his can equied a huge aoun of copuaion ie o obain an opiu soluion, since he equaion (7) us be evaluaed fo evey C cobinaion of he inpu space diensions. 3.. Pseudo P-DHMM Consucion Since hand iages ae wo-diensional, i is naual o believe ha he DHMM, an exension o he sandad HMM, will be helpful and offe a gea poenial fo analyzing and ecognizing gesue paens. Howeve a fully conneced DHMMs lead o an algoih of exponenial coplexiy (Levin and Pieaccini, 99). o avoid he poble, he conneciviy of he newo has been educed in seveal ways, wo aong which ae Maov ando field and is vaians (Chellapa and Chaejee, 985) and pseudo DHMM (Agazzi and Kuo, 993). he lae odel, called PDHMM, is a vey siple and efficien -D odel ha eains all of he useful HMM feaues. his pape focuses on he eal-ie consucion of hand gesue PDHMM. Ou PDHMM use obsevaion vecos ha ae coposed of wodiensional Discee Cosine ansfo (D DC) coefficiens. In addiion, ou gesue ecogniion syse uses boh he epoal and chaaceisics of he gesue fo ecogniion. Unlie os ohe schees, ou syse is obus o bacgound clue, does no use special glove o be won and ye uns in eal ie. Fuheoe, he ehod o cobine hand egion and epoal chaaceisics in PDHMM faewo is new conibuion of his wo. Use of boh hand egions, feaues of locaion, angle, and velociy and oion paen ae also novel feaue in his wo. 3.. Descipion. Pseudo -DHMMs in his pape ae ealized as a veical connecion of hoizonal HMMs (λ,). Howeve i is no he only one. In ode o ipleen a coninuous fowad seach ehod and sequenial coposiion of gesue odels, he foe ype has been used in his eseach. hee ae hee inds of paaees in he PDHMMs. Howeve, since he hand iage is wo-diensional, we fuhe divided he Maov ansiion paaees ino supe-sae ansiion and sae ansiion pobabiliies; each is denoed as a P( = l ),, l and l = + =

6 aij = P( q + = j q = i), i, j M () whee denoes a supe-sae which coesponds o a HMMs λ, and q denoes a sae obseving a ie. he ode has supe-saes and he HMMs λ, is defined as sandad HMM consising of M saes. 3.. Evaluaion algoih. Le us conside a h hoizonal fae, obsevaion fuue veco O = o,..., os,. his is a one-diension feaue sequences lie ha of O in P( Oλ ) = P( O, Qλ) = π b ( o) a b ( o ) () allq q q q q q q, q,.., q = Hee, q is one of he saes fo Q, he se of saes, a ie. his is odeled by a HMM λ, wih lielihood P( O λ ). Each HMM λ ay be egaded as a supe-sae whose obsevaion is a hoizonal fae of saes. S P ( O λ ) = P( O, Qλ ) = π b ( o ) a b ( o ) () allq q q q q q s q, q,.., q s= ow le us conside a hand egion iage, which we define as a sequence of such hoizonal faes as O = O, O,.., O. Each fae will be odeled by a supe-sae o a HMM. Le Λ be a sequenial concaenaion of HMMs. hen he evaluaion of Λ given feaue sequence O of he saple iage X is P( O Λ ) = P ( O ) a P ( O ) (3) R = whee i is assued ha supe-sae pocess sas only one fo he fis sae. he P funcion is he supe-sae lielihood. oe ha boh of he Eqs. () and (3) can be effecively appoxiaed by he Viebi scoe. One iediae goal of he Viebi seach is he calculaion of he aching lielihood scoe beween O and HMM. he objecive funcion fo an HMM is defined by he axiu lielihood as S ( O, λ ) = ax a b ( o ) (4) Q s= qs qs whee Q =q,q,,q s is a sequence of saes of λ, and aq. q = π q ( O, ) λ is he siilaiy scoe beween wo sequences of diffeen lengh. he basic idea behind he efficiency of DP copuaion lies in foulaing he expession ino a ecusive fo δ s ( j) = axδ s ( i) aijb j ( os), j =,..., M, s =,..., S, =,..., K i s whee δ ( j) denoes he pobabiliy of obseving he paial sequence o,..., os in odel along he bes sae sequence eaching he sae j a ie/sep s. oe ha ( O, λ ) = δ S ( ) whee is he final sae of he qs s sae sequence. he above ecusion consiues he DP in he lowe level sucue of he PDHMM. he eaining DP in he uppe level of he newo is siilaly defined by D ( O, Λ ) = ax a ( O, λ ) (5) = ha can siilaly be efoulaed ino a ecusive fo. Hee denoes he pobabiliy of ansiion fo supe-sae o. Accoding o he foulaion descibed hus fa, a PDHMM add only one paaee se, i.e., he supe-sae ansiions, o he convenional HMM paaee ses. heefoe i is siple exension o convenional HMM. Alhough siple in fo, he ie equieen is exponenial. hans o he use of he DP echnique, his can be copued in linea ie in. Howeve when i coes o DHMM foulaion, even he DP echnique alone is no enough. One eseach diecion is he sucual siplificaion of he odel, and he pseudo DHMM is one soluion. Supe-saes Hand ROI paiion Fig. 5 PDHMM Fo each gesue hee is a P-DHMM, Fig. 5 shows a P- DHMM odel consiss of 5 supe-saes and hei saes in each supe-sae ha odel he sequence of ows in he iage. he opology of he supe-sae odel is a linea odel, whee only self ansiions and ansiions o he following supe-saes ae possible. Inside he supe-saes, hese ae linea one diension hidden Maov odel o odel each ow. he sae sequence in he ows is independen of he sae sequences of neighboing ows Repesenaion of Visual Aibues by Subses of DC Coefficiens Fo hand gesue, ou appoach is o joinly odel visual infoaion ha is localized in space, fequency, and oienaion. o do so, we decopose visual appeaance a long hese diensions. Below we explain his decoposiion and in he nex secion we specify ou visual aibues based on his decoposiion. Fis, we decopose he appeaance of he objec ino pas wheeby each visual aibue descibes a spaially localized egion on he objec. We would lie hese pas o be suied o he size of he feaues on each objec. Howeve, since ipoan cues fo hands a ay size, we need uliple aibues ove a ange of scales. We will saes

7 define such aibues by aing a join decoposiion in boh space and fequency. We would lie hese pas o be suied o he size of he feaues on each objec. Finally, by decoposing he objec spaially, we do no wan o discad all elaionships beween he vaious pas. We believe ha he spaial elaionship of he pas is an ipoan cue fo ecogniion. Wih his epesenaion, each feaue vecos now becoes a join disibuion of aibue and aibue posiion. Using DC coefficien as feaues insead of gay values of he pixels in he shif window whee os of he iage enegy is found. hey end o be insensiive o iage noise as well as iage oaions o shifs, and changes in illuinaion. o ceae visual aibues ha ae localized in space, fequency, and oienaion, we need o be able o easily selec infoaion ha is localized along hese diensions. In paicula, we would lie o ansfo he iage ino a epesenaion ha is joinly localized in space, fequency, and oienaion. o do so, we pefo a DC of iage. DC ansfo is no he only possible decoposiion in space, fequency, and oienaion. We use an ovelap of 75% beween adjacen sapling windows, we have also conside he neighboing sapling of a sapling window. Suppose we allow a defoaion of up o ± d (d is a posiive inege) pixels in eihe X o Y diecions. We have consideed all he neighboing sapling wihin he disance d in ode o deec a possible defoaion. We use a shif window o ipove he abiliy of he HMM o odel he neighbohood elaions beween he sapling blocs. Whee P ( i ) is he -D veco of he obsolue posiion of he hand pal cenoid inhe sceen a ie i. Using hese definiions we ae assigning he se laye o analyze a noalized ajecoy and he PDHMMs o analyze fine hand posues vaiaion fo pe-seleced ajecoy. Each P-DHMM is ained by hand gesue in he daabase obained fo he aining se of each of he gesue using he Bau-Welch algoih due o peanalysis achieved by Se newo, which educes he coplexiy of he poble. hen, he poposed - CoPDHMM sucue has expeced o be geneal and easily ainable. Fo gesue ecogniion, he Viebi algoih is used o deeine he pobabiliy of each hand odel. he iage is ecognized as he hand gesue, whose odel has he highes poducion pobabiliy. Due o he sucue of he P-DHMM, he os liely sae sequence is calculaed in wo sages. he fis sage is o calculae he pobabiliy ha ows of he individual iages have been geneaed by one-diensional HMMs, ha ae assigned o he supe-sages of he P-DHMM. hese pobabiliies ae used as obsevaion pobabiliies of he supe-saes of he P-DHMM. Finally, on he level second Viebi algoih is execued. 4. EXPERIMEAL RESULS 4.. Hand Deeco Fo ou expeiens, exaple is shown in he video clip ha accopanies he pape and is available fo he fis auho s websie. A few faes deeco an ealie fo CCD caea ae also shown in Figue 7. Sapling windows Fig. 6 he feaues exacion o selec he feaues which define he S subspace, he ineclass sepaabiliy easueen opiizaion pocedue descibe in he secion 3. can be applied. In his wo, we seleced fo he inpu veco, he wo coodinaes coesponding o he P () hand posiion on he sceen o copose he S subspace, geneaing wospace S and is copleenay S space, o be used by - CobPDHMM odel. I is efficien due o he naual uncoelaion exising in he hand posue, descible by S, and he hand ajecoy, descibed by S.. o obain invaiance of he oion o he caea elaive possiion, we use he noalized velociy easueen P () insead of absolue possiion, defined P( ) P( ) i i P( i ) = and P( ) = (6) P( ) P( ) i i Fig. 7 he esuls of he hand deecos cuenly a hp://

8 4.. Resuls of -CoPDHMM Based Gesue Recogniion he aining se consiss of 36 hand gesues fo vocabulay of 36 gesues including he ASL lee spelling alphabe and digis. Each one of he 36 gesues was pefoed 6 ies by one peson o ceae a daabase. he iages of he sae gesue wee aen a diffeen ies. Fo he daabase, a se coposed of 3 exaples fo each hand gesue is used o ceae he aining se. he eaining 3 exaples have eseved o he es se. hus, each se has coposed of 8 iages. he cobined using of ie spaializaion and PDHMM in he poposed -CoPDHMM odel ovecoing he classical appoaches achieving a 98.5% of coec ecogniion ae. he exaple is shown in he video clip ha accopanies he pape and is available fo he fis auho s websie. able. Recogniion aes and coplexiies of HMMs fo hand gesues ecogniion. Coplexiy Recogniion Rae Classical D-HMM 85% Opiized D-HMM 9% Classical ( ) ( D-HMM = ) 96% PD- ( ) ( HMM ( + = ) ) 98.5% -Co- PHMM ( ) (( ( ) ) + )/ M = 99.5% = nube of supe saes, ( ) = nube of saes in he h supe sae, = nube of veical obsevaions, = nube of hoizonal obsevaions, M = nube of PDHMMs banch ) 5. COCLUSIOS his wo pesened a new feaue exacion ehod using join saisics of a subse of DC coefficiens o hand gesues, and inoduced a new sucue - CoPDHMM, dedicaed o he ie seies ecogniion. he -CoPDHMM sucue uses a ie oalized Leaning Veco Quanizaion in he Se newo and PDHMM. We build he -CoPDHMM odel based upon he PDHMM, which allows i o do epoal analysis and o be used in lage se of huan oveens ecogniion syse. he esuls obained fo a se of 36 diffeen gesues of ASL show a 99.5 % of coec ecogniion ae. his esuls deonsae ha he join use of ie spaializaion echniques, naual ie pocessing echniques and PDHMM given good esuls. 6. REFERECES []. Sane, and Penland, Real-ie Aeican Sign Language Recogniion fo Video Using Hidden Maov Models, R-375, MI Media Lab, 995 [] O.E. Agazzi and S.S.Kuo, Pseudo wo-diensional hidden aov odel fo docuen ecogniion, A& echnical Jounal, 7(5), pp. 6-7, Oc, 993 [3]. D. Binh, E. Shuchi and. Ejia, Real-ie Hand acing and Gesue Recogniion Syse, Poceedings of Inenaional Confeence on Gaphics, Vision and Iage Pocessing (GVIP-5), pp , Decebe, 5 [4]. Kiishia, K. Sao, and K. Chihaa, Real-ie Gesue Recogniion by Leaning and Selecive Conol of Visual Inees Poins, IEEE ansacions on Paen Analysis and Machine Inelligence, Vol. 7, o. 3, pp , 5 [5] R. Locon, A. W. Fizgibbon, Real-ie gesue ecogniion using deeinisic boosing, Peceedings of Biish Machine Vision Confeence, pp , [6] V.I. Pavlovic, R. Shaa,.S. Huang, Visual inepeaion of hand gesues fo huan-copue ineacion, A Review, IEEE ansacions on Paen Analysis and Machine Inelligence 9(7), pp , 997 [7] J.Davis, M.Shah, Recognizing hand gesues. In Poceedings of Euopean Confeence on Copue Vision, ECCV, pp , 994 [8] Hyeon-Kyu Lee, Jin H. Ki, An HMM- based heshold odel appoach fo gesue ecogniion, IEEE ans. Paen Anal. Mach. Inell. (), pp , 999 [9] Ho-Sub Yoon, Jung Soh, Younglae J. Bae, Hyun Seung Yang. Hand gesue ecogniion using cobined feaues of locaion, angle, velociy, Paen Recogniion 34, pp. 49-5, [] Pavlovic V.I., Shaa R., Huang.S. Visual inepeaion of hand gesues fo huan-copue ineacion, A Review, IEEE ansacions on Paen Analysis and Machine Inelligence, 9(7): pp , 997 cuenly a hp://