/99 $10.00 (c) 1999 IEEE

Transcription

1 Recognzng Hand Gesure Usng Moon Trajecores Mng-Hsuan Yang and Narendra Ahuja Deparmen of Compuer Scence and Beckman Insue Unversy of Illnos a Urbana-Champagn, Urbana, IL 611 fmhyang,ahujag@vson.a.uuc.edu hp://vson.a.uuc.edu Absrac We presen an algorhm for exracng and classfyng wo-dmensonal moon n an mage sequence based on moon rajecores. Frs, a mulscale segmenaon s performed o generae homogeneous regons n each frame. Regons beween consecuve frames are hen mached o oban 2-vew correspondences. Affne ransformaons are compued from each par of correspondng regons o defne pxel maches. Pxels maches over consecuve mages pars are concaenaed o oban pxel-level moon rajecores across he mage sequence. Moon paerns are learned from he exraced rajecores usng a medelay neural nework. We apply he proposed mehod o recognze hand gesures of Amercan Sgn Language. Expermenal resuls show ha moon paerns n hand gesures can be exraced and recognzed wh hgh recognon rae usng moon rajecores. 1 Inroducon In hs paper, we presen an algorhm for exracng wo-dmensonal moon felds of objecs across a vdeo sequence and classfyng each as one of a se of a pror known classes. The algorhm s used o recognze dynamc vsual processes based on spaal, phoomerc and emporal characerscs. An applcaon of he algorhm s n sgn language recognon where an uerance s nerpreed based on, for example, hand locaon, shape, and moon. The performance of he algorhm s evaluaed on he ask of recognzng complex hand gesures of Amercan Sgn Language (ASL). The algorhm consss of wo major seps. Frs, each mage s paroned no regons usng a mulscale segmenaon mehod. Regons beween consecuve frames are hen mached o oban 2-vew correspondences. Affne ransformaons are compued from each par of correspondng regons o defne pxel maches. Pxel maches over consecuve mage pars are concaenaed o oban pxel-level moon rajecores across he vdeo sequence. Pxels are also grouped based on her 2-vew moon smlary o oban a moon based segmenaon of he vdeo sequence. Only some of he movng regons correspond o vsual phenomena of neres. Boh he nrnsc properes of he objecs represened by mage regons and her dynamcs represened by he moon rajecores deermne wheher hey comprse an even ofneres. For example, s suffcen o recognze mos gesures n ASL n erms of shape and locaon changes of palm regons. Therefore, palm and head regons are exraced ou n each frame and he palm locaons are specfed wh reference o he usually sll head regons. To recognze moon paerns from rajecores, we use a me-delay neural nework (TDNN) [11]. TDNN s a mullayer feedforward nework ha uses medelays beween all layers o represen emporal relaonshps beween evens n me. An npu vecor s organzed as a emporal sequence, where only he poron of he npu sequence whn a me wndow sfed o he nework a one me. The me wndow s shfed and anoher poron of he npu sequence s gven o he nework unl he whole sequence has been scanned hrough. The TDNN s raned usng sandard error backpropagaon learnng algorhm. The oupu of he nework s compued by addng all of hese scores over me, followed by applyng a nonlnear funcon such as sgmod funcon o he sum. TDNNs wh wo hdden layers usng sldng npu wndows over me lead o a relavely small number of ranable parameers. We adop TDNN o recognze moon paerns because gesures are spao-emporal sequences of feaure vecors defned along moon rajecores. Our expermenal resuls show ha moon paerns can be learned by a me-delay neural nework wh hgh recognon rae. 2 Relaed Work Snce Johansson's semnal work [7] ha suggess human movemens can be recognzed solely by mo /99 $1. (c) 1999 IEEE

2 on nformaon, moon profles and rajecores have been nvesgaed o recognze human moon by several researchers. In [8] Ssknd and Morrs conjecure ha human even percepon does no presuppose objec recognon. In oher words, hey hnk vsual even recognon s performed by a vsual pahway whch s separaed from objec recognon. To verfy he conjecure, hey analyze moon profles of objecs ha parcpae n dfferen smple spaalmoon evens. Ther racker uses a mxure of color based and moon based echnques. Color based echnques are used o rack objecs defned by se of colored pxels whose sauraon and value are above ceran hresholds n each frame. These pxels are hen clusered no regons usng a hsogram based on hue. Movng pxels are exraced from frame dfferences and dvded no clusers based on proxmy. Nex, each regon (generaed by color or moon) n each frame s absraced by an ellpse. Fnally, feaure vecor for each frame s generaed by compung he absolue and relave ellpse posons, orenaons, veloces and acceleraons. To classfy vsual evens, hey use a se of Hdden Markov Models (HMMs) whch are used as generave models and raned on moves of each vsual even represened by a se of feaure vecors. Afer ranng, a new observaon s classfed as beng generaed by he model ha assgns he hghes lkelhood. Expermens on a se of 6 smple gesures, pck up," pu down," push," pull," drop," and hrow," demonsrae ha gesures can be classfed based on moon profles. Bobck and Wlson [3] adop a sae based approach o represen and recognze gesures. Frs, many samples of a gesure are used o compue s prncpal curve [5] whch s parameerzed by arc lengh. A by-produc of calculang he curve s he mappng of each sample pon of a gesure example o an arc lengh along he curve. Nex, hey use lne segmens of unform lengh o approxmae he dscrezed curve. Each lne segmen srepresened by avecor and all he lne segmens are grouped no a number of clusers. A sae s defned o ndcae he cluser o whch a lne segmen belongs. A gesure s hen defned by an ordered sequence of saes. The recognon procedure s o evaluae wheher npu rajecory successfully passes hrough he saes n he prescrbed order. Conrased o her work where each example of a gesure s a sngle rajecory n space, each gesure n our work s represened by a se of moon rajecores correspondng o he moons of dfferen pars of, say, he palm, nsead of a sngle represenave pon. Thus, each example of a gesure n our work s represened by a se of moon rajecores. Our expermenal resuls show ha an ensemble of rajecores yelds beer generalzaon Recenly, Isard and Blake have proposed he CON- DENSATION algorhm [6] as a probablsc mehod o rack curves n vsual scenes. Ths mehod s a fuson of he sascal facored samplng algorhm wh asochasc model o search amulvarae parameer space ha s changng over me. Objecs are modeled as a se of parameerzed curves and he sochasc model s esmaed based on he ranng sequence. Expermens on he proposed algorhm have been carred o rack objecs based on her hand drawn emplaes. Black and Jepson [2] exend hs algorhm o recognze gesures and facal expressons n whch human moons are modeled as emporal rajecores of some esmaed parameers (whch descrbe he saes of a gesure) over me. The major dfference beween our approach and hese mehods s ha we propose a mehod o exrac moon rajecores from an mage sequence whou hand drawn emplaes [6] or dsnc rackable cons [2]. Moon paerns are hen learned from he exraced moon rajecores. No pror knowledge s assumed or requred for he exracon of moon rajecores, alhough doman specfc knowledge can be appled for effcency reasons. 3 Moon Segmenaon To capure he dynamc characerscs of objecs, we segmen an mage frame no regons wh unform moon. Our moon segmenaon algorhm processes an mage sequence wo successve frames a a me. For a par of frames, (I ;I +1 ), he algorhm denfes regons n each frame comprsng he mulscale nraframe srucure. Regons a all scales are hen mached across frames. Affne ransforms are compued for each mached regon par. The affne ransform parameers for regon a all scales are hen used o derve a sngle moon feld whch shensegmened o denfy he dfferenly movng regons beween he wo frames. The followng secons descrbe he major seps n he moon segmenaon algorhm. 3.1 Mulscale Image Segmenaon Mulscale segmenaon s performed usng a ransform descred n [1] whch exracs a herarchy ofre- gons n each mage. The general form of he ransform, whch maps an mage o a famly of aracon force felds, s defned by F(x; y; ff g (x; y);ff s (x; y)) = RR R d g( I;ff g (x; y)) d s (~r; ff s (x; y)) ~r jj~rjj dwdv where R = doman(i(u; v))nf(x; y)g and ~r = (v x) ~ +(w y) ~ j. The parameer ff g denoes a homo /99 $1. (c) 1999 IEEE

3 geney scale whch reflecs he homogeney of a regon o whch a pxel belongs and ff s s spaal scale ha conrols he neghborhood from whch heforce on he pxel s compued. The homogeney of wo pxels s gven by he Eucldean dsance beween he assocaed m-dmensonal vecors of pxel values (e.g., m = 3 for a color mage): I = ji(x; y) I(v; w)j The spaal scale parameer, ff s, conrols he spaal dsance funcon, d s ( ), and he homogeney scale parameer, ff g, conrols he homogeney dsance funcon, d g ( ). One possble form for hese funcons sasfyng crera dscussed n [1] s unnormalzed Gaussan: q d g ( I;ff g ) ο 2ßffg 2N I (;ffg) 2 ρ p 2ßff 2 d s (~r; ff s ) ο s N jj~rjj(;ffs); 2 jj~rjj» 2ff s ; jj~rjj > 2ff s The force feld encodes he regon srucure n a manner whch allows easy exracon. Regon boundares correspond o dvergng force vecors n F and regon skeleons correspond o convergng force vecors n F. An ncrease n ff g causes less homogeneous srucures o be encoded and an ncrease n ff s causes large srucures o be encoded. 3.2 Regon Machng The machng of moon regons across frames s formulaed as a graph machng problem a four dfferen scales where scale refers o he level of deal capured by he mage segmenaon process. Three parons of each mage are creaed by slcng hrough he mulscale pyramd a hree preseleced values of ff g. Regon parons from adjacen frames are mached from coarse o fne scales, wh coarser scale maches gudng he fner scale machng. Each paron s represened as a regon adjacency graph, whn whch each regon s represened as a node and regon adjacences are represened as edges. Regon machng a each scale consss of fndng he se of graph ransformaon operaons (edge deleon, edge and node machng, and node mergng) of leas cos ha creae an somorphsm beween he curren graph par. The cos of machng a par of regons akes no accoun her smlary wh regard o area, average nensy, expeced poson as esmaed from each regon's moon n prevous frames, and he spaal relaonshp of each regon wh s neghborng regons. Once he mage parons a he hree dfferen homogeney scales have been mached, machngs are hen obaned for he regons n he frs frame of he frame par ha were denfed by he moon segmenaon module usng he prevous frame par. The mach n he second frame for each of hese moon regons s gven as he unon of he se of fnes scale regons ha comprse he moon regon. Ths gves a fourh mached par of mage parons, and s consdered o be he coarses scale se of maches ha s ulzed n affne esmaon. The deals of he algorhm can be found n [9]. 3.3 Affne Transformaon Esmaon For each par of mached regons, he bes affne ransformaon beween hem s esmaed eravely. Le R be he h regon n frame and s mached regon be R +1. Also le he coordnaes of he pxels whn R be (x j ;y ), wh j j =1:::jRj where jrj s he cardnaly ofr, and he pxel neares he cenrod of R be (μx ; μy ). Each (x j ;y )smappedbyan j affne ransformaon o he pon (^x j ; ^y ) accordng j o x j y j! R =» ^x j ^y j x A j μx k y j μy + T ~ +1 μx k + +1 μy k where he subscrp k denoes he eraon number, and R[ ] denoes a vecor operaor ha rounds each vecor componen o he neares neger. The affne ransformaon comprses a 2 2 deformaon marx, A k, and a ranslaon vecor, Tk ~. ndcaor funcon, (x; y) = ρ 1; (x; y) 2 R ;else he amoun of msmach s measured as By defnng he (M )= P x;y ji (x; y) I +1 (^x; ^y)j (x; y)+ +1 (^x; ^y) (x; y) +1 (^x; ^y) The affne ransformaon parameers ha mnmze M are esmaed eravely usng a local descen creron. 3.4 Moon Feld Inegraon The compued affne parameers gve a moon feld a each of he four scales. These moon felds are hen combned no a sngle moon feld by akng he coarses moon feld and hen performng he followng compuaon recursvely a four scales. A each mached regon, he mage predcon error generaed by he curren moon feld and he moon feld a he nex fner scale are compared. A any regon where he predcon error usng he fner scale moon mproves by a sgnfcan amoun, he curren moon s replaced by he fner scale moon. The resul s a se of bes mached" regons a he coarses accepable scales. Λ /99 $1. (c) 1999 IEEE

4 3.5 Moon Feld Segmenaon The resulng moon feld M ~ ;+1 s segmened no areas of unform moon. We use a heursc ha consders each par of bes mached regons, R and R, j whch share a common border, and merges hem f he followng relaon s sasfed for all (x k ;y ) and k (x jl ;y jl ) ha are spaally adjacen o one anoher: jj ~ M ;+1 (x k ;y k ) ~ M ;+1 (x jl ;y jl )jj max(jj ~ M ;+1 (x k ;y k )jj; jj ~ M ;+1 (x jl ;y jl )jj) <m ffg where m ff g s a consan less han 1 ha deermnes he degree of moon smlary necessary for he regons o merge. The segmened moon regons are each represened n MS ;+1 by a dfferen value. Because each ofhe bes mached regons have maches, he maches n frame + 1 of he regons n MS ;+1 are known and comprse he coarses scale regons ha are used n he affne esmaon module for he nex frame par. I should be noed ha he moon segmenaon does no necessarly correspond o he movng objecs n he scene because he moon segmenaon s done over a sngle moon feld. Nonrgd objecs, such as humans, are segmened no mulple, pecewse rgd regons. In addon, fas objecs movng a raes less han one pxel per frame canno be denfed. Handlng boh hese suaons requres examnng he moon feld over mulple frames. Fgure 1 shows frames from an mage sequence of a complex ASL sgn called cheerleader" and Fgure 2 shows he resuls of moon segmenaon. Dfferen moon regons are dsplayed wh dfferen gray levels. Noce ha here are several moon regons whn he head and palm regons because hese pecewse rgd regons have unform moon. 4 Color and Geomerc Analyss Moon segmenaon generaes regons ha have unform moon. However, only some of hese moon regons carry mporan nformaon for moon paern recognon. To recognze hand gesures consdered here, s suffcen o exrac he moon regons of head and palm regons. Towards hs end, we use color and geomerc nformaon of palm and head regons. Human skn color has been used and proved o be an effecve feaure n many applcaons. We use a Gaussan mxure o model he dsrbuon of skn color pxels from a Mchgan daabase of 2,447 mages whch consss of human faces from dfferen ehnc groups. We use CIE LUV color space and dscard he lumnescence value of each pxel o mnmze he effecs of lghng condon. The parameers n he Gaussan mxure are esmaed usng an EM algorhm. A moon regon s classfed o have skn color f mos of he pxels have probables of beng skn color above a hreshold. Coupled wh moon segmenaon, moon regons of skn color can be effcenly exraced from mage sequences. Snce he shape of human head and palm can be approxmaed by ellpses, and he human hand s a hn recangular regon, moon regons ha have skn color are merged unl he shape of he merged regon s approxmaely ellpc or recangular. The parameers of a recangular shape can be obaned from he boundng box of each regon easly. The orenaon of an ellpse s calculaed from he axes of he leas momen of nera. The exens of he major and mnor axes of he ellpse are approxmaed by he exens of he regon along he axs drecons, and hus generae he parameers for he ellpse. The larges ellpc regon exraced from an mage s denfed as human head and he nex wo smaller ellpc regons are palm regons. Fgure 1 shows he mage sequence of a complex ASL sgn called cheerleader" and Fgure 3 shows he resuls of color and geomerc analyss on he moon regons. 5 Moon Trajecores Alhough moon segmenaon generaes affne ransformaons ha capure moon deals by machng regons a fne scales, s suffcen o use coarser moon rajecores of denfed palm regons for gesure recognon consdered n hs paper. Affne ransformaon of palm regon n each frame par s compued based on equaons n Secon 3.3. The affne ransformaons of successve pars are hen concaenaed o consruc he moon rajecores of he palm regon. Fgure 4 shows such rajecores for a number of frames n he mage sequence cheerleader." Snce all pxel rajecores are shown ogeher, hey form a hck blob. Fgure 5 showsa1o1subsamplng of he moon rajecores. 6 Moon Paern Classfcaon We employ TDNN o classfy gesural moon paerns of palm regons snce TDNNs have been demonsraed o be very successful n learnng spaoemporal paerns. TDNN s a dynamc classfcaon approachn ha he nework sees only a small wndow of he moon paern and hs wndow sldes over he npu daa whle he nework makes a seres of local decsons. These local decsons have o be negraed no a global decson a a laer me. In her semnal work, Wabel e al. [11] demonsraed excellen resuls /99 $1. (c) 1999 IEEE

5 (a) frame 14 (b) frame 16 (c) frame 19 (d) frame 22 (e) frame 25 (f) frame 29 (g) frame 31 (h) frame 34 () frame 35 (j) frame 37 (k) frame (l) frame 44 (m) frame 46 (n) frame 49 (o) frame 52 (p) frame 55 Fgure 1: Image sequence of ASL sgn cheerleader" (a) frame 14 (b) frame 16 (c) frame 19 (d) frame 22 (e) frame 25 (f) frame 29 (g) frame 31 (h) frame 34 () frame 35 (j) frame 37 (k) frame (l) frame 44 (m) frame 46 (n) frame 49 (o) frame 52 (p) frame 55 Fgure 2: Moon segmenaon of he mage sequence cheerleader" (pxels of he same moon regon are dsplayed wh same gray level and dfferen regons are dsplayed wh dfferen gray levels) (a) frame 14 (b) frame 16 (c) frame 19 (d) frame 22 (e) frame 25 (f) frame 29 (g) frame 31 (h) frame 34 () frame 35 (j) frame 37 (k) frame (l) frame 44 (m) frame 46 (n) frame 49 (o) frame 52 (p) frame 55 Fgure 3: Exraced head and palm regons from mage sequence cheerleader" (a) #14-#16 (b) #16-#19 (c) #19-#22 (d) #22-#25 (e) #25-#29 (f) #29-#31 (g) #31-#34 (h) #35-#37 () #37-# (j) #-#44 (k) #44-#46 (l) #46-#49 (m) #49-#52 (n) #52-#55 Fgure 4: Exraced gesural moon rajecores from segmens of ASL sgn cheerleader" (snce all pxel rajecores are shown, hey form a hck blob) /99 $1. (c) 1999 IEEE

6 Gesure negraon Oupu Layer #37 #29 #22 #46 #49 # #52 #44 #14 #25 #55 #31 #19 #34,#35 #16 wndow lengh= wndow lengh=1 18 slos 37 slos Hdden Layer (a) Moon rajecores of a (b) Moon rajecory of one sample se of palm pons for palm pon for he ASL sgn he ASL sgn cheerleader" cheerleader" wndow lengh=5 x y v θ Hdden Layer 1 46 slos Inpu Layer 5 slos Fgure 6: Archecure of TDNN (c) Moon rajecores of a (d) Moon rajecory of one sample se of palm pons for palm pon for he ASL sgn he ASL sgn cheerleader" cheerleader" Fgure 5: Exraced gesural moon rajecores (subsampled by a facor of 1) of ASL sgn cheerleader" for phoneme classfcaon usng TDNN and showed ha acheves lower error raes han hose acheved by a smple HMM recognzer. The desgn of TDNN s aracve because s compac srucure economzes on weghs and makes possble for he nework o develop general feaure deecors. Mos mporanly, s emporal negraon a he oupu layer makes he nework shf nvaran (.e. nsensve o he exac posonng of he gesure). Fgure 6 shows our TDNN archecure for he expermens, where posve values are shown as gray squares and negave values as black squares. The npus o our TDNN are vecors of (x; y; v; ) for moon rajecores exraced from a gesure mage sequence, where x, y are posons wh respec o he cener of he head, and v, are magnudes and angle of velocy respecvely; he oupus are he gesure classes; and he learnng mechansm s error backpropagaon. 7 Expermens We use a vdeo daabase of ASL sgns for expermens. Each vdeo consss of an ASL sgn whch lass abou 3 o 5 seconds a 3 frames per second wh mage sze of 1 1 n Quckme forma. Fgure 1 shows one complex ASL gesure from he sequence cheerleader." Noe ha he hand movemen consss of roaon and repeons. Each mage sequence of he gesures n he expermen has o 1 frames. #46 #44 #25 # #49 #22 #52 #55 #29 #19 #37 #14 #31 #34,#35 #16 Dscardng he frames n whch palms do no appear n he mages (.e. frames n sarng and endng phase), each mage sequence has abou 5 frames. Moon regons wh skn color are denfed by her chromac characerscs. These regons are hen merged no palm and head regons shown n Fgure 3 based on geomerc analyss dscussed n Secon 4. Affne parameers of mached palm regons are compued, whch gve pxel moon rajecores for each mage par. By concaenang he rajecores for consecuve mage pars, connuous moon rajecores are generaed. Fgures 4 shows he exraced moon rajecores from a number of frames and Fgure 5 shows he rajecores from he whole mage sequence. Noe ha he moon rajecores of palm regon mach he movemen n he real scene well. Tranng of TDNN s performed on he corpus of % of he exraced dense (38 on he average) rajecores from each gesure, usng an error backpropagaon algorhm. The res % of he rajecores are hen used for esng. Based on he expermens wh ASL gesures, he average recognon rae on he ranng rajecores s 98:14% and he average recognon rae on he unseen es rajecores s 93:42%. Snce dense moon rajecores are exraced from each mage sequence, he recognon rae for each gesure can be mproved by a vong" scheme (.e. he majory rules) on he classfcaon resul of each ndvdual rajecory. The resulng average recognon rae on he ranng and esng ses for gesure recognon are 99:2% and 96:21%, respecvely. 8 Dscusson and Concluson We have descrbed an algorhm o exrac and recognze moon paerns usng rajecores. For concreeness, he expermens have been carred ou o recognze hand gesures n ASL. Moon segmenaon /99 $1. (c) 1999 IEEE

7 s performed o generae regons wh unform moon. Movng regons wh salen feaures are hen exraced usng color and geomerc nformaon. The affne ransformaons assocaed wh hese regons are hen concaenaed o generae connuous rajecores. These moon rajecores encode he dynamc characerscs of hand gesures and are classfed by a me-delay neural nework. Our expermens demonsrae ha hand gesures can be recognzed, wh hgh accuracy, usng moon rajecores. The conrbuons of hs work can be summarzed as follows. Frs, a general mehod ha exracs moon rajecores s developed. Ths s n conras o much work on gesure recognon ha uses color hsogram racker [8] [4] [2], magnec sensors [3], hand drawn emplae [6], and sereo [1] o oban a represenaon of he gesure. Second, we use a TDNN o recognze gesures based on he exraced rajecores. Usng an ensemble of rajecores helps acheve hgh recognon raes. I would be neresng o compare hese recognon raes wh hose obaned usng oher recognon mehods such as HMM, CONDEN- SATION algorhm [6] [2] and prncpal curve [3]. Acknowledgemens The suppor of Advanced Telecommuncaon Research Inernaonal s graefully acknowledged. References [1] N. Ahuja. A ransform for mulscale mage segmenaon by negraed edge and regon deecon. IEEE Trans. Paern Anal. Mach. Inell., 18(12): , [2] M. J. Black and A. D. Jepson. A probablsc framework for machng emporal rajecores: Condensaon-based recognon of gesure and expressons. In Proceedngs of European Conference on Compuer Vson, pages , [3] A. F. Bobck and A. D. Wlson. A sae-based approach o he represenaon and recognon of gesure. IEEE Trans. Paern. Anal. Mach. Inell., 19(12): , [4] J. L. Crowley and F. Berard. Mul-modal rackng of faces for vdeo communcaons. In Proceedngs of IEEE Conference on Compuer Vson and Paern Recognon, pages 6 645, [5] T. Hase and W. Suezle. Prncpal curves. Journal of Amercan Sascal Assocaon, 84(6):52 516, [6] M. Isard and A. Blake. Condensaon - condonal densy propagaon for vsual rackng. Inernaonal Journal of Compuer Vson, 29(1):5 28, [7] G. Johansson. Vsual percepon of bologcal moon and a model for s analyss. Percepon and Psychophyscs, 73(2):1 211, [8] J. M. Ssknd and Q. Morrs. A maxmum-lkelhood approach o vsual even classfcaon. In Proceedngs of he Fourh European Conference on Compuer Vson, pages 347 3, [9] M. Tabb and N. Ahuja. 2-d moon esmaon by machng a mulscale se of regon prmves. IEEE Trans. Paern Anal. and Mach. Inell., submed. [1] C. Vogler and D. Meaxas. Asl recognon based on a couplng beween hmms and 3d moon analyss. In Proceedngs of he Sxh Inernaonal Conference on Compuer Vson, pages , [11] A. Wabel, T. Hanazawa, G. Hnon, K. Shkano, and K. Lang. Phoneme recognon usng me-delay neural neworks. IEEE Trans. on Acouscs, Speech, and Sgnal Processng, 37(3): , /99 $1. (c) 1999 IEEE