Two-Handed Gesture Tracking Incorporating Template Warping With Static Segmentation

Transcription

1 Two-Handed Gesure Trackng Incorporang Templae Warpng Wh Sac Segmenaon Yu Huang *, Thomas S. Huang *, Henrch Nemann + * Beckman Insue, U. o Illnos a Urbana-Champagn, Urbana, IL61801, US + Dep. Compuer Scence, U. o Erlangen-Nuremberg, Erlangen, 91058, German e-mal: {uhuang, huang}@p.uuc.edu, nemann@normak.un-erlangen.de. Absrac In hs paper we presen a emplae-based mehod o moon esmaon whch racks wo-handed-gesures. In our mehod he gesurng normaon o emporal moon and spaal lumnance s ull ulzed. The domnan moon o he deeced regon correspondng o he racked objec (some hand or he head) s compued. Usng hs resul he objec emplae s warped o eld a predcon emplae. Incorporaed wh sac segmenaon usng he waershed algorhm he warped emplae wll be updaed hrough comparson o each sub-regon wh he predcon emplae. Trackng resuls or a se o wo-handed command gesures are gven o demonsrae s perormance. 1. Inroducon Moon esmaon and rackng o human bod pars s a challengng bu crcal ask or modellng, recognon and nerpreaon o human behavors. In recen ears here has been an ncreased neres n rng o undersand hand gesures, ha s, some meanngul or nenonal movemen o he human arms and hands [5,10,1]. Hand gesures provde a useul nerace or humans o nerac wh ohers as well as compuers. There are wo pes o gesure neracon: communcave gesures work as a smbolc language and manpulave gesures provde mul-dmensonal conrol. However, he laer prevals n he curren use or HCI. We can dvde gesures no sac gesures (hand posures) and dnamc gesures. Indeed he hand moon conves as much meanng as her posure does. For a gesure nerpreaon ssem, here are our man componens: gesure modelng, gesure analss, gesure recognon and gesure-based applcaon ssems [,4]. Here we ocus on dnamc gesure analss,.e. gesure rackng and s moon esmaon. Human gesures, especall communcave, naurall emplo movemens o boh hands. In ac wo-handed gesures have he sronger epresson capabl. Our gesures are dened o be some wo-handed command gesures, such as orward, backward, le, rgh, begn and sop ec. Normall s supposed researchers have nshed he process o gesure modellng beore undergong gesure analss. Esng approaches o gesure modellng conss o he 3D model-based and he appearance-based mehods. In hs paper, we wll go he laer wa. 1.1 Relaed Work Some papers on wo-handed gesure analss are addressed below: Brand e.al [1] also dd work on wo-handed gesure recognon usng a sel-calbrang sereo blob rackng ssem. The propose a Coupled Hdden Markov Models (CHMMs) o model and class comple gesures. Culer and Turk [] developed a real-me gesure recognon ssem, n whch he manl esmaed opc low and segmened no deren moon blobs. So he gesure eaures come rom hese blobs, such as relave moon and sze. Hong e. al [4] ever realzed a gesure learnng and recognon ssem. The cener posons o he head and boh hands n he mage are used as eaures, whch are locaed b a skn deecon and rackng echnque. Yang and Ahuja [1] pu orward an apporach or gesure rackng (head and boh hands) n an mage sequence. The perorm sas segmenaon and skn deecon, hen ane moon parameers are esmaed aer regon machng beween consecuve rames. Imagawa e. al [5] consruc a sgn language recognon ssem usng normaon rom boh hands. The dened gesure eaures nclude global eaures, such as hand movemen and locaon, and local eaures, such as hand shape and orenaon. Anwa, he emploed mehod or eracon o hose eaures s onl rackng o skn regons and cluserng. Sherrah and Gong [9] show a mehod o rackng he head and wo hands usng Baesan nerence rom a sngle D vew.

2 Based on a Baesan Bele Neworks (BBNs), he reason abou he bod pars hrough usng oher vsual cues such as skn color, moon and local nens orenaon wh coneual knowledge. Usum and Oha [10] ever proposed a mehod o rack 3D poson, posure and shapes o human hands rom mulple-vewpon mages. The consruced ssem can allevae he sel-occluson and hand-hand occluson b emplong mulple-vew and vewpon selecon mechansm. The gesure eaures are obaned rom hand slhouees and resuls o her dsance ransorm. 1. Overvew In hs paper, we propose a emplae-based mehod o undergo smulaneousl moon esmaon and rackng o wo hands and he head. Ths mehod ull ulzes he normaon o emporal moon and spaal lumnance. Domnan moon o he racked objec (he head, he le or rgh hand) s calculaed b a robus IWLS mehod. Usng he esmaed moon parameers he objec emplae s warped o gve a predcon emplae. Resuls o sac segmenaon are ncorporaed o mod he warped emplae, and he warpng errors are ulzed o help classcaon o some doubul sub-regons around he border o he predcon emplae. The ne secon wll descrbe he general ramework o our gesure rackng mehod or each objec (he head, he le or rgh hand). In Secon 3 some epermenal resuls are presened. Conclusons are gven n Secon 4.. General Framework o Gesure Trackng We assume he head and wo hands have been deeced n he rs rame usng some skn color deecon echnque [1], now he rackng process sars. Though we regard he head and each hand as ndependen objecs, he rackng procedures are he same. The low char o our mehod s llusraed n Fgure 1: In Module Moon Esmaon we calculae domnan moon o he deeced regon correspondng o he racked objec based on a paramerc moon model, hs resul wll be ed no Module Warp o regser he las rame o he curren rame; Meanwhle n Module Sac Segmenaon a waershed algorhm s emploed on he curren rame o generae man sub-regons (.e. waershed segmens), here deren objecs are assumed labeled no deren regons; In Module Regon Analss comparson o each sub-regon wh he warped emplae s perormed o rene he objec emplae n he curren rame, combned wh warpng error analss n each sub-regon. The deals wll be eplaned n he ollowng subsecons. The rend o our ramework s comparable o [6]: her mehod ulzed several knds o edge maps rom moon, nens and predcon o updae he objec conour, nsead we deal wh he objec emplae b moon, nens and regon. Recenl n [8] a new dea acve blobs was pu orward: In her mehod he non-rgd deormaon was represened n erms o egenvecors o a ne elemen mehod; The phoomerc varaon s sll consdered b addng new brghness and conras erms n he opmzaon; The used a moded Delauna renemen algorhm o consruc a conssen rangular mesh, nsead we use he waershed algorhm o generae small waershed segmens helpul or regon deormaon n rackng..1 Domnan Moon Esmaon Here we descrbe he problem as ollows: he nerrame moon s dened as (, + 1) = ( u( ; a), ), (1) wh (, ) as he brghness uncon n me nsan, = (, ) as he coordnae o he mage pel, and u ( ; a) as he moon vecor. Whou loss o general, we smpl selec ane ransorm as he moon model, u(, ) a + a + a 0 1 u ( ; a) = =, () v(, ) a + a + a T where a = ( a, a, a, a, a, a ) are he parameers o he ane model. So, domnan moon esmaon o he gven regon R s ormulaed as he ollowng robus M-esmaor, mn E ( u, v) D = ρ ( u + v +, σ ), (3) (, ) R Fgure 1 Flow Char o he Gesure Trackng Algorhm here,, s paral dervaves o brghness uncon wh respec o, and, and he ρ - uncon can be chosen as

3 he Geman-McClure uncon as robusl he moon parameers: one s graden-based, lke he Levenberg-Marquard mehod n [8], anoher s leas ρ(, σ ) =, (4) squares-based, such as he Ierave Weghed Leas Squares + σ (IWLS) mehod. Here we use he IWLS mehod shown n wh σ as he scale parameer. Equaon (5), To solve he problem, here are wo deren was o nd w w w w w w w w w w w w w w w w w w w w w w w w where w ( r) = ψ ( r) / r, wh dervave ψ ( r ) = dρ( r) / dr and error r = u + v +. The esmae o σ s gven b a robus measure as σ = 1.486medan r. (6) The algorhm begns b consrucng he Gaussan pramd (we se up hree levels). A he coarse level moon s nall se o zero. The number o eraons s chosen as 10. When he esmaed parameers are nerpolaed no he ne level, hese parameers are used o warp (realzed b blnear nerpolaon) he las rame o he curren rame. In he curren level onl he change n he parameners are esmaed n he erave updae scheme.. Sac Segmenaon b he Waershed Algorhm In sac segmenaon, he waershed algorhm o mahemacal morpholog s a powerul mehod. I regards he graden magnude mage as a landscape where he brghness values correspond o he elevaon. Areas where randrops would dran o he same mnmum are denoed as cachmen basns, and he lnes separang adjacen cachmen basns are called dvdng lnes or waersheds. Earl waershed algorhms are developed o process dgal elevaon models and are based on local neghborhood operaons on square grds. Some approaches use ``mmerson smulaons`` o den waershed segmens b loodng he mage wh waer sarng a nens mnma [11]. Improved graden ollowng mehods are devsed o overcome plaeaus and square pel grds [3]. Here we use he ormer mehod or segmenaon o he curren rame. A severe drawback o he compuaon o waershed algorhm s over-segmenaon. Normall waershed mergng s needed when hs algorhm s perormed. Bu here over-segmenaon s welcome, so durng rackng we om he mergng process, whch saves some compuaon coss..3 Templae Warpng and Updang w w w w w w w a0 w a1 w a w a3 w a4 w a5 = w w w, (5) w w w When he moon parameers have been compued b he algorhm shown n Secon.1, we warp he deeced regon (or emplae) o he racked objec rom he las rame o he curren rame. Then he warped emplae s used o deermne whch waershed segmens ener he emplae accordng o he ollowng measure: Gven ha he number o pels belongng o he warped emplae n he subregon (waershed segmen) R s Cp and he number o all pels n R s C, a rao r s compued as r = Cp / C. (7) Based on hs measure, we dscuss urher he classcaon problem o each subregon n hese ollowng suaons: 1) When r r0 (n hs paper r0 = 0.9), we class R as par o he nal objec emplae; ) When r0 > r r1 (here r1 = 0.4), anoher measure as MAE (Mean Absolue Error) o derence beween he warped rame and he curren rame s aken no accoun, M = R w, + 1) (, ) / C. (8) ( where w (, ) s he warped mage o (, ) usng he esmaed domnan moon parameers; I he warped error M o R s smaller enough (less han a gven hreshold, or nsance, 10), R s sll regarded as par o he updaed emplae; Oherwse, we eclude R ou o he objec regon. 3) When r < r1, R wll NOT be ncluded n he updaed emplae. Fgure gve an llusraon o hs process: (a) and (b) are a par o consecuve mages n he sequence, he regon n red s he deeced objec n he las rame, he regon n pnk s he real objec n he curren rame. For sake o smplc, we assume he deeced objec s quvalen o he real objec. (c) s he warped objec usng he esmaed moon parameers, and (d) s sac segmenaon o he curren rame (he waersheds drawn wh blue color). In (e) he warped emplae (enclosed wh green lnes), waershed segmens and he real objec are supermposed n order o llusrae clearl he comparson operaon. The subregon enclosed wh red lnes belongs o he rs case, he

4 subregon wh ellow lnes belongs o he second case, and he subregon wh brown lnes should belong o he hrd case. Fnall he updaed emplae o he objec n he curren rame s shown n (). elmnaes heav occlusons; boh hands should avod occlusons wh each oher and wh he head. ) The heav deormaon o he hands, lke he openng and closng acons o he palm, wll make our mehod nvald; So we ask he sae o he palm (eher openng or closng) unchanged durng rackng; 3) Dsance beween he hand and he camera s large enough compared wh s deph change, so he ane model s able o grasp he moon o hands and he head. We dene a small se o wo-handed command gesures, ncludng gesures lke orward, backward, le, rgh, begn and sop ec., llusraed n Fgure 3. The nal posure s he same shown n 3(a), and he nal posures o hose gesures are llusraed respecvel n 3(b)-3(g). (a) Inal posure Fgure Illusraon or he Procedure o Objec Trackng In summar, he process o emplae warpng gves a predcon o he objec new poson, he comparson o each subregon wh he warped emplae can rene he predcon. In our epermens, s ound he warpng error analss s ecen o avod some msclasscaon o small regons near he racked objec n he cluered background. A Kalman ler could be consdered o smooh he esmaon o moon parameers based on a smple knemac model [10]..4 Two-Handed Gesure Trackng (b) Le (d) Backward (c) Rgh (e) Forward Normall he emplae-based mehod has he capabl o handle small paral hand-hand occlusons and sel-occlusons, whch usuall appear n wo-handed gesures. Bu, some assumpons are sll needed n advance because o lmaons o he compuer vson algorhms: 1) Two hands mus be amed a he vewng camera, whch () Sar (g) Sop Fgure 3 Dened wo-handed gesures

5 I s possble boh hands move oo close o he head durng he gesurng process. In ac, he emplae-based rackng mehod possesses ceran abl o handle hs case even wh small occlusons. Here we assume hs slgh occluson onl happens n a shor perod o me (or n a ew rames). To deal wh heav occluson, a mulple-vewpons scheme lke [10] has o be assored. In order o show he rackng resuls clearl, we use an ellpse [7] o appromae he racked objec regon n all he epermens Anwa, hose shape parameers o boh hands can be also regarded as spaal paerns o posures or a gesure a some me nsan, meanwhle he cenrod o each ellpse wll relec he wo-handed gesure rajecores. All hs daa wll be useul or wo-handed gesure recognon. he racked regon conours (n red) and correspondng ellpses (n green) on each rame. In each sequence, here are some me nsans whle he ace s closer o some hand. I onl usng he skn color normaon, s ver hard o dsngush he skn regons o he head and hose o he hand, and he regon correspondence beween consecuve rames lke [1] wll al n hs case. Our mehod elds good rackng resuls, and boh he moon and shape parameers or dnamc gesure recognon are obaned oo..5 Some Noaons Yang s mehod [1] s smlar o our procedure, bu he derence les n ha durng rackng we don consder regon correspondence beween consecuve rames (whch s no solved robusl n compuer vson) and skn color deecon. And, wha Yang s mehod ddn ulze s normaon rom emplae warpng and sub-regon analss. 3. Epermen Resuls rame 8 rame1 We realze hs approach n Vsual C++ n Penum II 400M. Now he processng speed s abou 3 seconds per rame he mage sze s A he nalzaon, we manuall pu an ellpse on each hand and he head respecvel. Fgure 4 gves he nal ellpse ng manuall and s over-segmenaon usng he waershed algorhm. Those small waershed segmens wll conrbue o emplae updae aer he warpng operaon. The rackng procedure hen sars rom he marked ellpse regon. rame15 rame18 Fgure 5 resuls rom he Le gesure sequence rame 6 rame15 (a) ellpse ng (n green) (b) oversegmenaon (n blue) Fgure 4 Trackng nalzaon Fgure 5-8 gves he resuls rom he Le, Forward, Rgh and Sop gesure sequence respecvel (normall n a sequence here are 0-50 rames). We llusrae smulaneousl rame 3 rame 8 Fgure 6 Trackng resuls or he Forward gesure sequence

6 4. Concluson In hs paper we have proposed a wo-hands-gesure rackng mehod, s characer s ull ulzaon o he objec spao-emporal eaures n rackng.. The emplae warpng onl gves a predcon o he racked eaure poson, and he comparson o each sub-regon wh he warped are able o mod he predcon resul. Lke [1], we have esmaed he moon (ane) parameers, shape eaures (ellpse ng) and obaned (regon cenrod) rajecores o boh hands wh respec o he head. All he normaon could be useul or wo-handed dnamc gesure recognon. The dsadvanages o our mehod are also clear n he epermens. Frs, we rel on he moon esmaon o he racked objec; Even hough he IWLS mehod s more sable, we sll conroned he dvergence n nonlnear eraons. Second, whle we nroduce he sac segmenaon resul our mehod has srong dependence on he perormance o he emploed waershed algorhm. We epec he mages n he sequence are wh enough resoluon, and he moon blur are also no welcome. In uure, we plan o consder he varaons o llumnaon durng rackng [8], whch s also an mporan acor n rackng. Acknowledgemen Ths research s suppored parall b he NSF Grans EIA and IIS cameras, pp , IEEE CVPR 99. [11] Vncen L, Solle, 'Waersheds n dgal spaces: an ecen algorhm based on mmerson smulaons', IEEE T-PAMI, 13(6): , [1] Yang M, Ahuja N, Recognzng hand gesure usng moon rajecores, CVPR 99, pp89-897, rame 6 rame 11 rame 19 rame 4 Fgure 7 Trackng resuls or he Rgh gesure sequence Reerences [1] Brand, M e.al. Coupled Hdden Markov Models or comple acon recognon, IEEE CVPR 97, pp , [] Culer R, Turk M. "Vew-based Inerpreaon o Real-me Opcal Flow or Gesure Recognon", IEEE Inernaonal Conerence on Auomac Face and Gesure Recognon [3] Gauch J, 'Image segmenaon and analss va mulscale graden waershed herarches', IEEE T-IP, 8(1): 69-79, [4] Hong P e.al. Gesure modelng and recognon usng ne sae machnes, IEEE Inernaonal Conerence on Auomac Face and Gesure Recognon, pp , 000. [5] Imagawa I e. al, Recognon o local eaures or camera-based sgn language recognon ssem, Proc. ICPR 00, pp , 000. [6] Nguen H., Worrng M., Muleaure objec rackng usng a model-ree approach, IEEE CVPR, pp , 000. [7] Saber E, Tekalp A, Face deecon and acal eaure eracon usng color, shape and smmer-based cos uncons, ICPR 96, pp , [8] Sclaro S. and Isdoro J., Acve blobs, ICCV 98, [9] Sherrah J, Gong S., Trackng dsconnuous moon usng Baesan nerence, ECCV 00, 000. [10] Usum A., Oha J. Mulple-hand-gesure rackng usng mulple rame 7 rame 11 rame 14 rame 19 Fgure 8 Trackng resuls or he Sop gesure sequence