J. Sofware Engneerng & Applcaons, 00, 3, 060-066 do:0.436/jsea.00.35 Publshed Onlne Noveber 00 (hp://www.scrp.org/journal/jsea) 3D Huan Pose Esaon fro a Monocular Iage Usng Model Fng n Egenspaces Gel Bo, Kasunor Onsh, Tesuya Takguch, Yasuo Ark Graduae School of Engneerng, Kobe Unversy, Kobe, Japan. Eal: {akgu, ark}@kobe-u.ac.jp Receved Ocober 4 h, 00; revsed Ocober 6 h, 00; acceped Noveber 3 rd, 00. ABSTRACT Generally, here are wo approaches for solvng he proble of huan pose esaon fro onocular ages. One s he learnng-based approach, and he oher s he odel-based approach. The forer ehod can esae he poses rapdly bu has he dsadvanage of low esaon accuracy. Whle he laer ehod s able o accuraely esae he poses, s copuaonal cos s hgh. In hs paper, we propose a ehod o negrae he learnng-based and odelbased approaches o prove he esaon precson. In he learnng-based approach, we use regresson analyss o odel he appng fro vsual observaons o huan poses. In he odel-based approach, a parcle fler s eployed on he resuls of regresson analyss. To solve he curse of he densonaly proble, he egenspace of each oon s learned usng Prncpal Coponen Analyss (PCA). Fnally, he proposed ehod was esaed usng he CMU Graphcs Lab Moon Capure Daabase. The RMS error of huan jon angles was 6. degrees usng our ehod, an proveen of up o 0.9 degrees copared o he ehod whou egenspaces. Keywords: HOG, Regresson Analyss, Egenspaces, Parcle Fler, Pose Esaon. Inroducon The 3D confguraon esaon of cople arculaed objecs fro onocular ages has been wdely suded. Once he echnology s perfeced, here wll be poenal applcaons n any felds relaed o huan pose and kneac nforaon, such as copuer nerfaces ha ulze gesure npu, neracon wh he robos, vdeo survellance, and eneranen. However, onocular huan pose esaon s ereely challengng due o he coplcaed naure of huan oon and he led aoun of nforaon n D ages. The ehods of huan pose esaon can be suarzed no wo approaches: learnng-based and odelbased. In he learnng-based ehod [-4] feaures are drecly eraced fro he age, and he appng funcon for he huan poses s raned usng he age feaures. Through hs appng, he huan pose of an age can be esaed. Once he ranng process s copleed, he pose esaon s perfored rapdly. However, he esaon precson decreases when he npu age s no ncluded n he ranng daa. In he odelbased ehod [5-8], he pose esaon ehod follows Bayes' heore and odels he poseror probably densy usng observaon lkelhood or cos funcon. Ths ehod s copuaonally epensve, n general, and dependen on an nal pose. To solve hese probles, we propose a ehod o negrae he learnng-based and odel-based ehods o prove he esaon accuracy. An nal pose s deerned usng regresson analyss n he learnng-based approach, and he esaon ehod s swched o a parcle fler n he odel-based approach o prove he precson. Unforunaely, gven he large densonaly of a 3D huan odel space, s alos praccal o apply parcle flerng drecly as a large nuber of parcles s requred o adequaely approae he underlyng probably dsrbuon n he huan pose space. Therefore, we frs use PCA o learn he egenspace of each oon. Then, he opal huan pose s effcenly searched n he egenspaces seleced accordng o he esaed ype of huan oon n he npu ages.. Feaures.. Iage Feaures We use he HOG feaure [9], whch can descrbe he shape of an objec n an appearance-based approach [0]. HOG was proposed as a graden-based feaure for general objec recognon, where HOG descrbes he feaure
3D Huan Pose Esaon fro a Monocular Iage Usng Model Fng n Egenspaces 06 Inpu Iage Background subracon & noralzaon Graden copuaon Overlappng blocks & noralzaon Weghed voe no spaal & orenaon cells Noralzaon over overlappng blocks HOG descrpor Block: b w b h cell Fgure. The flow of feaure eracon. over he gven regon. Ths eans ha HOG can represen he rough shape of an objec. Moreover, snce HOG can olerae a range of varyng llunaon, s suable for pose esaon []. Fgure presens he coplee processng chan of he HOG feaure eracon brefly. We wll now dscuss he HOG encodng algorh n hs secon n deal.... Graden Copuaon Before eracon of he HOG feaure, we frs separae he huan regon fro he npu age usng background subracon, where he sze of he huan regon s noralzed, and he huan regon s locaed n he cener poson of he age. Then he age graden s copued as follows. f ( y, ) I (, y) I (, y) y, () f y ( y, ) Iy (, ) Iy (, ) y, where f and f y denoe and y coponens of he age graden, respecvely. I( y, ) denoes he pel nensy a he poson (, y ). The agnude y (, ) and orenaon (, y) are copued by y f y f y () (, ) (, ) y(, ) (, y) an ( fy(, y) f (, )) y (3) In order o ake he HOG feaure nsensve o clohng and he facal epresson, we use he unsgned orenaon of he age graden, whch s copued as follows. ( y, ), f ( y, ) 0 ( y, ) (4) (, y), oherwse... Orenaon Hsogras The graden age s dvded no cells c w ch pels as shown n Fgure. A each cell, he orenaon (, y) s quanzed no c b orenaon bns, weghed by s agnude y) (, o ake a hsogra. Tha s, a hsogra wh he c b orenaons s copued for each cell...3. Block Noralzaon Fgure shows he orenaon hsogra eraced a Cell: C w C h pel C b orenaon Fgure. Block noralzaon. every cell and he larger spaal blocks wh bw bh cells. Snce a cell has c b orenaons, he feaure denson of each block s d b b w b h c b for each block. Le v denoe a feaure vecor n a block, hj denoe he unnoralzed hsogra of he cell n he poson (, j), bw, j bh n a block. The feaure vecor of a ceran block s noralzed as follows. hj h j ( ) (5) v Snce he noralzaon s done by overlappng he block, he hsogras h j are repeaedly noralzed by a dfferen block... 3D Huan Model The huan body can be regarded as a ul-jon objec ha ransfors no varous shapes. In addon, he segened par ha connecs wo jons can be regarded as rgd objec. Therefore, s possble o epress a 3D huan odel wh jon angles. In hs research, we used he oon capure daa n he CMU Graphcs Lab Moon Capure Daabase []. The 3D huan odel s represened by 56 jon angles, so he denson of he pose sae vecor s 56. Fgure 3 shows an eaple of he 3D huan odel. 3. The Pose Esaon Mehod Huan pose esaon s carred ou usng wo approaches. In he odel-fng ehod, huan pose s esaed by an eraon procedure [5]. However, s probleac n ha he nal value has o be gven anually. Therefore, we adop he learnng-based ehod o auoacally oban he nal value, whch s
06 3D Huan Pose Esaon fro a Monocular Iage Usng Model Fng n Egenspaces N pars of ages and 3D huan odel angles HOG feaure z LMS Esaon R 3D huan odel Fgure 5. Regresson-based esaon ehod. Fgure 3. An eaple of 3D huan odel. 3D huan pose n egen space 3D huan pose for nal value Inpu age z Regresson Generae saples Selec egen space and projecon Lkelhood evaluaon Egen space 3D huan poses generaed by parcle fler Back projecon 3D huan poses n hgh densonal space Fng Oupu (3D huan pose) Fgure 4. Pose esaon syse. negraed no he eraon procedure of he odelbased ehod. One drawback n he odel-based ehod s he hgh densonaly of he sae space, whch akes he algorh copuaonally neffecve. Thus we use PCA o reduce he denson of he pose sae and esablsh he egenspace of each oon. Fgure 4 shows he proposed pose esaon syse. 3.. Learnng-Based Mehod Usng Regresson Analyss In he learnng-based ehod, we adop regresson analyss [,3] o esae he pose of npu age. Le denoe he vecor coposed of he angles a jons n he 3D huan odel. The relaon beween he HOG feaure vecor z and 3D pose vecor s lnearly approaed usng he followng forula: Rz (6) where s resdual error vecor. The 3D huan pose s esaed by converng he npu age feaure z o he 3D huan odel vecor. In odel ranng (esae R ), a se of n ranng pars (, z) n s gven (n our case, 3D poses and he correspondng age HOG feaures). The converson ar R s esaed by nzng he ean square error. Packng he ranng daa no 3D pose ar X ( n ) and age feaure ar Z ( zz zn ), he ranng s perfored as follows: R: argn RZ X (7) R In he esng phase, he 3D huan posure vecor s esaed by converng HOG feaures vecor z usng he copued converson ar R. Fgure 5 deonsraes he regresson-based esaon ehod. 3.. Model-Based Mehod Usng a Parcle Fler In he odel-based ehod, a parcle fler [4] s eployed. Followng a noaon slar o [4], we defne as he sae vecor a e, wh z denong he easureens a e. Furherore, le all he easureens unl e be gven by Z ( z,, z). Parcle flerng based on Bayes' heore s used o oban a poseror probably p ( Z) a each e-sep usng all he avalable nforaon as shown bellow. p ( Z) pz ( ) p ( Z ) (8) Ths equaon s evaluaed recursvely as descrbed below. The fundaenal dea of parcle flerng s o approae he poseror probably densy funcon (pdf) over by a weghed saple se S. Suppose ha N saples fro he poseror pdf p ( Z) are avalable and denoe he as (). Then he h weghed () () saple a e s represened by s (, () ) S. Frs, a cuulave hsogra of all he saples' weghs s copued a e. Then, accordng o () each parcle's wegh, s nuber of successors s deerned accordng o s relave probably n hs cuulave hsogra. A he predcon sep, he new sae s copued usng he followng Chapan-
3D Huan Pose Esaon fro a Monocular Iage Usng Model Fng n Egenspaces 063 Kologorov equaon. p ( Z ) p ( ) p ( Z ) d (9) A he easureen sep, he new sae s weghed accordng o s lkelhood o he new easureen z. The poseror densy p ( Z) s represened by a se of (0) (0) ( N) ( N weghed parcles ( s ), ) ( s, ), where he n weghs pz ( s ( n) N ( ) ( ). The new sae N ( n) ( ) s ) are noralsed so ha can be esaed by p ( Z) ( ) (0) The easureen sep of Equaon (0) and he predcon sep of Equaon (9) ogeher for he Bayes' forulaon Equaon (8). In he ordnary odel-based ehod, a parcle fler s ulzed o ach he 3D odel wh he npu age, and he nal value needs o be se anually [5,6]. In our ehod, he nal value can be obaned fro he learnng-based ehod. Therefore, he forer anual confguraon wll be replaced by an auoac esaon process. The pose esaed by regresson analyss s used as an nal value, and he parcles are sapled around. The lkelhood of each parcle s evaluaed as s wegh, and he parcles are generaed by a resaplng process based on he wegh. Afer repeang resaplng several es, he parcle wh he hghes lkelhood s consdered as he fnal sae. Egen spaces Hgh densonal pose space Fgure 6. Orgnal space and egenspaces. 3.3. Sae Space The 3D huan odel was nroduced n Subsecon.. In hs secon, s reaed as he sae vecor of parcle fler. However, n a real envronen, he denson of sae space s norally very hgh. Tha would cause boh low copuaonal effcency and poor convergence perforance. We propose he ehod of ulzng an egenspace of each oon consruced by PCA as a oon pror, whch consrans correspondng oon [7]. Sulaneously, s possble o search effcenly n he lowdensonal space usng denson reducon. Suppose ha he nuber of oon ypes s M. When PCA s carred ou usng he ranng daa of a ceran oon M, he 3D huan pose s projeced no he egenspace as follows: P ( ) () where denoes he ean pose vecor of a ceran oon, P s he base vecor ar, and denoes he pose vecor n he egenspace. Because PCA s carred ou for M ypes of oons n he ranng daa, M knds of egenspaces are consruced. The pose vecor s projeced o he egenspace of each oon and s used as he sae of parcle as shown n Fgure 6. The denson reducon usng PCA s decded accordng o he 95% cuulave proporon rae. 3.4. Lkelhood In he lkelhood calculaon sage, he sae vecor of every parcle s convered o a 3D huan pose n a hgh denson space usng PCA Inverse Transforaon. Then, we use MAYA o produce he CG age ha represens he pose of every parcle. Ths CG age copared wh he npu age. The perforance of he parcle fler depends largely on he age feaures ha are used o calculae he lkelhood. The deal age feaures should rean sable n varous scenaros and be easy o erac. In our case, we adop wo feaures o consruc weghng funcon: HOG for represenng he huan conforaon and slhoueng for evaluang he huan regon. Confguraon: HOG represens huan conforaon and s robus regardng changes n color, clohes and llunaon. The CG age s generaed fro he parcle sae, and HOG s eraced (Fgure 7(b)). The dsance beween he npu age and he sae s calculaed as follows: Ehog (, ) ( z, z) () D
064 3D Huan Pose Esaon fro a Monocular Iage Usng Model Fng n Egenspaces (a) (b) (c) Fgure 7. Feaure eracon (a) CG age (b) HOG descrpor (c) slhouee. (, zz) D c c ( z z ) c c' z z c' (3) where D denoes he denson of he HOG, c (, zz) s -dsance beween z and z. z ndcaes he c -h eleen of he HOG feaure vecor. Regon: The slhouee age s eraced usng he background subracon ehod (Fgure 7(c)). The slhouee can be used for evaluaon wh sably because s also robus o he changes of color, clohes and llunaon. Afer he slhouee s eraced, once agan a pel ap s consruced, hs e wh foreground pels se o and back ground o 0, and he dsance s copued as follows: K Eregon (, ) ( p (, )) (4) K where K s he nuber of pels, and p (, ) he values of he bnary EX OR operaon beween npu age and sae. Ne, fness C( ) on he age s copued as follows. C( ) ep ( E (, ) E (, )) (5) hog regon In our ehod, he soluon search s resrced o he egenspace of he correspondng oon. Neverheless, he pose of an age can be furher consraned n a ceran range of he egenspace, where we use he rajecory of oon as a pror consran. We regard he sequence of vecors n he egenspace as he rajecory of hs oon. Then he dsance s calculaed beween sae vecor and vecors of ranng daa n egenspaces, and can be used for he lkelhood evaluaon as a penaly. Fnally, he bes soluon s obaned by he rule shown n Equaon (6). L C( ) (6) 3.5. Selecng he Egenspace In our ehod, s necessary o selec he proper egenspace accordng o he esaed oon of he npu age because he egenspace s consruced for each oon. Frs, any saples M,,, S are ebedded no each egenspace usng he ranng daa. The ean dsance beween he saples and nal pose obaned by he regresson analyss s copued, and he npu oon s decded as he oon wh he neares ean dsance. The ean dsances are copued n he egenspace and he hgh-densonal space respecvely as follows. S S D (7) S S (8) D Here D and D denoe he denson of egenspace and an orgnal space respecvely, and denoes he low-densonal pose vecor o whch s apped n an egenspace. denoes he hgh-densonal pose vecor n whch s revered o an orgnal space usng PCA Inverse Transforaon. S s he nuber of he saples. The oon of he npu age s decded by nzng he su of wo dsances defned by Equaon (7) and Equaon (8) as shown n Equaon (9). f ( ) s defned as a funcon o deerne whch an egenspace he oon belongs o: [,, M] f (arg n ) (9) 3.6. Esaon of he Huan Orenaon Even f he orenaon of he huan s unknown, he huan pose can be esaed wh regresson analyss because he age feaure changes accordng he orenaon. However, f he orenaon s no esaed n he odel-based ehod, he age canno be ached. Consequenly, he ages n all orenaons of 360 degrees are generaed by CG usng he resuls of he regresson analyss, and huan orenaon s esaed usng Equaon (5) as follows: ˆ arg a C ( ) (0) 360 where C ( ) denoes he fness beween npu age and he generaed age fro pose wh orenaon. The odel-fng s carred ou usng he esaed orenaon ˆ obaned by Equaon (0).
3D Huan Pose Esaon fro a Monocular Iage Usng Model Fng n Egenspaces 065 4. Esaon of he Huan Orenaon 4.. Eperen Seup We conduced he eperen usng he CMU Graphcs Lab Moon Capure Daabase. Frs, we use oon capure daa o produce a CG anaon whose resoluon s 640 480 pels. Then we roae he fgure on he horzonal plane n egh drecons and ake he CG age n each drecon as eperen daa. We carred ou eperens for hree knds of oon (walkng, runnng, and jupng). The ages used for ranng are suarzed n Table. If he es oon s a cyclc oveen, only four ages of he ypcal pose are needed o represen a ceran cyclc oon. In order o represen he oon suffcenly, we used egh ages n each drecon for a connuous acon. Table lss he deals of he es daa. 4.. Eperen Resuls RMS of absolue dfference errors was copued beween he rue jon angles and esaed jon angles, by Equaon (). ndcaes he nuber of jons. D (, ) ( ) od80 () Fgure 8 shows he RMS error over all jons angles for all he oons. The esaon precson was proved by eraon procedure. The horzonal as ndcaes he nuber of eraons. The regresson analyss resuls are shown n he prary eraon. Afer he ne eraon, pose esaon s acheved by repeaedly applyng he parcle fler. The resuls show ha he accuracy of esaon can be - RMS error (degree) RMS error (degree) 7.4 7. 7 6.8 6.6 6.4 6. 6 8 7.5 7 6.5 6 5.5 5 4.5 Proposed ehod Whou rajecory 3 4 5 6 Ieraon Fgure 8. Pose esaon resuls. 3 4 5 6 Ieraon Fgure 9. Esaon precson n egenspaces. Gven Unknown Table. The nuber of ranng daa. pose The nuber of fraes orenaon Toal (8 orenaons) Walkng 90 70 Runnng 89,5 Jupng 9,536 Toal 47 3,768 Table. The nuber of es daa. pose The nuber of fraes orenaon Toal (8 orenaons) Walkng 8 64 Runnng 8 64 Jupng 8 64 Toal 4 9 Table 3. Confuson ar [%]. Walkng Runnng Jupng Walkng 84.38 3..5 Runnng 9.37 78.3.5 Jupng 5.63 4.68 79.69 Fgure 0. Eperenal resul of he real age. proved sgnfcanly because of he use of he rajecory as a consran. Table 3 shows he resul of egenspace selecon descrbed n Subsecon 3.5. One reason for he ncreased accuracy s ha, n our ehod, egenspace selecon s no appled o sequence daa bu o jus one frae age. Therefore, he oon recognon accuracy s no so
066 3D Huan Pose Esaon fro a Monocular Iage Usng Model Fng n Egenspaces good. The selecon of a proper egenspace wll grealy affec he fnal esaon resuls. Fgure 9 copares he accuracy of wo dfferen esaon ehods. The blue broken lne represens he scenaro n whch a oon ype of npu age has been gven, and s esaon s carred ou n he correspondng egenspace. Coparng hese resuls wh he red lne for an unknown oon ype, can be concluded clearly ha our ehod shows a beer perforance n ers of esaon precson. The resuls of huan pose esaon fro real ages [8,9] are shown n Fgure 0. The frs row s he npu real ages and he second row represens he synhec ages generaed fro he esaed poses. I s confred ha our ehod works effecvely for he real ages. 5. Conclusons In hs paper, we presened an approach o esae 3D huan pose fro a onocular age, whch negraes he learnng-based and odel-based esaon ehods no one fraework. Furherore, hrough he consrucon of an egenspace, ore effcen parcle fler perforance s obaned. Consequenly, he precson of esaon s obvously proved, and eperenal resuls deonsraed ha our approach s effecve. The parcle fler dd no need o have he nal value provded anually, and could ge he convergence soluon n he suaon wh less eraons. In fuure work, n order o furher prove he esaon accuracy, we are plannng o use vdeo daa for he npu daa because he eporal coherence beween fraes ay provde useful nforaon regardng he selecon of he egenspace of each oon. REFERENCES [] A. Agarwal and B. Trggs, 3D Huan Pose fro Slhouees by Relevance Vecor Regresson, IEEE Copuer Socey Conference on Copuer Vson and Paern Recognon, Vol., 004, pp. 88-888. [] X. Zhao, H. Nng, Y. Lu and T. Huang, Dscrnave Esaon of 3D Huan Pose Usng Gaussan Processes, Proceedngs of 9h Inernaonal Conference on Paern Recognon (ICPR 08), Deceber 008, pp. -4. [3] C. Snchsescu, A. Kanauja and D. N. Meaas, 3 BM E : Dscrnave Densy Propagaon for Vsual Trackng, IEEE Transacons on Paern Analyss and Machne Inellgence, Vol. 9, No., Noveber 007, pp. 030-044. [4] H. Nng, Y. Hu and T. Huang, Effcen Inalzaon of Mures of Epers for Huan Pose Esaon, 5h IEEE Inernaonal Conference on Iage Processng (ICIP008), Ocober 008, pp. 64-67. [5] M. Lee, I. Cohen, A Model-Based Approach for Esang Huan 3D Poses n Sac Iages, IEEE Transacons on Paern Analyss and Machne Inellgence, Vol. 8, No. 6, June 006, pp. 905-96. [6] T. Jaeggl, E. Koller-Meer and L. V. Gool, Learnng Generave Models for Monocular Body Pose Esaon, Proceedngs of he 8h Asan Conference on Copuer Vson, Vol., 007. [7] S. Hou, A. Galaa, F. Callee, N. Thacker and P. Broley, Real-e Body Trackng Usng a Gaussan Process Laen Varable Model, IEEE h Inernaonal Conference on Copuer Vson, Ocober 007, pp. -8. [8] G. Peng, W. Aleander, A. O. Balan and M. J. Black, Esang Huan Shape and Pose fro a Sngle Iage, IEEE h Inernaonal Conference on Copuer Vson (ICCV009), Sepeber 009, pp. 38-388. [9] N. Dalal and B. Trggs, Hsogras of Orened Gradens for Huan Deecon, IEEE Copuer Socey Conference on Copuer Verson and Paern Recognon (CVPR005), Vol., June 005, pp. 886-893. [0] G. Mor and J. Malk, Recoverng 3D Huan Body Confguraons usng Shape Cones, IEEE Transacons on Paern Analyss and Machne Inellgence, Vol. 8, No. 7, 006, pp. 05-06. [] K. Onsh, T. Takguch and Y. Ark, 3D Huan Posure Esaon Usng he HOG Feaures fro Monocular Iage, 9h Inernaonal Conference on Paern Recognon (ICPR008), Deceber 008, pp. -4. [] CMU Huan Moon Capure Daabase. Avalable onlne: hp://ocap.cs.cu.edu/ [3] A. Fossa, M. Salzann and P. Fua, Observable subspaces for 3D huan oon recovery, IEEE Conference on Copuer Verson and Paern Recognon (CVPR009), June 009, pp. 37-44. [4] M. Isard and A. Blake, Condensaon-Condonal Densy Propagaon for Vsual Trackng, Inernaonal Journal of Copuer Vson, Vol. 9, No., 998, pp. 5-8. [5] J. Deuscher, A. Blake and I. Red, Arculaed Body Moon Capure by Annealed Parcle Flerng, IEEE Copuer Socey Conference on Copuer Verson and Paern Recognon, Vol., June 000, pp. 6-33. [6] L. Ye, Q. Zhang and L. Guan, Use Herarchcal Genec Parcle Fler o Fgure Arculaed Huan Trackng, Inernaonal Conference on Muleda and Epo (ICME008), 008, pp. 56-564. [7] X. Zhao and Y. Lu, Trackng 3D Huan Moon n Copac Base Space, IEEE Workshop on Applcaons of Copuer Vson (WACV 07), February 007, p. 39. [8] H. Sdenbladh, M. Black and D. Flee, Sochasc Trackng of 3D Huan Fgures Usng D Iage Moon, Copuer Vson ECCV 000, Vol. 843, 000, pp. 70-78. [9] R. Urasun, D. Flee and P. Fua, Monocular 3D Trackng of he Golf Swng, IEEE Copuer Socey Conference on Copuer Verson and Paern Recognon (CVPR005), Vol., June 005, pp. 93-938.