Analyzing Human Movements from Silhouettes using Manifold Learning

Transcription

1 Analyzng Human Movemens from Slhouees usng Manfold Learnng Lang Wang and Davd Suer ARC Cenre for Percepve and Inellgen Machnes n Complex Envronmens Monash Unversy, Clayon, VIC, 38, Ausrala {lang.wang, d.suer}@eng.monash.edu.au Absrac * A novel mehod for learnng and recognzng sequenal mage daa s proposed, and promsng applcaons o vson-based human movemen analyss are demonsraed. To fnd more compac represenaons of hgh-dmensonal slhouee daa, we explo localy preservng proecons (LPP) o acheve low-dmensonal manfold embeddng. Furher, we presen wo knds of mehods o analyze and recognze learned moon manfolds. One s correlaon machng based on he Hausdorrf dsance, and he oher s a probablsc mehod usng connuous hdden Markov models (HMM). Encouragng resuls are obaned n wo represenave expermens n he areas of human acvy recognon and ga-based human denfcaon.. Inroducon Vsual analyss of human movemens [] ams o deec, rack and recognze people, and more generally, o undersand human behavours. Ineres n hs s srongly drven by a wde specrum of promsng applcaon areas such as smar survellance, percepual nerface, ec. Prevous sudes exrac varous feaures from raw vdeo daa for human moon analyss, e.g., opcal flow [], spaoemporal gradens [], local descrpors [], he racked raecores [8], ec. However, rackng s complex due o he large varably n he shape and arculaon of he human body. When usng mage measuremens n erms of spaoemporal gradens, opcal flow or oher nensy-based feaures, he recognon resuls depend grealy on he mage recordng condons. In conras, human slhouee exracon from vdeos s easer and more feasble for curren vson echnques, especally n he envronmens wh saonary cameras. Human moon can be regarded as emporal varaons of human slhouees. Thus he mehod ha we presen prefers o drecly analyze movng slhouees for human movemen analyss. Snce all mages colleced durng movemens generally le on a low dmensonal manfold embedded n he hgh dmensonal mage space, wll be deal o analyze human moons n a more compac low dmensonal space. Recenly, some promsng frameworks for dmensonaly reducon have been nroduced, e.g., somerc feaure mappng (Isomap) [7], local lnear embeddng (LLE) [6] and localy preservng proecons (LPP) [8]. Accordngly, some researchers are explorng hese newer mehods for dfferen vson applcaons, e.g., Elgammal and Lee [9] proposed an approach o nferrng 3D body pose from slhouees usng ga manfold learned by LLE. Wang e al. [] learned he nrnsc obec srucure by Isomap o enhance rackng of parameerzed conours. However, research on he manfold learnng for more complex human movemen analyss and recognon s sll very lmed. Based on he above consderaons, hs paper proposes an effecve framework o analyze human movemens from slhouees, n whch we explore LPP o acheve he low-dmensonal embeddng of dynamc slhouee daa. Two knds of mehods are hen presened o recognze he learned manfolds, one of whch s he neares manfold mehod usng he mean Hausdorrf dsance merc, and he oher s a probablsc modellng and recognon mehod based on HMM. We demonsrae real applcaons of he proposed mehod o human ga and acvy analyss. The man purpose and conrbuons of hs paper are summarzed as follows. ) Our am s o examne he feasbly of usng he feaures avalable drecly from (probably mperfec) space-me slhouees for analyzng human moons. ) We propose a general framework for vsual learnng and recognon of sequenal slhouee daa. 3) We successfully explo real applcaon of LPP o dscover nrnsc srucure of dynamc daa manfolds. 4) Two knds of recognon mehods are presened n he manfold subspace. Ther good performance for human ga and acvy recognon s examned. 5) Relavely, he proposed mehod s easy o undersand and mplemen. The use of only bnary slhouee cue make our mehod free from some problems arsng n mos prevous sudes, e.g., mperfec D or 3D feaure rackng, expensve and nose-sensve opcal flow compuaon, ec. on Vdeo and Sgnal Based Survellance (AVSS'6) /6 $. 6

2 . Relaed work In hs paper, wo areas of neres are human acvy recognon and ga-based human denfcaon whch we now brefly survey. Human acvy recognon: Tradonal mehods of human acvy analyss are based on rackng models n eher D or 3D spaces [8,5]. In he work of Yaccob and Black [8], an acon was represened by 4 curves derved from he rackng resuls of fve body pars of a cardboard people model. Oher work obans nensy or graden based feaures for moon recognon. Zelnk-Manor and Iran [] used margnal hsograms of spaoemporal gradens a a few emporal scales o cluser and recognze vdeo evens. The work of Efros e al. [] adoped a spaoemporal descrpor based on blurred opcal flow measuremens o recognze acons on balle, enns and fooball daases. There has also been sgnfcan neres n approaches ha explo local descrpors on neres pons n mages or vdeos. Schuld e al. [] consruced vdeo represenaons n erms of local space-me feaures for acon recognon. Slhouee-based mehods are becomng popular. Bobck and Davs [3] proposed vew-based emporal emplaes for represenaon and recognon of aerobcs acons. Blank e al. [4] performed acon recognon by ulzng he properes of he soluon o he Posson equaon o exrac feaures from he space-me slhouees. Ga recognon: Recenly, some mehods have been suggesed for he ask of human denfcaon by usng ga [3-7], based on he observaon ha people can recognze ohers by smply observng her gas. Mos of he exsng mehods exrac feaures from he slhouees of he person and denfy ndvduals based on hose feaures or her emporal varaons. Collns e al. [4] esablshed a mehod based on emplae machng of body slhouees n key frames for human denfcaon. Lee e al. [6] descrbed a momen-based represenaon of ga appearance for he purposes of person denfcaon and gender classfcaon. Phllps e al. [5] proposed a baselne algorhm for human denfcaon usng drec spaoemporal correlaon of slhouee mages. 3. Learnng moon manfolds I s a formdable ask o learn he complee srucure of he moon manfold n he hgh dmensonal mage space. Our dea s o embed he nonlnear manfold of human moons n a low dmensonal subspace for more compac feaure exracon and represenaon. 3.. Vsual slhouee npus Our basc assumpon s ha an assocaed sequence of foreground slhouees of a movng person can be obaned from he orgnal vdeo. Each slhouee mage s hen cenred and normalzed on he bass of keepng he aspec rao propery of he slhouee so ha he resulng mages conan as much foreground as possble, do no dsor he moon shape, and are of equal dmensons for all npu frames. We drecly use he normalzed slhouee mages as vsual npus for manfold learnng. Fgure shows wo examples of vsual slhouee npus. Fgure. Examples of vsual npus from he acons of walkng (op) and umpng ack (boom) 3.. Manfold learnng usng LPP We choose LPP o fnd manfold subspaces based on a few reasons: a) LPP can explcly model and dscover he nrnscally nonlnear manfold srucure of moons by he use of an adacency graph; b) Lke LLE, LPP has localy preservng characersc, whch makes less sensve o oulers; c) LPP s a lnear embeddng, hus s compuaonally more effcen han nonlnear mehods; and d) Many nonlnear mehods (e.g., Isomap and LLE) are defned only on he ranng daa pons and how o evaluae he maps on new es daa pons remans unclear. However LPP can be easly appled o any new daa pons. Accordng o [8], he maor procedure of manfold learnng usng LPP s descrbed as follows. Consruc he daa marx. Gven m dfferen classes of moons and each class represens a sequence of npu slhouees. Each slhouee mage wh he resoluon of r c s represened by an h-dmensonal (h=r c) vecor f n a raser-scan manner. Le f, be he h npu frame n he h class and n he number of such npus n he h class. The oal number of ranng samples s n=n +n + +n m, and he whole ranng daa se can be represened by X=[f,, f,,, f,n, f,,, f m,nm]=[x, x,, x n ]. Consruc he adacency graph. Le G be a graph wh n nodes. An edge wll be pu beween nodes and f x and x are close, where he close can be defned by ε-neghbourhoods ( x < ε, ε R ) or he K-neares x neghbours [8]. We choose he K-neares neghbours o consruc he adacency graph,.e., x and x wll be conneced f x s among he K-neares neghbours of x or x s among he K-neares neghbours of x. To measure he dsance beween x and x, we use he cosne smlary cos (, x x x x ) = () x x ( ) on Vdeo and Sgnal Based Survellance (AVSS'6) /6 $. 6

3 We also use he supervsed form of LPP (named SLPP) by negrang he class nformaon when consrucng he affny graph. Tha s, x and x wll be drecly conneced f hey belong o he same class. Choose he weghs. The wegh marx W s a sparse symmerc n n marx wh w represenng he wegh of he edge onng verces and, and f here s no such edge. There are wo knds of varaons for weghng,.e., hea kernel ( w x x / = e, R ) and smple-mnded - weghng [8]. We choose he - weghng rule. Egenmaps: Compue he egenvecors and egenvalues for he generalzed egenvecor problem [8] T XLX e = γxdx e () where D s a dagonal marx whose enres are column (or row) sums of W,.e., D = w, L=D-W s he Laplacan marx. Le he column vecors e,, e l- be he soluons of (), ordered accordng o her egenvalues λ <λ < <λ l-. The embeddng s represened by T y = E x, E = [ e, e, L, el ] (3) Each daa pon s embedded no a pon n he low dmensonal feaure space, hus a movemen s mapped no a curve wh emporal order n such subspace. 4. Recognzng moon manfolds 4.. Machng-based mehod The manfold curves of movemens can be hemselves used n a nave way for machng-based recognon. Snce he compued manfold of each moon sequence depends on s duraon and emporal shf, an deal dsance merc should be able o handle such changes. The Hausdorff dsance provdes an elegan soluon by deermnng he resemblance of one pon se o anoher. The manner n whch s compued mplcly ncludes emporal consrans beween observaon vecors. Moon smlary measure: Assume ha wo moon sequences are respecvely proeced no M (l T ) and M (l T ), where l s he reduced dmensonaly, and T and T are he duraons of hese wo moons, respecvely. A varan of he Hausdorff merc,.e., he mean value of he mnmums, s used here. ( ) ( ) ( ) ( ) ( ) M M S M =, M mean mn (4) T T M M Snce he Hausdorff dsance s orened, he smlary measure s hus modfed o ensure symmery d = S( M, M ) + S( M, M) (5) Neares-manfold classfcaon: Moon classfcaon s performed n a neares neghbour framework T ( ) c = arg mnd TM,RM (6) where RM represen he h reference moon paern, =,,, m, and TM s a es sequence. 4.. Sae-space mehod Alhough he Hausdorff dsance can reflec emporal assocaon of moons, s no explc n modellng such emporal consrans. Also, he machng-based mehod s subec o ndvdual-frame nose n npu daa. Sae-space models are more deal o explcly represen emporal ranson process of he movemen. In parcular, HMMs [] have been demonsraed o a poen ool for analyzng me-varyng daa, and sophscaed algorhms for he HMM-based learnng and recognon are avalable. Parameer ranng of HMM: In he ranng sage, we specfy he number of saes for each class of moon emprcally, and use he daa-drven desgn of HMM wh no resrcon of he opology. In deal, he model parameers descrbng an HMM s represened by he rple γ={π, a, b }, where π s he nal probably of he h sae beng he frs sae, a denoes he ranson probably of he h sae occurrng mmedaely afer he h one, and b s he probably for a feaure vecor O condoned on he h sae. Assume ha he sae se s q { s, L, s Ns } wh he number of saes N s, and he sae-condonal observaon densy s smply modelled as a mulvarae Gaussan model, we have π = Ρ q = s, N a b ( ) s = Ρ( q+ = s q = s ),, N s ( O ) = Ρ( O q = s ) = Ν( ;, ) (7) O u The parameers of he HMM are nalzed o random values and he Baum-Welch algorhm s used o esmae he parameers eravely usng he forward-backward procedure []. Gven a se of moon manfolds from he same class, we may exend he ranng o nclude mulple sequences. A each me of eraon, he conrbuon from ndvdual sequences s summed up n he procedure of forward-backward parameer esmaon. HMM-based recognon: Once he separae HMMs are raned for all classes of moons, recognon of a new es sequence can be performed based on he lkelhood compued for he npu n erms of ndvdual HMMs. Gven m classes of HMMs γ, γ,, γ m, and he assocaed manfold Y=[y, y,, y T ] of a es sequence, hs es s declared o belong o he class c represened by he HMM wh he maxmum lkelhood,.e., c = arg max P( Y γ ) (8) 5. Expermens 5.. Expermen I: Human acvy recognon on Vdeo and Sgnal Based Survellance (AVSS'6) /6 $. 6

4 Due o he lack of a common evaluaon daabase n he doman of human acvy recognon, we use a recen daabase repored n [4] *. To he bes of our knowledge, hs daabase s one of few concurren acon daabases avalable n he publc doman, and s apprecably szed n erms of he number of subecs, acons and vdeos. I consss of 8 low-resoluon vdeos (8 44, 5fps) from 9 people, each performng 9 naural acves,.e., bendng (bend), umpng-forward-on-wo-legs (ump), umpng-n-place-on-wo-legs (pump), umpng ack (ack), runnng (run), walkng (walk), gallopng-sdeways (sde), wavng-one-hand (wave), and wavng-wo-hands (wave). Togeher wh one more recenly added acvy of skppng (skp), hs daase n oal ncludes acves and 9 vdeos. The sample mages are shown n Fgure. Dfferen people have dfferen physcal szes and perform acves dfferenly boh n syles and speeds. Ths daase asks dfferen people o perform he same acves, hus provdng more realsc daa for he es of he mehod s versaly. Fgure. Example mages of each knd of acvy. From op lef o boom rgh: bend, ack, ump, pump, run, sde, skp, walk, wave, and wave, respecvely We drecly adop he masks from [4] for subsequen processng. Wheher he oher acves n hs daase are n essence perodc or no, people are asked o perform hem mulple mes n a repeve manner (excep for bendng). We exrac 98 sequences from he orgnal vdeos by perodcy deecon and segmenaon, each of whch ncludes a complee acon. The numbers of each knd of acvy sequences are respecvely 9, 3, 4, 7, 4,, 5, 6, 9, and 9 for bend, ack, ump, pump, run, sde, skp, walk, wave, and wave. We normalze all slhouee mages no he same dmenson (.e., 48 3 pxels). Each mage s denoed by a 536-dmensonal vecor, and a consderable number of such vsual npus are used o learn acvy manfolds. Fgure 3 shows spaoemporal proecons of acves, where he pons * hp:// wh same colours are from he same acvy ( Bend Jack Jump Pump Run Sde Skp Walk Wave Wave). From Fgure 3, we can see ha he SLPP has beer vsual cluserng effec for each class of acvy han LPP, bu boh of hem have compac cluserng whn he same acves. Noe ha dsrbuons of ump, run and skp are relavely closer due o her hgh smlares Fgure 3. Acvy manfolds: SLPP (lef) and LPP wh K= (rgh) To compue an overall unbased esmae of he rue recognon raes, we use he leave-one-ou rule. Each me, we frs leave one sequence ou (he sequences aken from he same orgnal vdeo s removed, whle oher acves of he same subec reman), hen ran on all he remanng sequences, and fnally classfy hs lef-ou sequence accordng o s dfferences wh respec o he res examples. If hs lef-ou sequence s classfed correcly, mus exhb a hgh smlary o a sequence from a dfferen person performng he same acvy. Afer obanng acvy manfolds, we use he mehods descrbed n Secon 4 o perform acvy recognon. Fgure 4 shows par-wse smlares (98 98) usng he mean Hausdorff dsance, n whch he darker he pxel s, he more smlar wo acvy sequences are. From each squared sub-marx along he dagonal lne (.e., 9 9, 3 3,, 9 9), we can see here are lower smlary values whn he sequences wh he same acvy, and hgher values beween he sequences wh dfferen acves. For he ranng of HMM parameers, mulple sequences ncludng he same acvy are used o esmae ndvdual HMM parameers. We specfy he number of sae n he HMM desgn n a range of 3-5 emprcally. Fgure 4. Parwse smlary: SLPP (lef) and LPP wh K= (rgh) Fgure 5 shows correc classfcaon raes (CCR) of acvy recognon, from whch he followng conclusons can be drawn: ) dynamc slhouee manfolds are ndeed nformave for classfyng human acves; ) generally, he supervsed LPP performs beer han he unsupervsed on Vdeo and Sgnal Based Survellance (AVSS'6) /6 $. 6

5 Ths s because negraes class label nformaon n ranng, hus ncreasng he dscrmnaon ably; and 3) he HMM-based mehod performs somewha beer han he Hausdorrf-dsance based mehod. Ths s probably because he sascal naure of he HMM renders overall robusness o represenaon and recognon. Fgure 5 gves a confuson marx, n whch he elemen of each row represens he probably ha ceran knd of acvy s classfed as oher knds of acves, from whch can be seen ha mos acves have perfec classfcaon, and only a few skp acves are confused. Hgh smlares among slhouees n hese moons wh smlar movng paerns may conrbue o he confuson. Correc Classfcaon Raes Hausdorff HMM LPP (K=) SLPP bend ack ump pump run sde skp walk wave wave bend Confuson Marx (Hausdorff+LPP) wave skp walkwave sde run pump ack ump Fgure 5. CCRs (lef) and confuson marx (rgh) Two mporan parameers n he LPP-based manfold learnng are he number of he neares neghbours and he reduced dmenson. All expermens have shown he choce of K n a range of -5 have smlar recognon resuls, whch suggess ha K s easly seleced o oban sable resuls. From he relaonshp beween he reduced dmensons and he recognon raes, we fnd ha SLPP generally needs a lower dmenson han LPP o oban he bes resuls; and our mehod generally does no need a hgh dmenson o oban good resuls. For conssency, here we repor all he resuls wh respec o l= and K=. We also compare he proposed mehod wh a relaed mehod descrbed n [9], whch uses lnear PCA on he flered mages usng an IIR (nfne mpulse response) fler for obanng low-dmensonal acvy descrpon. A bes recognon rae of 9.8% usng he neares cenrod manfold dsance was repored on a es daase of 8 acons and 68 sequences. We re-mplemen and evaluae hs mehod on our daase, and he bes recognon rae s 85.86%, whch s lower han any of our mehods. Ths s probably because ha, on he one hand, our mehod us uses bnary slhouees as npus, hus beng nsensve o he low colour conras and exure changes of clohes, and on he oher hand, compared wh PCA, he LPP s less sensve o oulers and nose, and more suable o fnd nrnsc srucures of acvy manfolds. The work n [4] repored an almos % recognon rae on 549 es cubes derved from he same daase (whou skppng here, bu he skppng s easly confused here). Our resuls are comparable o hose of Blank e al., bu our feaure exracon seems smpler han hers. 5.. Expermen II: Ga recognon Mos of ga recognon algorhms evaluae her performance on daases wh laeral vew because more apparen ga moons can be examned and capured n such a vewng angle. Here we selec he NLPR daase wh laeral vew [7] for hs expermen. I ncludes subecs, 4 sequences per subec, hus a oal of 8 ga sequences ( 4). These sequence mages are capured a a rae of 5 fps wh he resoluon The lengh of each sequence vares wh he pace of he walker, bu s generally above ga perods. Fgure 6 shows example mages. Relavely, hs daase s more challengng for human movemen analyss because hese sequences belong o he same walkng acvy. Bu hey are performed by dfferen subecs wh dfferen physcal srucures and moon manners. Fgure 6. Example mages n he NLPR ga daase We drecly use he slhouee daa obaned n [7] for algorhm evaluaon. Smlarly, each slhouee mage s normalzed no a 48 3 resoluon, and he supervsed or unsupervsed LPP mehods are used o learn walkng manfolds. Fgure 7 shows 3D vsualzaon ncludng only 6 subecs, n whch he same colour represens he dsrbuons of walkng sequences from he same subec x x -3 Fgure 7. Walkng manfolds: SLPP (lef) and LPP wh K=5 (rgh) We realze human denfcaon usng he leave-one-ou rule. Noe ha class n hs expermen means human ID (.e., labels -). Fgure 8 shows par-wse smlares (8 8) usng he mean Hausdorff dsance, n each of whch each squared sub-marx along he dagonal lne (.e., 4 4) has relavely hgher smlary, especally for SLPP, whch suggess dsngushable ables among dfferen subecs of walkng gas. For he HMM-based mehod, he numbers of saes are se o 5 for all sequences. Fgure 9 shows CCRs of human denfcaon. Noe ha all resuls repored here are wh respec o l= and K=5. From Fgure 9, we can draw some smlar conclusons o Expermen I. Alhough he vsual effecs of he sequences from dfferen subecs n he manfold on Vdeo and Sgnal Based Survellance (AVSS'6) /6 $. 6

6 subspace may no be as apparen as n Expermen I (wh relavely bgger varaons beween dfferen classes ), he classfcaon resuls are sasfacory because of he nroducon of emporal relaon durng recognon. Fgure 8. Parwse smlary: SLPP (lef) and LPP wh K=5 (rgh) Correc Recognon Raes Hausdorff HMM LPP (K=5) SLPP Correc Recognon Raes BenAbdelkader Collns Lee Phllps Wang 3 Hausdorff + LPP Hausdorff + SLPP HMM + LPP HMM + SLPP Rank= Rank= Rank=5 Fgure 9. CCRs (lef) and algorhm comparson (rgh) We also compare he performance of he proposed mehod wh hose of a few slhouee-based mehods descrbed n [3-7] on he same slhouee daa, as shown n Fgure 9. We fnd ha ) he HMM-based mehod always performs beer han all oher algorhms; and ) For SLPP, he Hausdorff-based mehod ouperforms all oher algorhms; for he unsupervsed LPP, performs worse han [6], bu superor o [3,4,5,7]. 6. Summary and fuure work In hs paper, our emphass has been placed on human acvy and ga analyss. To hs end, we have proposed a general framework o learn and recognze sequenal slhouee daa n low-dmensonal manfold space, and demonsraed encouragng applcaons of hs echnque. Alhough he proposed framework performs well, much work sll remans open, e.g., furher algorhm evaluaon on a larger daabase, fuson of shape and knemacs cues, vew-nvaran feaure exracon, ec. [6] L. Lee and W. Grmson, Ga analyss for recognon and classfcaon, AFG, pp. 55-6,. [7] L. Wang e al., Slhouee analyss based ga recognon for human denfcaon, PAMI, 5 (): 55-58, 3. [8] Y. Yacoob and M. Black, Parameerzed modellng and recognon of acves, CVIU, 73 (): 3-47, 999. [9] O. Masoud and N. Papankolopoulos, Recognzng human acves, AVSS, pp. 57-6, 3. [] A. Efros e al., Recognzng acon a a dsance, ICCV, : , 3. [] L. Zelnk-Manor and M. Iran, Even-based analyss of vdeo, CVPR, : 3 3,. [] C. Schuld, I. Lapev, and B. Capuo, Recognzng human acons: a local SVM approach, ICPR, 3: 3 36, 4. [3] A. Bobck and J. Davs, The recognon of human movemen usng emporal emplaes, PAMI, 3 (3): 57 67,. [4] M. Blank e al., Acon as space-me shapes, ICCV, : 395-4, 5. [5] R. Green and L. Guan, Quanfyng and recognzng human movemen paerns from monocular vdeo mages, TCSVT, 4 (): 79-9, 4. [6] S. Rowes and L. Saul, Nonlnear dmensonaly reducon by locally lnear embeddng, Scence, 9: 33 36,. [7] J.B. Tenenbaum, V. de Slva, and J.C. Langford, A global geomerc framework for nonlnear dmensonaly reducon, Scence, 9: 39-33,. [8] X. He and P. Nyog, Localy preservng proecons, NIPS, 3. [9] A. Elgammal and C-S. Lee, Inferrng 3D body pose from slhouees usng acvy manfold learnng, CVPR, : , 4. [] Q. Wang, G. Xu, and H. A, Learnng obec nrnsc srucure for robus vsual rackng, CVPR, : 7-33, 3. * Ths work s suppored by ARC Cenre for Percepve and Inellgen Machnes n Complex Envronmens, Monash Unversy, Ausrala References [] L.R. Rabner, A uoral on hdden Markov models and seleced applcaons n speech recognon, Proceedngs of he IEEE, 77: 57-86, 989. [] D. Gavrla, The vsual analyss of human movemen: a survey, CVIU, 73 (): 8-98, 999. [3] C. BenAbdelkader e al., EgenGa: moon-based recognon of people usng mage self-smlary, AVBPA, pp ,. [4] R. Collns, R. Gross, and J. Sh, Slhouee-based human denfcaon from body shape and ga, AFG,. [5] P. Phllps e al., The ga denfcaon challenge problem: daa ses and baselne algorhm, ICPR,. on Vdeo and Sgnal Based Survellance (AVSS'6) /6 $. 6