Eended MHT Algorhm for Mulple Obec Trackng Long Yng, Changsheng Xu aonal Lab of Paern Recognon, Insue of Auomaon, Chnese Academy of Scences, Beng, Chna {lyng, csu}@nlpr.a.ac.cn Wen Guo, Elecronc Engneerng eparmen, Shandong Insues of Busness Technology, Yana, Chna wguo@nlpr.a.ac.cn ABSTRACT In hs paper, we propose an mproved effcen MHT algorhm negraed wh HSV-LBP appearance and repulson-nera model for mul-obec rackng. Smulaneously rackng mulple obecs s crcal o vdeo conen analyss and vrual realy. The man ssues we wan o address n hs paper are negraon of vdeo mage pach nformaon no daa assocaon and ambguous observaons caused by obecs n close promy. A lkelhood funcon of HSV-LBP hsogram wh sraegy of emplae updang s consruced. A repulson-nera model s adoped o eplore more useful nformaon from ambguous deecons. Epermenal resuls show ha he proposed approach generaes beer raecores wh less mssng obecs and deny swches. Caegores and Subec escrpors.4.8 [Image Processng and Compuer Vson]: Scene Analyss moon, sereo, rackng. General Terms Algorhms, Epermenaon Keywords Mul-obec Trackng, aa assocaon, MHT, Paches rackng. ITROUCTIO Trackng s he mos fundamenal ask n compuer vson. The ably o smulaneously rack mulple obecs s crcal o vdeo conen analyss and vrual realy, whch provdes mporan nformaon for hgher level analyss and decson. Mul-obec rackng ams a nferrng raecores for each obec from vdeo sequence, whch can be consdered as a spaoemporal groupng problem. All mage regons are classfed as deecon of specfc obec or background. In real senses, here are many condons such as varance of number of obecs, smlar appearances of dfferen obecs, varance of obec appearance, comple neracons, long me occlusons and cluer background, whch generae he uncerany and make mul-obec rackng a Permsson o make dgal or hard copes of all or par of hs work for personal or classroom use s graned whou fee provded ha copes are no made or dsrbued for prof or commercal advanage and ha copes bear hs noce and he full caon on he frs page. To copy oherwse, or republsh, o pos on servers or o redsrbue o lss, requres pror specfc permsson and/or a fee. ICIMCS, Sepember 9, 0, Wuhan, Hube, Chna. Copyrgh 0 ACM 978--4503-600-//09 $0.00. challengng problem. Several frameworks ha ncorporae appearance, moon and arsng-dsappearng models have been proposed o fnd a on soluon, whch make sae space of mul-obec rackng very comple. To effcenly resolve, he sae space s usually resrced o a fne se of canddae locaons, eher by hresholdng he observaon lkelhood or by regularly dscrezng he locaon space. Unlke sngle obec rackng whch focus on appearance presenaon and dynamc search, he core ssue n mul-obec rackng s daa assocaon. Earler works only used curren and pas observaons o esmae he curren sae. Jon Probablsc aa Assocaon Fler (JPA [] based on Kalman fler red o esmae saes for fed number of obecs frame by frame by summng dfferen hypoheses. Label swch easly occurred n crowded scenes. Mehods base on parcle fler [, 3] eplored mulple hypoheses smulaneously, parly resolvng hs problem. However, he parameers needed o be carefully se o ensure convergence. Usng boh pas and fuure nformaon o esmae he curren sae s usually more effecve o overcome he ambgues of occlusons, spurous observaons and mssng observaons. An nuve dea s mulple hypoheses rackng (MHT [4] whch s frs used n radar sgnal processng. Orgnal MHT algorhm enumeraes all possble assocaons n a perod of me o fnd ou curren global opmal soluon. However, hs edon wll suffer eponenally ncreasng hypoheses. Varous heurscs are developed o overcome hs compley, such as prunng, gang, cluserng, -scan-back logc, and k-bes hypoheses [5]. Base on he formulaon of mul-obec rackng proposed n MHT algorhm, many approaches have been proposed recenly. MCMC [6] daa assocaon uses Meropols-Hasngs algorhm o consruc an rreducble and aperodc Markov chan, whch s employed o effcenly sample from he poseror dsrbuon of sae space and oban appromae opmal soluon. Some approaches consruc graph models for global assocaon, converng mul-obec rackng o quadrac neger program [7] whch s resolved by graph cu, or o neger lnear program (ILP [8] whch s appromaely solved hrough LP-relaaon. Ohers consruc nework flow models [9, 0], mappng he mamum-aposeror (MAP daa assocaon problem no cos-flow nework wh non-overlap consran on raecores, whch s solved by nework opmzaon. Alhough many new approaches have been proposed, effcen MHT s he mos classcal approach for daa assocaon n rackng by deecon framework. I s praccal n real-me
rackng wh moderae compuaonal compley, and performs robusly for applcaon. In hs paper, we propose an mproved MHT algorhm for mulple pedesran head-shoulder srucures rackng. The man conrbuon of our work s o adap moon model for paches rackng, negrae appearance model no probably calculaon, and propose a repulson-nera model smlar o [3] o eplore more dynamc nformaon for daa assocaon from ambguous observaons when obecs are n close promy. The res of he paper s organzed as follows: Secon gves he overvew of our rackng framework; Secon 3 revews he effcen MHT algorhm whch s he bass of our mehod; Secon 4 demonsraes our mproved MHT algorhm; Secon 5 repors epermenal resuls on survellance vdeo and analyss; We conclude he paper n secon 6.. OVERVIEW The rackng framework proposed n hs paper concenraed on daa assocaon, whch ncludes four seps: ( consrucon of appearance descrpor for each observaon; ( rack rees growng and lkelhood calculaon; (3 sae recalculaon of ambguous hypoheses; (4 generaon of k-bes hypoheses and prunng. In he frs sage, HSV-LBP hsogram s consruced for each observaon n newly receved vdeo frame. In he second sage, rack rees grow accordng o affnes beween raecores a prevous me and observaons, and appearance emplae of new raecory hypoheses s consruced. In he hrd sage, raecory hypoheses assocaed wh ambguous observaons are found, and a repulson-nera model s adoped o recalculae saes of hese hypoheses. The fnal sage consss of remander seps of MHT algorhm ncludng cluserng, k-bes hypoheses, and prunng. Ths framework s llusraed n Fgure. Vdeo Frames Verfy deermned rack nodes Obec deecon Prunng rack rees or removng Updae clusers Generae k-bes hypoheses Feaure Erac HSV+LBP Calculae lkelhood Eend rack rees Recalculae ambguous sae Fgure. The framework of proposed mehod Updae appearance emplae 3. EFFICIET MHT ALGORITHM Our work s based on MHT algorhm. To make hs paper selfconaned, we revew he effcen MHT algorhm [5]. 3. Formulaon of Trackng Task In rackng by deecon framework, measuremens can be receved a every momen. Each measuremen may eher ( belong o a prevously known obec, ( be he sar of a new obec, (3 be a spurous measuremen. For obecs ha are no assgned measuremens, here s he possbly of (4 ermnaon, e.g. movng ou of he feld of vew and alernavely he possbly of (5 connuaon of an obec. k Le Z be he se of all measuremens unl me k. A sngle raecory hypohess s defned as an ordered ls of measuremens. An assocaon hypohess k l s defned as a se of raecory hypoheses. An addonal raecory 0 conans all spurous measuremens. When measuremens Zk ( a me k are receved, a parcular global hypohess k l can be derved from ceran hypohess k ml ( a me k-. l ( k denoes he specfc se of assgnmens of he orgns of all measuremens receved a me k wh all he obec assumed by he paren hypohess k, whch s defned o ml ( conss of measuremens from known obecs, measuremens from new obecs, spurous measuremens, and obsolee obec. A consran ha one observaon orgnaes from a mos one obec and one obec has a mos one assocaed observaon s mposed o reduce he sze of he search space. The probably of a global hypohess can be calculaed usng Bayes rule, P Z P Z k k Z c k k k k l ( l (, m( l, k k k k l ( m( l, m( l P k Z P Z where c s a normalzaon consan. Mul-obec rackng ask s o fne ou he MAP hypohess. I s assumed ha a measuremen obeys Gaussan dsrbuon f s assocaed wh obec : ( ; (, ( z k z k k S k ( where z ˆ ( k k denoes he predced measuremen for obec, and S ( k s he nnovaon covarance. If he probables of deecon and ermnaon of rack are P and P respecvely, and he numbers of spurous measure-mens and new obecs are assumed o obey Posson dsrbued wh denses F and respecvely, he poseror probably of an assocaon hypohess can be epress as: m P Z z k c k k k m F ( ( P ( P ( P ( P P Z k k F F l( m where, and are ndcaor varables defned by: z ( k comes from a known obec 0 oherwse k f obec n ml ( s deeced a me k 0 oherwse k f obec n ml ( ermnaes a me k 0 oherwse 3. Effcen Soluon In praccal applcaon, rack rees are consruced nsead of keepng an eplc hypohess ree. The compuaonal compley of MHT algorhm s manly caused by dramac ncrease of combnaons. Tracks ha do no compee for common ( (3
measuremens can be paroned no separae clusers whch makes a sgnfcanly reducons n number of hypoheses. Enumerang all possble global hypoheses for opmal soluon s mpraccal. Generang he k-bes hypoheses drecly can acheve more effcen performance and an appromaely opmal soluon. Ths s mplemened by opmzng Mury s algorhm [] n 3 ( k. To reduce compuaonal compley whch ncreases wh accumulaon of me, -scan-back prunng whch assumes ha any ambguy a me k s resolved by me k+ s used. Moreover, hypoheses whose probably s much lower han he bes hypohess can be prunng by a hreshold of probably rao. 4. IMPROVE MHT ALGORITHM MHT algorhm s wdely used n radar and sonar sgnal processng. When hs mehod s appled n mage paches rackng such as pedesran head-shoulder srucures, he affny merc and dynamc model should be adaped o he ask. Kalman fler s used as dscree-me dynamc of he obec n classcal MHT, whch esmaes curren moon sae, calculaes moon affny and predcs moon sae when no observaon s receved. In pedesran rackng, lnear dynamc model s suffcen. However, addonally scale varables should be negraed no moon saes for more dscrmnave descrpon. T The moon sae s defned as, d, y, dy, w, h, where and y are he poson coordnaes, d and dy are velocy componens, w and h are half wdh and half hegh of he deeced pach. 4. Inegrae Appearance Model Classcal MHT only ulzes moon nformaon for daa assocaon. However, negrang appearance affny no vdeo rackng can help o overcome some ambguous n daa assocaon. Compared wh RGP color space, he couplng among hree componens of HSV s much lower, whch generaes a beer decomposon of color space. We consruc a hsogram wh 0 bns, n whch 00 bns for counng on occurrence of hue and sauraon componen, and 0 bns for value componen. In order o effcenly calculae hsograms of deeced paches n curren mage, negral mage s used. Local Bnary Paerns (LBP [] s a non-paramerc kernel whch summarzes he local specal srucure of an mage. Moreover, s nvaran o monoonc gray-scale ransformaons, hence less sensve o changes n llumnaon. The dscree occurrence hsogram of LBP compued over an mage pach s shown o a very powerful eure feaure. We usng 8-b LBP operaor o ge a hsogram wh 56 bns. As he HSV feaure and LBP feaure are ndependen, we sch hese wo vecors ogeher, obanng a hsogram wh 366 bns. Every raecory hypohess a prevous me manans a emplae hsogram o calculae appearance lkelhood of currenly receved observaons. Bhaacharyya coeffcen s used o measure he affny of wo hsograms. The emplaes of derved new raecory hypoheses whch assocaed wh ceran observaon wll be updaed accordng o observaon hsogram. Inegraed wh appearance model, he lkelhood beween currenly receved observaons and prevous raecores can be defned as: ( ( ( ( ( ( a P z k k z k P z k k (4 Because moon lkelhood obaned n Kalman fler s probably densy whle Bhaacharyya coeffcen s he cosne of angle beween wo normalzed vecors, a Logsc funcon should be used o map Bhaacharyya coeffcen o confdence of parcular obec. Anoher coeffcen whch corresponds o he Gaussan dsrbuon coeffcen should be mulpled, so ha he wo pars of lkelhood can complemen each oher. P z ( k ( k a ep a b bha( hz, h where bha( h, h s Bhaacharyya coeffcen beween emplae z hsogram and observaon hsogram; a and b are parameers of Logsc funcon. 4. Repulson-Inera Model Unlke pons rackng, mage paches n close promy may cause ambguous observaons due o occlusons. For eample, observaons of wo obecs may merge or one observaon wll mss, as shown n Fgure. In hese cases, he Bhaacharyya coeffcen wll decrease and fall n seep nerval of Logsc funcon. The hsogram emplae of nvolved new raecory hypoheses should no be updaed. To overcome hese ambgues, raecory predcon s mporan. However, because of low deecon rae and conssen occlusons, raecory predcon usually fals. Eplorng more nformaon from ambguous deecons may help o acheve beer effec. We propose a repulson-nera model o esmae moon sae of hypoheses assocaed wh common ambguous deecon, whch s nspred by [3]. When raecores compee for common observaon and he overlap rae of esmaed obec paches s above a hreshold, s assumed ha here are repulsve forces beween observaons correspondng o esmaed saes, prevenng error mergng. To oban new raecory hypoheses, neracve observaons of observaon z (observaon of obec a me, should be consdered, denoed as J z. A new densy p( z J, z s nroduced o characerze he neracon among observaons. J: J: p( 0: z:, z: k p( z p( 0: p( 0: z:, z: (6 J p( z, z Parcle fler s used for each raecory o ge MAP esmae. The, n, n poseror densy s characerzed by se 0:, s w, where Fgure. Ambguous observaon and assocaon model n, 0:,,..., s n s a se of suppor parcles wh assocaed ˆ z ˆ Valdang Gae n (5
, wegh w n. However, rough esmae usng us fewer parcles s enough. w p( z p(, p( z, z, n, n, n, n J, n, n, n w, n, n J q(, z, z, n, n J where q(, z, z s mporance densy. n, The repulson for parcle s defned as,,, J,, J d n p( z n, n (, n z z z (8 where s a normalzaon erm, s pror consan, and d, n, s he dsance beween observaon of curren parcle and he neracve observaon z. Inera model s consruced o make raecores end o keep orgnal dynamc saes, prevenng obec denes swchng. p(, z p( (,, (9,,,,,,,, n, v ˆ v n, n, n (,, ep ep (7 (0 where s a normalzaon erm, and are pror consans,,, v n n, v ˆ ; n, s he angle beween v and v ˆ. For raecores whch compee for common ambguous observaon currenly, neracve vrual observaons are locaed recursvely accordng o saes esmaed n he las eraon. For each wo of hese raecores, saes are esmaed by akng repulson and nera process eravely for several mes. When all pars of raecores have been calculaed, opmal saes of raecores assocaed wh ambguous observaon are obaned, whch erac more nformaon from orgnal observaon. Ths process s llusraed n Fgure 3. However, f moon or appearance lkelhood of an esmaed raecory hypohess s oo low, wll be deleed from rack rees. 5. Epermenal Seup Mul-obec rackng s a praccal sysemac proec whch akes vdeos as npu, auomacally obans raecores of obecs prelmnarly defned. Our work concenraed on daa assocaon, hence only a publc head-shoulder srucure deecor based on HOG feaure and adaboos classfer s used. The deecon resul s unsasfacory, havng large amoun of mssng observaons n conssen frames and some spurous observaons caused by luggage, whch consrans he performance of rackng algorhm. There are many parameers n proposed algorhm. ( Pror consans of rackng framework, such as, F, P and P ; ( Parameers of Kalman fler, such as process covarance, measuremen covarance and nal sae covarance; (3 Appearance model parameers, such as lkelhood wegh and parameers of Logsc funcon; (4 Parameers of repulson-nera model. These parameers are relevan o he vdeo sense and he behavor and appearance of obecs, rough range of whch can be esmaed by pror knowledge. Then we epermenally se he parameers n several sages, whch reman dencal for all vdeo sequence. Our approach s mplemened n C++ wh OpenCV lbrary. On 5 3.0GHz PC wh 4.00GB RAM, rackng can be eecued n real me (5pfs for abou 0 mage paches whou parallel processng. 5. Trackng Resul and Analyss We adop he mos commonly used CLEAR mercs [3] for evaluaon. CLEAR mercs conss of MOTP whch calculaes he rao of nersecon over unon of boundng boes, rae of msses, rae of false posves, rae of deny swches, and MOTA whch evaluaes he overall suaon of obec denes by consderng msses, false posves and deny swches. Traecores wh ambguous observaon Generae vrual observaons Search parwse observaons Traecores Y Fnsh All parewse? Generae vrual observaons Converge? Fgure 3. Sae esmae by repulson-nera model Repulsonnera lkelhood Esmae saes 5. EXPERIMETS In order o demonsrae he effecveness of our approach, wo survellance vdeo sequences from TRECVI 0, whch are aken n an arpor hall, wh soluon 70 576, conanng frequenly occluson of movng or sac ravelers wh luggage, are used n he epermen. The ground ruhs of hese vdeos are avalable. Y Fgure 4. Trackng Resuls : sample frames of headshoulder deecon (frs row, pons MHT (second row, and paches MHT wh appearance (hrd row. Qualave resuls of pons MHT algorhm and paches MHT negraed wh appearance affny proposed by us are conrasvely shown n Fgure 4, n forms of sample frames. I can be seen ha daa assocaon grealy mproves he effec of unsasfacory deecon. Meanwhle, quanave evaluaon s also aken compared wh ground ruh usng CLEAR mercs, whch s shown n he frs and second rows n Table.
Paches MHT wh appearance performs much beer han pons MHT. In mos cases, hs approach correcly keeps he denes of raecores when nersecons occur, such as raecory, raecory 5 and raecory 8. I also correcly assocaes some mssng observaons, such as raecory 8. However, due o unsasfacory deecon, he performance of daa assocaon s consraned. Table. Performance of head-shoulder srucures rackng Mehod MOTA Mss False Swch MOTP Pons MHT MHT wh appearance MHT wh RI model 0.68 0.756 0.0736 0.0073 0.644 0.770 0.758 0.0639 0.0063 0.603 0.780 0.754 0.069 0.0048 0.63 Ambguous observaons n rackng by deecon framework are of specal concern. Repulson-nera model s negraed no paches MHT algorhm o address deny swches caused by ambguous observaons. Qualave resuls of he proposed mehod and paches MHT whou repulson-nera model are conrasvely shown n Fgure 5. I s obvous ha he proposed mehod grealy reduces deny swches. For eample, raecory n he second row of Fgure 5 s successfully formed, conrasng wh raecores 0 and n he frs row. So s raecory 9 n he second row of Fgure 5, conrasng wh raecores 0 and n he frs row. Fgure 5. Trackng Resuls : sample frames of paches MHT wh appearance (frs row and paches MHT wh repulson-nera model (second row Meanwhle, quanave evaluaon s also aken usng CLEAR mercs, as shown n he hrd rows of Table. The rae of deny swches declnes obvously. However, he rae of msses ncreases slghly due o addonal neracon n ceran comple suaons of unsasfacory deecon. The overall mercs such as MOTA and MOTP slghly ncrease, whch means beer performance of he proposed mehod. 6. COCLUSIO We have proposed an mproved MHT algorhm whch ncorporaes HSV-LBP appearance and repulson-nera model for mage paches rackng. In our approach, appearance affny of mage pach s negraed wh moon affny o calculae hypoheses lkelhood. Ambguous observaons n rackng by deecon framework are of specal concern. Repulson-nera model are used o make raecores compeng for common observaons repel each oher and end o keep orgnal dynamc saes. Epermenal resuls demonsrae he effecveness and prospec of our approach. However, due o unsasfacory deecon, he performance of daa assocaon s consraned. Our fuure work s o couple obec deecon wh daa assocaon for beer performance n survellance vdeo. 7. REFERECES [] Y. Bar-Shalom, F. aum, J. Huang. The probablsc daa assocaon fler. IEEE Conrol Sysems Magazne, Volume: 9 Issue: 6, 8-00, ec. 009. [] Breensen, M.., Rechln, F., e al. Robus rackng-bydeecon usng a deecor confdence parcle fler, In Proceedngs of ICCV 009, pages: 55-5. [3] W. Qu,. Schonfeld, and M. Mohamed, Real me dsrbued mul-obec rackng usng mulple neracve rackers and a magnec-nera poenal model, IEEE Transacons on Mulmeda, 9(3: 5 59, Apr. 007. [4]. Red. An Algorhm for Trackng Mulple Targes. IEEE Trans on auomac conrol, Volume: 4, Issue: 6, Pages: 843 854, 979. [5] I. Co and S. Hngoran. An Effcen Implemenaon of Red s Mulple Hypohess Trackng Algorhm and Is Evaluaon for he Purpose of Vsual Trackng. IEEE Trans on PAMI, 996. [6] Yu and G. Medon. Mulple-Targe Trackng by Spaoemporal Mone Carlo Markov Chan aa Assocaon. IEEE Trans on PAMI, Vol.3, o., 009. [7] W. Brendel, M. Amer and S. Todorovc. Mulobec Trackng as Mamum Wegh Independen Se. In Proceedngs of CVPR, 0. [8] H. Shr, J. Berclaz, F. Fleure e al. Trackng Mulple People under Global Appearance Consrans. In Proceedngs of ICCV, 0. [9] L. Zhang, Y. L and R. evaa. Global daa assocaon for mul-obec rackng usng nework flows. In Proceedngs of CVPR, 008. [0] H. Prsavash,. Ramanan and C. Fowlkes. Globally- Opmal Greedy Algorhms for Trackng a Varable umber of Obecs. In Proceedngs of CVPR, 0. [] Mller, M.L., Sone, H.S. and Co, I.J. Opmzng Mury's ranked assgnmen mehod, IEEE Transacons on Aerospace and Elecronc Sysems, Volume: 33, Issue: 3, 997. [] Oala, T., Pekanen, M. and Maenpaa, T., Mulresoluon gray-scale and roaon nvaran eure classfcaon wh local bnary paerns, IEEE Transacons on Paern Analyss and Machne Inellgence, Volume: 4, Issue: 7, 00. [3] K. Bernardn and R. Sefelhagen. Evaluang Mulple Obec Trackng Performance: he Clear Mo Mercs. EURASIP Journal on Image and Vdeo Processng, 008.