CamShf Guded Parcle Fler for Vsual Trackng Zhaowen Wang, Xaokang Yang, Y Xu and Songyu Yu Insue of Image Communcaon and Informaon Processng Shangha Jao Tong Unversy, Shangha, PRC 200240 E-mal: {whereaswll,xkyang, xuy, syyu}@sju.edu.cn Absrac Parcle fler and mean shf are wo mporan mehods for rackng objec n vdeo sequence, and hey are exensvely suded by researchers. As her srengh complemens each oher, some effor has been naed n [1] o combne hese wo algorhms, on whch he advanage of compuaonal effcency s focused. In hs paper, we exend hs dea by explorng even more nrnsc relaonshp beween mean shf and parcle fler, and propose a new algorhm, CamShf guded parcle fler (CAMSGPF). In CAMSGPF, wo basc algorhms - CamShf and parcle fler - can work cooperavely and benef from each oher, so ha he overall performance s mproved and some redundancy n algorhms can be removed. Expermenal resuls show ha he proposed mehod can rack objecs robusly n complex envronmen, and s much faser han he exsng mehods. I. INTRODUCTION Trackng vsual objecs hrough mage frames has been a fundamenal opc n compuer vson feld and s wdely appled o survellance, robocs, human machne nerface, objec based vdeo codng, ec. However, he ask of robus rackng s challengng regardng fas moon, occluson, srucural deformaon, llumnaon varaon, background cluers, real-me resrcon, ec. To handle hese problems, many effors have been pad o devse good rackng algorhms. One promsng caegory s sequenal Mone Carlo mehods, also known as parcle flers, whch esmae he mos lkely poseror wh dscree samplewegh pars n a Bayesan Nework frame. The basc dea s nroduced by Hammersley e al n [2], and s mplemened no varous mproved versons over he las decade. Due o parcle flers non-gaussan non-lnear assumpon and mulple hypohess propery, hey are successfully appled o vsual rackng, e.g. n [3], [4], and show exra mers n cluered envronmen. However, he neffcency n samplng and he huge compuaonal complexy lm he usefulness of parcle fler n on-lne rackng. Anoher popular rackng mehod s mean shf procedure, whch fnds he local maxmum of probably dsrbuon n he drecon of graden. Comancu e al. propose a mean shf based rackng mehod n [5]. Bradsk [6] exends o CamShf by adapvely changng he scale of search wndow. As a deermnsc mehod, mean shf keeps sngle hypohess and hus s compuaonally effcen. Bu may run no rouble when smlar objecs are presened n background or when occluson happens. Based on he pros and cons of parcle fler and mean shf, Shan e al. [1] proposed a new algorhm, he Mean Shf Embedded Parcle Fler (MSEPF), o negrae he advanages of he wo mehods for objec rackng. In MSEPF, mean shf s performed on each of he parcles afer hey are propagaed, so ha he parcles are herded o nearby local modes wh large weghs. In hs way, he poseror can be beer esmaed even wh a smaller sample se, and he compuaon complexy of parcle fler s reduced proporonally. In MSEPF, mean shf s jus used as a subsdary ool o draw beer samples for parcle fler. In fac, we fnd parcle fler can also play a role n asssng he mean shf procedure. Therefore, n hs paper, we exend he dea of MSEPF and propose a novel algorhm, CamShf Guded Parcle Fler (CAMSGPF). In CAMSGPF, CamShf faclaes parcle fler n drawng a good sample se as s n MSEPF; and n reurn, parcle fler helps o mprove he scale esmaon and smplfy he complexy of CamShf. In hs way, CamShf and parcle fler can work ogeher coherenly. The proposed CAMSGPF algorhm s effcen and robus, and surpasses oher parcle flers and mean shf based rackers n general. The remander of he paper s organzed as follows. We presen n deal how CAMSGPF negraes CamShf no parcle fler n Secon II, and hen show expermenal resuls n Secon III, and fnally conclude he paper n Secon IV. II. CAMSHIFT SHIFT GUIDED PARTICLE FILTER The man frame of CAMSGPF proposed n hs paper s based on he well-known Sequenal Imporance Resamplng (SIR) [7] parcle fler, wh CamShf negraed. In he algorhm, he sae probably dsrbuon of he arge s esmaed va a fne se of N samples (parcles) wh sae { { } x }. Gven he sample se a he prevous me, x 1, he CAMSGPF sars by propagang each sample wh a sochasc dsplacemen accordng o dynamc model. The resulng samples { x } are furher shfed by a} modfed verson of CamShf, so ha he new sample se, { x, wll be more close o dsrbuon modes. Then we assgn wegh { } w o each sample by evaluang her lkelhood agans observaon z a he new sae and akng no accoun he basng effec of CamShf prevously appled. Fnally, a resample sep mulples samples wh large wegh and generaes an unweghed sample se { } x, whch corresponds o he arge dsrbuon a curren me. In hs way, CAMSGPF can recursvely fnd he laes arge sae. The flowchar of CAMSGPF s llusraed n Fg. 1. In CAMSGPF, CamShf s adoped for acve scale adapaon. And more mporanly, he cooperave relaonshp
=1,, N=8 parcles { x 1, w 1} Propagang ~, w { x 1 } Smplfed CamShf for par of he parcles { x, w 1} Wegh evaluaon & updae { x, w } Resample { x, w } Fg. 1. The framework of CAMSGPF. Each blob s poson and area represen he sae and wegh of a sample. The hghlghed color sgnfes weghs evaluaed drecly from observaon model. beween parcle fler and CamShf s fully exploed. On one hand, he parcles are concenraed by CamShf o hgher probably nearby modes, n erms of boh he poson and scale of he arge. On he oher hand, CamShf can acheve a beer scale adapaon han beng used alone wh parcle fler s mul-hypohess naure; and CamShf can be appled n a much smplfed way o furher boos he algorhm effcency due o he 2nd AR dynamc model predcon. In he followng paragraphs, we shall explan n deal how hese pons are aaned n CAMSGPF. A. Sysem Models In our sysem, he arge o be racked n vdeo sequence s modeled by a sae vecor: x = (x c, y c, w, h) T (1) so ha a recangle box cenered a coordnae (x c, y c ) wh wdh w and hegh h jus covers he arge area. The dynamc of sae ranson corresponds o a sandard 2nd order auoregressve process: x = x 1 + (x 1 x 2 ) + n (2) Tha s, he curren sae x s predced as he sum of hree erms: he prevous sae a me 1, he dsplacemen of las ranson, and a gaussan nose, n. Ths smple model can well smulae mos ordnary objec moons, and plays a val role n smplfyng he overall algorhm as can be seen n laer dscusson. Followng [4], we use HSV color hsogram o buld he observaon model. Gven he curren observaon z (.e. he curren mage frame), he canddae color hsogram q(x ) s calculaed on z n he regon specfed by x. Then s compared wh he reference color hsogram q by Bhaacharyya smlary merc D[, ], resulng o he lkelhood dsrbuon: p(z x ) e λd2 [q,q(x )] where λ s se o 20 for mos applcaons. B. Effcen Poson Shf In CAMSGPF, he samples { x } drawn accordng o sae dynamc Eq. (2) are frs shfed n her poson subspace by mean shf vecor [5] for a few eraons. Le s denoe he curren poson of a sample x as p [x c, y c ] T, hen s new poson afer one eraon wll be: M p =1 = a w(a )g( p a 2 h ) M =1 w(a )g( p a 2 (4) h ) where {a } =1...M are pxel coordnaes whn he recangle area specfed by sae x, w(a ) s he wegh ndcang he rao of hsogram bn values correspondng o a n he curren and reference color hsogram. g() s a kernel profle funcon, and h s wndow radus o normalze he coordnae a. In MSEPF, he mean shf eraon n Eq. (4) s appled on every sample n sample se, and goes on unl convergence or maxmum number of eraons s reached. I s clamed n [1] hs wll grealy reduce he me consumpon on parcle fler, because fewer parcles are requred o accomplsh rackng. Unforunaely, hs mprovemen n effcency wll be parly canceled ou by he nroducon of mean shf, whch wll ake exra me for he whole algorhm. Wha s worse, a radonal mean shf procedure [5] s more complcaed han a generc parcle fler recurson cycle, so each parcle n MSEPF wll spend mos of s me n he mean shf sep. To furher mprove he algorhm effcency, we mus smplfy he mean shf procedure n CAMSGPF. Ths s approached n he followng hree ways: 1) Mean shf s appled only on some parcles randomly seleced. 2) Only a small and fxed number of eraons are carred ou on each sample. 3) Unlke he orgnal mean shf procedure, whch checks he correcness of mean shf vecor n each eraon, we om a all n CAMSGPF. Because he oal me consumed on mean shf s proporonal o he number of parcles subjeced o and he number of eraons each ndvdual mean shf undergoes, our smplfcaon wll boos he algorhm speed subsanally. Moreover, can be found ha, n he conex of parcle flerng, he smplfcaons above wll have lle nfluence on he performance of mean shf. The reason les n he dsplacemen erm of Eq. (2). Once a parcle s shfed, he (3)
effec of mean shf wll be bul no he dsplacemen erm and carred o he fuure ransons. Then even f a parcle s no shfed a every me sep, or s no shfed horoughly o he maxmum nearby mode n a resrced number of eraons, he parcle wll sll move n he drecon o he maxmum nerally as long as he dsrbuon does no change dramacally. Furhermore, wh he resamplng sep of parcle fler, here s no need o worry abou he correcness of mean shf n each eraon. In CAMSGPF, badly shfed parcles wll be removed by resamplng, and parcles shfed owards local modes wll be resampled more han once. In hs way, parcle fler performs he evaluaon on mean shf eraons and furher smplfes he mean shf procedure. Now we can safely apply he smplfed mean shf on par of he samples a each me sep. To keep every parcle beng mean-shfed evenly, once for a whle, we sor he samples accordng o her weghs and deermne wheher o apply mean shf or no by her modular-ed ndex: { MeanShf (x, I) f %N s < n 1 (5) No Mean Shf oherwse where s he ndex of he sored parcles, N s s a fracon of parcle number N, and n 1 s an neger usually aken o be half of N s. We have used MeanShf (, ) o denoe he smplfed mean shf, whch akes argumen x as he nal search poson, and I as he number of eraons. As parcles weghs are changed dynamcally, hs operaon wll guaranee all he samples a far chance o ge mean-shfed. C. Adapve Scale Adjusmen Afer he samples n { x } are mean-shfed n poson subspace, hey are furher refned n scale subspace usng CamShf algorhm n CAMSGPF. As noed n [6], CamShf exends he mean shf algorhm so ha he sze of he searchng wndow can be adjused o f he changng scale of he arge. The calculaon s based on pxel s lkelhood wegh w(a ), whch s he byproduc n evaluang mean shf vecor accordng o Eq. (4). Snce a denser pxel wegh dsrbuon (larger zeroh momen of he lkelhood dsrbuon mage) mples a larger arge sze, he wndow sze s esmaed emprcally n [6] as a funcon of he zeroh momen: M00 l = k (6) 256 where l s he wdh or hegh of he wndow, k s a consan. And he zeroh momen M 00 of he correspondng recangle wndow area of he curren sae x s calculaed by M 00 = M =1 w(a ) (7) where {a } =1...M and w(a ) are defned as hey are n Eq. (4). The momen should be frs normalzed by he maxmum value of probably dsrbuon (256 n he 8-b case). Snce he relaonshp beween zeroh momen and arge sze n Eq. (6) s found emprcally, s praccal only n some specfc cases. For example, wh parameer seng n [6], (a) Fg. 2. Wh he momen-sze relaonshp of Eq. (6), CamShf succeeds n rackng a man s face (a); whle he scale adapaon fals when rackng a red car n feld (b). CamShf can rack human face successfully, bu may fal o adjus he wndow sze properly n more general applcaons (Fg. 2). In CAMSGPF, we apply CamShf wh dfferen momensze-esmaon funcons on dfferen parcles. Some of hese funcons end o scale he parcles sze up or down, whle ohers do no change he sze a all (n hs case, CamShf degrades o mean shf). Afer dong some manpulaons on Eq. (6), we can oban wo scale facors used n our mehod, s u for scalng up and s d for scalng down: M00 s u = k 1 (8a) 256 w h M00 s d = k 2 (8b) 256 w h where w and h are he wdh and hegh componen of he curren sample sae x; k 1 and k 2 are consans sasfyng k 1 > k 2. The scale subspace of he sae x s hus updaed o be: w s w (9a) h s h (b) (9b) where s s he scalng facor. I s randomly seleced from s u, s d and 1 for each parcle, so ha dfferen scales are red on dfferen parcles. Then afer hese parcles are evaluaed usng observaon model, he beer scaled ones can be pcked ou, whle some nappropraely scaled ones wll be elmnaed by resamplng. As w and h can boh ncrease and decrease n he scale subspace, hey wll evenually sele on he exac sze of he arge. Wh hs mul-hypohess and es paradgm, we can adjus he sze o he bes whou knowng he precse momen-sze-esmaon funcon. To ensure ha all he parcles wll have equal opporuny o ry on dfferen choces of scales, we employ he echnque smlar o Eq. (5). The scalng facor s s seleced for a sored parcle x as: s = s u f %N s = n 2 s d f %N s = n 3 1 oherwse (10) where parameers n 2 and n 3 can be any negers less han n 1 n Eq. (5), as parcles o be scaled mus be mean-shfed a
frs hand. And n our mplemenaon hey are aken as 1 and 2 for smplcy. The physcal meanng of N s s also clear from Eq. (10): s defned such ha each one ou of N s parcles wll be scaled up and down. To combne Eq. (5) and Eq. (10) n a more compac form, le CamShf (,, ) (o be elaboraed n Table II laer) be he concaenaon of he smplfed mean shf MeanShf (, ) and a subsequen CamShf scalng, and he selecve meanshfng and scalng mechansm can be rewren n he form: No CamShf f %N s N s /2 CamShf (x, I, +1) f %N s = 1 CamShf (x, I, 1) f %N s = 2 CamShf (x, I, 0) oherwse (11) where he 3rd argumen n CamShf (,, ) sgnals he choce of scalng facor. So far, we can fnd ha n CAMSGPF, he funcon of CamShf s o gve he parcles some crude mplcaons on he drecon o propagae (for boh poson and scale), whle he judgmen and feedback s lef o parcle fler. I s because of hs neracve relaonshp ha we name hs algorhm CamShf guded parcle fler. D. Wegh Evaluaon Havng CamShf-ed samples from { x } } { x o, we are ready o updae her weghs { } w a he new saes. The new wegh of each sample s found as follows: w w 1 p(z x )p( x x 0: 1) q( x x 0: 1, z (12) ) where q( x x 0: 1, z ) s he proposal dsrbuon, p(z x ) s gven by Eq. (3), and pror dsrbuon p( x x 0: 1) can be derved from sae dynamc Eq. (2) o be: p( x x 0: 1 ) = N(2x 1 x 2, σ) (13) where N(, ) denoes Gaussan dsrbuon, σ s he covarance. The dffculy o evaluae Eq. (12) les n fndng he proposal q(). Before CamShf s embedded no parcle fler, q() jus akes he same form as he pror Eq. (13). However, n CAMSGPF, he effec of CamShf should also be aken no accoun; oherwse, he poseror esmaed by { x, } w would be based. We can vew CamShf as a subsequen sample drawng sep condoned on he samples { x }, whch s drawn beforehand accordng o he pror dsrbuon. Then he new proposal dsrbuon becomes: q( x x 0: 1, z ) = p( x x, z )p( x x 0: 1)d x (14) where p( x x, z ) s he probably ha sae x wll be CamShf-ed o x gven he observaon z. As CamShf s deermnsc, he value of p() s eher 1 or 0, and can be deermned explcly by checkng wheher he resul of CamShf-ng x by Eq. (11) wll collde wh x. However, dong CamShf on all possble x s me-exhausng, n pracce we use a Gaussan model smlar o [8] o approxmae p(): p( x x, z ) N( x, Σ) (15) where Σ s a dagonal covarance marx. As CamShf wll no shf a sample oo far away from s nal sae, he approxmaon n Eq. (15) can counerbalance CamShf s basng on poseror o some exen. E. Summary of CAMSGPF For clary, we brefly encapsulae he overall CAMSGPF algorhms n Table I, whch mahemacally descrbes he flowchar n Fg. 1. TABLE I ALGORITHM OF CAMSHIFT GUIDED PARTICLE FILTER x, w = CAMSGPF( x 1, w 1 ) for ( = 1 : N) Propagae parcle x 1 by Eq. (2) o ge x Selecvely CamShf x o x accordng o Eq. (11) Evaluae he observaon lkelhood p(z x ) by Eq. (3) Updae wegh w by Eq. (12) endfor Sor x, w accordng o wegh w Resample x, w, producng un-weghed sample se x,1/n And he modfed CamShf (,, ) algorhm n Eq. (11) s shown n Table II. x = CamShf ( x 0, I, ǫ) TABLE II ALGORITHM OF THE MODIFIED CAMSHIFT Se he curren search wndow o he recangle represened by x j 1 Evaluae all he pxel wegh w(a ) nsde he wndow Shf he poson componen (x c, y c) T of x j 1 by Eq. (4) : f (ǫ = +1) Fnd scalng facor s by Eq. (8a) elsef (ǫ = 1) Fnd scalng facor s by Eq. (8b) else se s = 1 Updae he scale componen w and h of x j 1 usng Eq. (9a), (9b) f (j = I) sop, se x = x j else j++, and reurn o he frs sep III. EXPERIMENTAL RESULTS In hs secon, he performance of CAMSGPF s compared wh oher rackers n a number of aspecs. In he expermens, he parameers of CAMSGPF are se as follows: k 1 = 1.2; k 2 = 1; N s = 10; I = 2. All he ess are carred ou on 320 240-pxel sequences wh a Penum IV 2.8G PC. We compare he general rackng ably of dfferen algorhms by examnng he mnmal number of parcles requred by each of hem o barely acheve successful rackng. From he expermens on hockey sequence (shown n Fg. 3, rackng resuls by CAMSGPF only), we observe ha hs number s 35 a leas for generc parcle fler, whle 10 for boh of MSEPF and CAMSGPF. I s clear ha mean shf or
Fg. 3. hockey sequence successfully racked by CAMSGPF wh 10 parcles (frame 56, 211, 429) me (ms) me (ms) 36 34 32 30 28 26 24 22 20 18 16 21 20 19 18 17 16 15 14 CAMSGPF MSEPF 10 20 30 40 50 60 70 80 90 100 parcles (a) CAMSGPF MSEPF 13 10 20 30 40 50 60 70 80 90 100 parcles (c) me (ms) 18 17 16 15 14 13 12 CAMSGPF MSEPF 11 10 20 30 40 50 60 70 80 90 100 parcles Fg. 5. Comparson of me consumpon versus parcle number beween CAMSGPF and MSEPF. (a) redeam ; (b) hockey ; (c) eges01 ; (d) rackng samples correspondng o (a)-(c) (b) (d) Fg. 4. soccer sequence racked by MSEPF, CamShf and CAMSGPF (n he columns from lef o rgh). (frame 114, 123, 176, 193) CamShf can help he parcle fler a lo, reducng he number of parcles by 71%. The ably of scale adapaon s esed n he soccer sequence, as shown n Fg. 4. The rackng resul of CAMS- GPF s dsplayed n he hrd column, n comparson wh hose racked by MSEPF and CamShf (wh momen-szeesmaon funcons uned o hs sequence), n he frs and second column. Our mehod urns ou o adap o scale change of he arge bes, snce he boundary box bes maches he arge sze. To valdae he sgnfcance of applyng he smplfed CamShf on par of he parcles, we compare he me consumpon of CAMSGPF wh MSEPF on several sequences. Boh of he algorhms can rack he arges correcly mos of he me, as Fg. 5(d) demonsraes. The average me consumed per frame by he wo s ploed n Fg. 5 (a)-(c) correspondngly, wh regard o dfferen number of parcles used. For all ess, he me consumpon grows lnearly wh he number of parcles, and CAMSGPF s conssenly faser han MSEPF. The curves n Fg. 5 do no pass hrough he orgn, due o some fxed asks n each frame (vdeo fle readng, color space converson and dsplayng). So we deduc he fxed me for boh of he wo algorhms and compare he average me consumpon per parcle for each sequence,.e., he slopes of he lnes n Fg. 5, n he lef columns of Table III. We can see akes abou 20%~50% less me for CAMSGPF o process a parcle han MSEFP. In he rgh columns of Table III, he average Bhaacharyya dsance beween he racked area and he arge emplae s shown as an ndcaor of he rackng accuracy. All he 3 ess confrm ha he nfluence due o smplfcaon n CAMSGPF on he rackng accuracy s rval wh respec o he me saved. IV. CONCLUSIONS A novel rackng algorhm, CAMSGPF, has been proposed by explorng he neracon beween parcle flerng and CamShf. Wh he ads of CamShf, he parcles are guded o more possble modes of observaon, so ha he samplng effcency s mproved grealy. A he same me, CamShf s conduced under he supervson of parcle flerng. In hs way, he scale adapaon of CamShf becomes funconal n more unversal suaons; and CamShf can be appled on parcles n a more economc way whou much sacrfce n performance. The expermenal resuls demonsrae ha CAMSGPF ouperforms CamShf and MSEPF n boh rackng robusness and effcency. ACKNOWLEDGMENT Ths work was suppored by Naonal Naural Scence Foundaon of Chna under Gran No.60502034, Shangha
TABLE III COMPARISON OF EFFICIENCY AND ACCURACY BETWEEN MSEPF AND CAMSGPF Sequence Tme/Parcle(ms) Average dsance Tme Saved MSEPF CAMSGPF MSEPF CAMSGPF Dsance dfference Redeam 0.1807 0.1380 23.63% 0.1150 0.1140-0.87% Hockey 0.0618 0.0432 30.03% 0.0903 0.0950 5.13% Eges01 0.0691 0.0377 45.49% 0.1213 0.1311 8.14% a smaller dsance ndcaes a beer mach beween wo objecs. Rsng-Sar Program under Gran No. 05QMX1435, H-Tech Research and Developmen Program of Chna (863) under Gran No. 2006AA01Z124, and Shangha Posdocoral Foundaon under Gran No. 06R214138. REFERENCES [1] C. Shan, Y. We, T. Tan and F. Ojardas, Real me hand rackng by combnng parcle flerng and mean shf, Sxh IEEE Inernaonal Conference on Auomac Face and Gesure Recognon 2004. [2] J.M. Hammersley and K.M. Moron, Poor man s Mone Carlo, 1954. Journal of he Royal Sascal Socey B, 16, 23-38 [3] M. Isard and A. Blake, Condensaon-condonal densy propagaon for vsual rackng, Inernaonal Journal on Compuer Vson, 29(1), pp. 5-28, 1998. [4] P. Perez, C. Hue, J. Vermaak and M. Gangne, Color-Based Probablsc Trackng, ECCV, 2002, pp. 661-675. [5] D. Comancu, V. Ramesh and P. Meer, Real-me rackng of non-rgd objecs usng mean shf, IEEE Conference on Compuer Vson and Paern Recognon, pages II: 142-149, Hlon Head, SC, June 2000. [6] G.R. Bradsk, Compuer vson face rackng as a componen of a percepual user nerface, he Workshop on Applcaons of Compuer Vson, pages 214-219, Prnceon, NJ, Oc. 1998. [7] N. Gordon, D. Salmond, and A. F. M. Smh, Novel approach o nonlnear and non-gaussan Bayesan sae esmaon, Proceedng of IEE, F, vol. 140, pp. 107-113, 1993. [8] Y. Ca, N. de Freas, J. Lle, Robus Vsual Trackng for Mulple Targes, ECCV, 2006, vol. 4, pp. 107-118