Multiple Target Tracking Based on Undirected Hierarchical Relation Hypergraph

Transcription

1 0 IEEE Conference on Copuer Vision and Paern Recogniion Muliple Targe Tracking Based on Undireced Hierarchical Relaion Hypergraph Longyin Wen Wenbo Li Junjie Yan Zhen Lei Dong Yi San Z. Li Cener for Bioerics and Securiy Research & Naional Laboraory of Paern Recogniion Insiue of Auoaion, Chinese Acadey of Sciences, China Absrac Dense Neighborhoods Muli-arge racking is an ineresing bu challenging ask in copuer vision field. Mos previous daa associaion based ehods erely consider he relaionships (e.g. appearance and oion paern siilariies) beween deecions in local liied eporal doain, leading o heir difficulies in handling long-er occlusion and disinguishing he spaially close arges wih siilar appearance in crowded scenes. In his paper, a novel daa associaion approach based on undireced hierarchical relaion hypergraph is proposed, which forulaes he racking ask as a hierarchical dense neighborhoods searching proble on he dynaically consruced undireced affiniy graph. The relaionships beween differen deecions across he spaioeporal doain are considered in a high-order way, which akes he racker robus o he spaially close arges wih siilar appearance. Meanwhile, he hierarchical design of he opiizaion process fuels our racker o long-er occlusion wih ore robusness. Exensive experiens on various challenging daases (i.e. PETS009 daase, ParkingLo), including boh low and high densiy sequences, deonsrae ha he proposed ehod perfors favorably agains he sae-of-he-ar ehods.. Inroducion Muli-arge racking is an ineresing bu difficul proble. Alhough nuerous racking ehods have been proposed in lieraures, heir perforance are unsaisfacory in pracical applicaions. As shown in Fig. (a), os of he previous ehods focus on he pairwise relaionships (e.g. appearance and oion paern siilariies) of he rackles in he local liied eporal doain, raher han aong uliple rackles across he whole video eporal doain in a global view. When he arges walk closely wih siilar appearance or oion paerns, as denoed by he circles Corresponding auhor Figure. (a) describes he previous ehods fail o handle he challenge ha he arges walk closely wih siilar appearance or oion paerns, and (b) describes our undireced hierarchical relaion hypergraph based racker successfully handle his challenge. The circles represen differen rackles and heir colors represen he inheren paerns (e.g. appearance and oion paerns). Siilar colors represen siilar paerns of he rackles. Previous ehods, which focus on he pairwise siilariies of spaial-eporal neighboring rackles in local liied eporal doain, generae wrong rajecories (blue splines). On he conrary, he proposed ehod searching he dense neighborhoods in he rackle relaion graph/hypergraph, which considers he siilariies aong uliple rackles across he eporal doain, generae correc rajecories (red splines). and in Fig., he ideniy swiches will follow he previous rackers [,, 6,, 8, 6,, 9, 0, 9]. To alleviae he issues, ehod based on iniu clique graphs opiizaion has been developed [5], which considers he relaionships beween differen deecions across he eporal doain. However, i is hard o handle he non-linear oion of he arges in crowded scenes, especially when he occlusion happens, ainly due o he unreliable hypoheical nodes generaion in opiizaion process. In his paper, an undireced Hierarchical relaion Hypergraph based Tracker (H T) is proposed, which forulaes he racking ask as searching uliple dense neighborhoods on he consruced undireced relaion affiniy graph/hypergraph, as described in Fig. (b). Differen fro he previous ehods, we consider he relaionships beween differen deecions across he eporal doain globally. Meanwhile, a local-o-global sraegy is exploied / $.00 0 IEEE DOI 0.09/CVPR

2 Figure. The overlooking of he rackles in opiizaion process of he sequence PETS009-SL. The rackles in four segens of he second layer are cobined o generae he arge rackles in he hird layer. Two exaples of he opiized rackles are presened as he red and purple pipelines in he figure. o cluser he deecions hierarchically, which grealy reduces he copuaional coplexiy in dense neighborhoods searching, while handles he sudden variaions of he arge s appearance and oion effecively. Firsly, he racking video is divided ino a few segens in he eporal doain and he dense neighborhoods are searched in each segen o consruc uliple longer rackles. Then he nearby segens are erged o consruc he new segen division for he nex layer. These wo seps are ieraed unil only one segen exiss in he layer, i.e. he whole video span, and he dense neighborhoods searching is carried ou in ha segen o obain he final arge rajecories. The ain conribuions of his paper are suarized as follows. () The uli-arge racking is firs odeled as a dense neighborhoods searching proble on he hierarchically consruced rackle undireced affiniy graph/hypergraph. () The oion properies and appearance inforaion of he arges are fully exploied in he opiizaion process by considering he high-order relaionships beween uliple rackles in he consruced hypergraph. () Tracking experiens on various publicly available challenging sequences deonsrae he proposed ehod achieves ipressive perforance, especially when he long-er occlusion and siilar appearance challenges happen in he crowded scene.. Relaed Work Recenly, racking-by-deecion ehods [,,, 6,,, 8, 6,,, 5, 9, 0, 9,, ] becoe popular, which associae uliple inpu deecion responses in differen fraes o generae he rajecory of arges. Soe researchers forulae he associaion ask as a aching proble, which ach he deecions wih siilar appearance and oion paerns in consecuive fraes, e.g. bi-parie aching [] and uliple fraes aching [9]. Unforunaely, he liied-eporal-localiy degrades heir perforance when long-er occlusion, coplex oion, or spaially close arges wih siilar appearance challenges happen. Oher researchers consruc a k-parie graph o describe he relaionships beween differen deecions and use soe opiizaion ehods o coplee he associaion ask (e.g. Nework flow [6, 8], K-Shores Pah (KSP) [], Maxiu Weigh Independen Se [6], Linear Prograing []). These ehods, generally ered global, ake an effor o reduce or reove he liied-eporal-localiy assupion in opiizaion, which can overcoe he firs wo aforeenioned challenges o soe degree. However, hey also only consider he relaionships of deecions in local consecuive fraes, so ha hey have difficulies in discriinaing he spaially close arges wih siilar appearance. Recenly, in [], a coninuous energy iniizaion based ehod is proposed, in which he local iniizaion of a nonconvex energy funcion is well found by he sandard conjugae gradien opiizaion. The subsequen work [] designs a discree-coninuous opiizaion fraework, which decoposes he racking ask as wo ieraively opiizaion seps, i.e. one is he daa associaion of he rackles and he oher one is he rajecories fiing. In addiion, soe ehods odel he associaion ask as a linking proble and use he hierarchical Hungarian algorih [9,, ] o coplee he racking ask. Anoher work relaed o his paper is [8], which siulaneously handles he uli-view reconsrucion and uliarge racking asks in he D world coordinaes. In [8], a direced graph is consruced o describe he associaion relaionships beween candidae couplings, which are consiued by he deecions appeared in differen caera-view a he sae ie. The edges in he graph describe he associaion probabiliy beween he eporal consecuive pairwise couplings. Differen fro [8], we consruc an undireced hypergraph, in which he nodes represen he rackles, and he hyperedges are consiued by uliple rackles across he eporal doain. Meanwhile, since he hypergraph consrucion and ask objecive are differen, he opiizaion sraegies in hese wo ehods are oally differen.. Hierarchical Relaion HyperGraph based Tracker Given he frae-by-frae deecions, he racking ask is odeled as a hierarchical dense neighborhoods searching proble on he rackle affiniy relaion graph/hypergraph, which is consruced dynaically o describe he relaionships beween he rackles generaed in he previous layer. 77 8

3 Noably, a deecion is regarded as a degenerae rackle, conaining only one deecion response. The racking video is firsly divided ino u segens in he eporal doain. Le Δ l r represen he ie inerval of he r-h segen in layer l and T be he oal frae lengh of he video. The ie inerval se of hese u segens is represened as {Δ l,, Δ l u} such ha Δ l i Δ l j =0and u i= Δ = T. A relaion affiniy graph is consruced in each segen a he curren layer. Le Gr l =(Vr l, Er) l represen he consruced graph in he r-h segen of l-h layer, where Vr l and Er l are he node and edge se of he graph respecively. The graph node se is furher represened as Vr l = {vi,r l }n i=, where v i,r s he i-h node and n is he nuber of nodes in he graph. Each node in Gr l corresponds o a rackle in he curren segen. The corresponding rackle se is defined as T l r = {Ti,r l }n i=. The edges of he graph describe he relaionships beween differen rackles and heir weigh indicae he probabiliy of he rackles belonging o he sae arge. Wihou abiguiy, we use vi,r l and Ti,r nerchangeably o represen he i-h rackle in he r-h segen a layer n he res of his paper. Differen fro previous works where only he pairwise relaionships are considered, in his ehod, he high-order relaionships aong uliple rackles are inegraed in he relaion graph for he firs a few layers, i.e. he edges in he graph involve ore han jus wo verices. In his way, he oion and rajecory soohness consrains of he arge can be fully used o ensure he racking perforance. We exend he dense neighborhoods searching algorih [] o handle he dense neighborhoods revealing proble in boh graph and hypergraph, and generae he longer rackles by siching he rackles in each revealed dense neighborhoods. Noably, he graphs/hypergraphs in all segens are processed in he sae fashion. Afer copleing he dense neighborhoods searching proble in all segens of curren layer, he nearby segens are erged o generae a new segen division Δ l+ for he nex layer. Then, we also consruc he graph/hyerpgrah in each segen of Δ l+ and reveals he dense neighborhoods on each graph/hypergraph o generae he longer rackles. These wo seps are ieraed unil only one segen reains in he layer. Finally, he dense neighborhoods searching is perfored again on he consruced relaion affiniy graph of ha segen o obain he final arge rajecories. As an exaple, he opiizaion process fro layer o of PETS009-SL is presened in Fig.. Wihou abiguiy, we oi he segen index r and he layer index s he following secions.. Undireced Affiniy Graph Consrucion For he curren segen, we consruc a global relaion affiniy graph G = (V, E), which is a coplee graph describing he relaionships beween differen rackles. V = {v,, v n } is he graph node se. E is he {}}{ graph edge/hyperedge se, i.e. E V V, where is he nuber of verices involving in each edge/hyperedge. We use he bold sybol e =(v,, v ) o represen he -uple verices involving in he edges/hyperedges of he graph. Obviously, he consruced graph is a hypergraph when > and degenerae o a general graph when =. The affiniy array A of he graph G is a {}}{ n n super-syery array, in which he eleens reflec he probabiliy of he rackles in e belong o he sae arge. The affiniy value in he array A(e )=0,if e / E; oherwise A(e ) 0. The design of calculaing he affiniy values in he undireced relaion affiniy graph plays a cenral role in H T, which indicaes how probably he rackles belong o he sae arge. We calculae he affiniy value A(e ) of he edge/hyperedge e by hree facors: appearance, oion and rajecory soohness. The appearance facor indicaes he appearance siilariy beween he rackles in e ; he oion facor indicaes he oion consisence beween he rackles in e ; and he soohness facor indicaes he physical soohness of he erged rackles consiued by he rackles in e. Therefore, he edge affiniy value is calculaed as: A(e )=ω A a (e )+ω A (e )+ω A s (e ), () where A a ( ), A ( ) and A s ( ) are he appearance, oion and soohness affiniy, respecively. ω, ω and ω are he prese balance paraeers of hese hree facors. Obviously, if he appearing eporal doain of he rackles v i and v j in e overlap each oher, hey should no belong o he sae arge, so we se A(e )=0. Thus, we jus need o consider he case ha neiher wo rackles in e overlap each oher of eporal doain, when we calculae he appearance, oion and soohness affiniies... Appearance Affiniy The appearance of an objec is a crucial par o disabiguae i fro he background and oher objecs. For our H T, wo kinds of feaures in he firs and las fraes of he rackle are adoped o describe is appearance, i.e. color hisogra feaure and shape gradien hisogra feaure. We use 8 bins for each channel of RGB color hisogra and 6 diensions for shape gradien hisogra. Wihou loss of generaliy, he rackle v i is assued o appear before v j in eporal doain. Then he appearance affiniy is calculaed as follows: A a (e )= λ k H( f k (v i ), ˆf k (v j )), () v i,v j e k=, 78 8

4 where f (v i ) and f (v i ) are he color and shape hisogras of v i s las frae, ˆf (v i ) and ˆf (v i ) are he color and shape hisogras of v i s firs frae, H(, ) represens he cosine disance beween wo feaure vecors and λ, λ are he prese balance paraeers... Moion Affiniy A defining propery of racking is ha he arges ove slowly relaive o he frae rae, leading o a fac ha he velociy of a arge can be se as a consan in a sall eporal doain, which is powerful enough o consrain he rajecories of he arges. The oion affiniy reflecs he oion consisency of he rackles in e, which is defined as: A (e )= S (v i, v j ), () v i,v j e where S (v i, v j ) is he oion siilariy beween he rackles v i and v j. Specifically, here is only one deecion response in each rackle of he firs layer, leading o no oion inforaion conained in each rackle. So we se he oion affiniy value A (e )=0a he firs layer. We assue he rackle v i appears before v j. Le D vi = {ds vi } Tv i s= be he deecion se of he rackle v i in ascending order of eporal doain, where ds vi is he s-h deecion in he rackle and T vi is he nuber of deecions in v i. We define a linear oion funcion P(l( ), Δ, υ), which generaes a prediced posiion saring fro l( ), i.e. P(l( ), Δ, υ) =l( )+Δ υ, where l( ) is he posiion of he deecion, Δ is he ie gap and υ is he consan velociy in he eporal doain. Le (ds vi ds vj ) be he ie gap beween he deecions ds vi and ds vj. Due o fac ha he srong correlaions beween nearby fraes and he weak correlaions beween disan fraes, we only use τ deecions in he las a few fraes of v i and he firs a few fraes of v j o calculae he oion siilariy beween he. To reduce he influence of noise, he deviaions of boh v i backward predicion and v j forward predicion are used o easure he oion consisency beween he wo rackles. Le l vi p,q(d vi )=P ( l(d vi ), (d vi d vj vi l(dp ) l(d ), v i q )) (d v i p d v i q ) be he prediced posiion saring fro he deecion d vi wih he average velociy beween he posiions of dp vi and dq vi, = T vi is he nuber of deecion responses in he rackle T vi. Then, he oion siilariy beween wo rackles v i and v j is calculaed as S (v i, v j )= + l i p= τ q=p+ τ p exp ( l(d vi ) l vj p,q(d vj ) /σ ) p= q= exp ( l(d vj ) p,q(d lvi vi ) /σ). ().. Trajecory Soohness Affiniy The arge rajecories should be coninuous and sooh in spaio-eporal doain, which provides us wih effecive inforaion o easure he confidence of he rackles in e belonging o he sae arge. We firsly erge he rackles in he hyperedge e according o heir appearing ie o ge he hypoheical rackle T e. Le D e = i= vi {dj }li j= be he sored deecion response se in he rackle T e according o he ascending order of eporal doain, where = T vi represens he nuber of deecion responses in he rackle. We saple soe deecions in he rackle T e wih equal sep δ o ge he fiing deecion poin se D f e = {d i} i T vi i=+k δ,k N. The reained deecion poin se is De r = D e \ D f e. We use D f e o ge he fied rajecory T e of he erged hypoheical rackle using he cubic spline fiing algorih, and calculae he deviaions beween he deecion poins in he se De r and he poins in he fied rajecory T e wih he sae ie index. Thus, he sooh affiniy of hyperedge e is calculaed as: A s (e )=exp ( l(d i ) l( T e ((d i ))) /σ ) s, d i De r (5) where l(d i ) is he posiion of he deecion response d i, (d i ) is he appearing ie of he deecion response d i, and T e ((d i )) represens he deecion response in he rajecory T e a ie (d i ). 5. Trackles Dense Neighborhoods Searching Afer consrucing he rackle relaion graph, we reveal he dense neighborhoods on i. The core proble in dense neighborhoods revealing is how o ge he nuber of neighborhoods and he nuber of nodes in each dense neighborhood. Here, he nuber of neighborhoods and he nuber of nodes in each neighborhood are regarded as he hidden variables in opiizaion process, which are inferred by axiizing he average affiniy value of he neighborhoods. In his way, uliple dense neighborhoods can be successfully discovered. Then, he final rackles in each segen can be obained by siching he rackles in each neighborhood correcly. In his secion, we deail he dense neighborhoods searching in each segen. To ensure all dense neighborhoods are revealed, we se each node in he graph as a saring poin and search is dense neighborhood. If he rackle belongs o a real dense neighborhood, he affiniies beween i and oher nodes in his real dense neighborhood are usually large. Thus, is he dense neighborhood will be found ou. On he conrary, if he rackle has weak relaionships wih oher rackles, he affiniies beween he will be low, which indicaes ha i does no belong o any coheren dense neighborhoods. Thus, he rackle will be reaed as a false posiive and ig

5 nored in he final clusers. For a saring poin v p, we ai o find ou is dense neighborhood N (v p ), which conains he axial average affiniy value. The opiizaion proble is forulaed as N (v p ) = arg ax C(v p N(v p )) N (v p) (6) s.. N (v p ) V, v p / N(v p ), N (v p ) = k. where C(v p N(v p )) is he affiniy easure funcion, ha reflecs he affiniy disribuion in he graph. Le U = {v p } N(v p ) represen he se conaining he verex v p and is k neighborhood verices. Thus, he subse U V conains k +verices. Le y R n be he indicaor vecor of he subse U, i.e. y i =,ifv i U ; oherwise, y i =0. Then, he subjecs in (6) are convered o n i= y i = k +, y i {0, } and y p =. The firs wo consrains reflec ha here exiss k +rackles belonging o he sae arge, which is indicaed by he soluion y, and he las one reflecs he soluion us conain he rackle v p. Le E U be he edge se corresponding o he verex se U. If he rackles in U belong o he sae arge, os of he edges in E U should have large affiniy values. Naurally, he oal affiniy value of he edge se E U is calculaed as C(U )= e E U A(e ). In our racking ask, he affiniy values in he graph/hypergraph are all non-negaive, i.e. A(e ) 0. Obviously, C(U ) usually increases as he nuber of verices in subse U increases. Thus, i is ore reasonable o use he average affiniy value o describe he confidence of he dense neighborhoods han he oal affiniy value, which can successfully handle he diension diversiy beween differen dense neighborhoods. Since n i= y i = k +, here are (k +) suands in C(U ). The average value C(U ) is aken as he objecive o indicae he rue dense neighborhood, ha is C(U )= (k +) C(U )= {}}{ A(e y ) k + y k +. e E U Then he opiizaion proble in (6) can be furher siplified as: ax g(x) = A(e {}}{ ) x x x e E U n s.. x i =, x i {0,ɛ}, x p = ɛ. i= where x i = yi k+ and ɛ = k+. Essenially, his is a cobinaorial opiizaion proble, which is NP-hard. To reduce is coplexiy, he subjecs in (7) are relaxed o x i [0,ɛ], i.e. 0 x i ɛ. Then, he pairwise updaing algorih [] is used o solve he proble in (7) effecively. Please refer o [] for ore deails abou he opiizaion sraegy of he proble in (7). (7) 6. Pos-Processing As discussed in secion 5, we se each node in he relaion graph as a saring poin o ge he opial clusers (dense neighborhoods) Ψ={ψ i } n i= and he corresponding average affiniy values, where n is he oal nuber of saring poins. The average affiniy value reflecs he reliabiliy of he neighborhood o be correc. Ψ is sored o ge he processed clusers Ψ ={ ψ i } n i= according o he average affiniy values in he descending order. Le Ψ be he opial clusers afer pos-processing. We se Ψ = a firs and add he clusers in Ψ sequenially. For he i-h coponen ψ i Ψ, we check wheher i inersecs wih he clusers in Ψ. If here is no inersecion beween ψ i and all clusers in Ψ, we add ψ i direcly o Ψ, i.e. Ψ Ψ { ψ i }. Oherwise, we use he designed Conservaive Sraegy or Radical Sraegy o add he cluser ψ i in differen layers. Suppose he cluser ψ i inersecs wih ψk. In he firs a few layers in opiizaion, he rackles are so shor ha conain liied evidence o deerine which arge i belongs o. To avoid ideniy swiches, a Conservaive Sraegy is designed as reoving he inersecion par fro he cluser ψ i and hen adding i o Ψ, i.e. ψi ψ i /ψk and Ψ Ψ { ψ i }. On he oher hand, he rackles in he las a few layers conain enough evidence o deerine which cluser i belongs o. Thus, in order o reduce he fragenaions, a Radical Sraegy is designed as direcly erging he clusers ψ i and ψk, i.e. ψ k ψ k ψ i. In his way, he pos-processed dense neighborhood se Ψ is obained. According o he dense neighborhood se Ψ, he opial rackles in he segen are acquired by siching he rackles in each cluser. 7. Experiens We evaluae H T on six challenging publicly available video sequences, including boh high-densiy and lowdensiy sequences. Five of he are par of he PETS009 daabase [5] and he res one is he ParkingLo sequence fro [5]. Noably, we rack all he arges in he D iage plane. For he quaniaive evaluaion, we rely on he widely used CLEAR MOT erics [5]. The Muli-Objec Tracking Accuracy (MOTA) cobines all errors (False Negaives (FN), False Posiives (FP), Ideniy Swiches (IDs)) ino a single nuber. The Muli-Objec Tracking Precision (MOTP) averages he bounding box overlap over all racked arges as a easure of localizaion accuracy. Mosly Los (ML) and Mosly Tracked (MT) scores are copued on he enire rajecories and easure how any Ground Truh rajecories (GT) are los (racked for less han 0% of heir life span) and racked successfully (racked for a leas 80%). Oher erics include Recall (Rcll), Precision (Prcsn), Fragenaions of he arge rajecories (FM) and False Alars per Frae (Fa/F). As discussed in [7], he inpu deecion responses and 80 86

6 anually annoaed evaluaion groundruh grealy influence he quaniaive resuls of he racking perforance. For fair and coprehensive coparison, we use he sae inpu deecion responses and anually annoaed evaluaion groundruh for all rackers in each sequence and ake soe racking resuls direcly fro he published papers. Since soe rackers coplee he racking ask in D world coordinaes, siilar as [6], we evaluae he racking perforance in D world coordinaes for he sequences fro PETS009. D evaluaion is exploied on he ParkingLo sequence because of he lack of caera paraeers. For he D evaluaion, he hi/iss hreshold of he disance beween he oupu rajecories and he groundruh on he ground plane is se o. For he D evaluaion, he hi/iss hreshold of he inersecion-over-union of he bounding boxes beween he oupu rajecories and he groundruh is se o 50%. Table presens he quaniaive coparison resuls of our H T and seven oher sae-of-hear rackers [8,,,, 5, 9, 6]. Soe qualiaive racking resuls of H T are presened in Fig.. Alhough MOTA reveals he coprehensive perforance, IDs, FM and MT sill play iporan roles in deerining perforance of he racker. As expeced, our H T ouperfors he sae-ofhe-ar rackers in IDs, FM and MT erics on os of he six sequences and gives coparable or even beer perforances in MOTA siulaneously. Specifically, due o he differen seings of he racking area, depiced by he highlighed region in PETS009 sequences in Fig., he racking resuls of [8, 9] in PETS009 sequences are no lised here for coparison. 7.. Ipleenaion Deails We ipleen H T in C++ wihou any code opiizaion. Given he deecion responses, he proposed ehod runs abou -fps in he PETS009 sequences and 6-fps in he ParkingLo sequence. The experiens are carried ou on a Inel.GHz PC plafor wih 6 GB eory. The reference paraeers used in his paper are presened as follows. The weigh paraeers in he affiniy value calculaion of () are, ω =0.6, ω =0., and ω =0.. The balance paraeers beween he RGB color hisogra and shape gradien hisogra of () are, λ = 0. and λ =0.0. The siga in oion and rajecory soohness affiniy values calculaion of () and (5) are, σ =5.0and σs =.0. We use and order hypergraph in he firs and second layers, respecively. The radiional graph is used for he reained layers. Every 6-8 fraes are cobined o generae he segens for he s layer and -5 segens are cobined for he reained ones. For he pos-processing sraegy of H T, we use he Conservaive Sraegy for he firs wo layers and he Radical Sraegy for he reaining ones. 7.. Low-Densiy Sequences PETS009-SL. SL is he os widely used sequence in uli-arge racking ask. I consiss of 795 fraes and includes he non-linear oion, arges in close proxiiy and a scene occluder challenges. The deecor ay fail when he arges walk behind he scene occluder, which grealy challenges he perforance of he rackers. Alhough he resuls presened in Table see o saurae on MOTA eric, our racker achieves he lowes IDs (5) and highes MT (). ParkingLo. The sequence consiss of 000 fraes of a relaively crowded scene. There are up o pedesrians walking in parallen a parking lo. I includes frequen occlusions, issed deecions, and parallel oion wih siilar appearances challenges. As shown in Table, our racker ouperfors oher rackers in nearly all evaluaion erics. Our MOTA, MT, IDs and FM are 88.%,, and respecively. Noably, H T alos racks all he arges successfully and achieves he lowes IDs (less han he second lowes one by ). Discussion. As presened in Table, in hese wo sequences, H T ouperfors oher rackers by reliably high MOTA and MT as well as sably low IDs and FM. Jus considering he local siilariies of deecions, i is hard for oher rackers [8,,, ] o achieve robus racking perforance, especially when wo arges wih siilar appearance walk closely. Noe ha our H T perfors weln his challenge by considering he siilariies aong uliple differen rackles in a global view. 7.. High-Densiy Sequences PETS009-SL. SL consiss of 6 fraes wih high dense crowd. Noe ha i conains 7 pedesrians oving non-linearly. The severe occlusion happens frequenly in his sequence. Our racker has he bes perforance in MOTA, ML, FN, Rcll, Prcsn and has coparable perforance in oher erics. PETS009-SL. SL is a challenging sequence wih high crowd densiy. I consiss of 0 fraes wih up o pedesrians oving non-linearly. In addiion, his sequence also includes frequen occlusions, issed deecions and illuinaion variaion challenges. H T presens he persuasive racking perforance wih he highes MOTA, MT, ML, Rcll and Prcsn. PETS009-SL. PETS009-SL- and PETS009- SL- are wo dense sequences including and fraes respecively and boh of he include he arges wih linear oion. These wo sequences are originally inended for person couning and densiy esiaion. H T no only gives he ipressive MOTA, bu also sands ou wih he highes MT as well as lowes IDs. Discussion. Copared o he low-densiy sequences, he superioriy of H T on high-densiy sequences is ore 8 87

7 Table. Quaniaive coparison resuls of our H T wih oher sae-of-he-ar rackers. The inpu deecion responses and evaluaion groundruh used in each sequence are presened. The racking resuls of he ehods arked wih he aserisk are aken direcly fro he published papers and he ohers are obained by running he publicly available codes wih he sae inpu deecion responses and evaluaion groundruh used in our racker. The sybol eans higher scores indicae beer perforance while eans lower scores indicae beer perforance. The red and blue color indicae he bes and he second bes perforance of he racker on ha eric. Sequence Mehod MOTA MOTP GT MT ML FP FN IDs FM Rcll Prcsn Fa/F PETS-SL Anon e al. [6] 90.6% 80.% % 98.% 0.07 (Deecion []) Berclaz e al. [] 80.% 7.0% % 96.% 0.6 (Groundruh [5]) Anon e al. [] 86.% 78.7% % 97.6% 0. (795 fraes) Anon e al. [] 88.% 79.6% % 98.7% 0.06 (up o 8 arges) Pirsiavash e al. [8] 77.% 7.% % 97.% 0. H T 9.7% 7.9% % 98.% 0.08 PETS-SL Anon e al. [6] 56.9% 59.% % 89.8%. (Deecion [0]) Berclaz e al. [].% 60.9% % 9.% 0. (Groundruh [5]) Anon e al. [] 8.5% 6.0% % 9.7% 0.69 (6 fraes) Anon e al. [] 8.0% 6.6% % 9.7% 0.56 (up o arges) Pirsiavash e al. [8] 5.0% 6.% % 95.% 0.6 H T 6.% 5.7% % 90.%.7 PETS-SL Anon e al. [6] 5.% 6.6% % 90.9% 0.70 (Deecion [0]) Berclaz e al. [] 8.8% 6.8% % 95.7% 0.9 (Groundruh [5]) Anon e al. [] 5.% 5.% % 9.9% 0.60 (0 fraes) Anon e al. [] 6.9% 57.8% % 96.% 0.8 (up o arges) Pirsiavash e al. [8].0% 6.0% % 97.0% 0.9 H T 55.% 5.% % 9.0% 0.6 PETS-SL- Anon e al. [6] 57.9% 59.7% % 9.8% 0.6 (Deecion [5]) Berclaz e al. [] 5.5% 6.8% % 9.6% 0. (Groundruh [5]) Anon e al. [] 8.0% 6.5% % 97.% 0.5 ( fraes) Anon e al. [] 5.% 6.% % 96.5% 0. (up o 0 arges) Pirsiavash e al. [8] 5.% 66.8% % 99.5% 0.0 H T 57.% 5.8% % 97.8% 0. PETS-SL- (Deecion [5]) Anon e al. [] 0.0% 69.% % 97.9% 0.5 (Groundruh [5]) Anon e al. [] 7.6% 65.8% % 96.8% 0. ( fraes) Pirsiavash e al. [8].8% 76.5% % 97.8% 0. (up o arges) H T.% 7.9% % 99.7% 0.0 ParkingLo Zair e al. [5] 90.% 7.% % 98.% - (Deecion [7]) Shu e al. [9] 7.% 79.% % 9.% - (Groundruh [7]) Anon e al. [] 60.0% 70.7% % 9.% 0.65 (000 fraes) Anon e al. [] 7.% 76.5% % 89.%.0 (up o arges) Pirsiavash e al. [8] 65.7% 75.% % 97.8% 0.6 H T 88.% 8.9% % 98.% 0.6 obvious. In he crowded scene, e.g. PETS009-SL, PETS009-SL, PETS009-SL-, and PETS009- SL-, he appearance of he arges are siilar wih each oher, and he occlusion happens frequenly aong he arges, which grealy challenges he robusness of he rackers. As shown in Table, H T ouperfors oher rackers in high-densiy sequences ainly due o he dense neighborhoods searching on rackle relaion hypergraph and he local-global hierarchical srucure in opiizaion, which considers he relaionships aong uliple rackles globally. However, our racker obains relaive worse perforance in MOTP eric, especially in he sequences PETS009-SL and PETS009-SL, ainly due o he non-linear oion of he arges when i is occluded and our linear inerpolaion based rajecory recover echanis akes i hard for our racker o recover he precise arge saes in he occluded fraes. On he oher hand, oher ehods achieve higher MOTP, e.g. [8] and [] always fail o idenify he arges when he occlusion happens and iss he arges copleely, refleced by he MT and FN erics. The arges sae in each frae of hese rackers are generaed by he inpu deecion responses, which are precise enough o obain he higher MOTP. Since he bes perforance are achieved in he os iporan erics for he uli-arge racking ask, i.e. MOTA, MT, IDs and FM, we can conclude ha our H T works bes. 8. Conclusion In his paper, a hierarchical dense neighborhoods searching based uli-arge racker is proposed. The uli-arge racking is forulaed as a dense neighborhoods searching proble on he uliple relaion affiniy graphs/hypergraphs consruced hierarchically, which considers he relaionships beween differen rackles across he eporal doain o resrain he IDs and Fragenaions. The appearance, oion and rajecory soohness properies are naurally inegraed in he graph affiniy values. Then, he dense neighborhoods searching is solved by he pairwise updaing algorih effecively. Experienal coparison wih he sae-of-he-ar racking ehods deonsrae he superioriy of our racker. In fuure work we plan o ake our racker reach real-ie perforance by ore efficien ipleenaion. 8 88

8 ParkingLo PETS009-SL- PETS009-SL PETS009-SL PETS009-SL- PETS009-SL Figure. Tracking resuls of our racker in sequences ParkingLo, PETS009-SL-, PETS009-SL, PETS009-SL, PETS009- SL-, and PETS009-SL. The highligh area of PETS009 sequences is he racking area, which is se o be he sae as [6]. Acknowledgen This work was suppored by he Chinese Naional Naural Science Foundaion Projecs 6050, 6056, 60507, 6067, 67507, Naional Science and Technology Suppor Progra Projec 0BAK0B0, Chinese Acadey of Sciences Projec KGZD-EW-0-, and AuhenMeric Research and Developen Funds. References [] M. Andriluka, S. Roh, and B. Schiele. People-racking-by-deecion and people-deecion-by-racking. In CVPR, 008. [] A. Andriyenko and K. Schindler. Muli-arge racking by coninuous energy iniizaion. In CVPR, pages 65 7, 0. [] A. Andriyenko, K. Schindler, and S. Roh. Discree-coninuous opiizaion for uli-arge racking. In CVPR, pages 96 9, 0. [] J. Berclaz, F. Fleure, E. Türeken, and P. Fua. Muliple objec racking using k-shores pahs opiizaion. PAMI, (9):806 89, 0. [5] K. Bernardin and R. Siefelhagen. Evaluaing uliple objec racking perforance: The clear o erics. EURASIP J. Iage and Video Processing, 008, 008. [6] W. Brendel, M. R. Aer, and S. Todorovic. Muliobjec racking as axiu weigh independen se. In CVPR, pages 7 80, 0. [7] Dehghan, O. Oreifej, E. Hand, and M. Shah. hp://crcv.ucf. edu/daa/parkinglot. [8] M. Hofann, D. Wolf, and G. Rigoll. Hypergraphs for join uliview reconsrucion and uli-objec racking. In CVPR, pages , 0. [9] C. Huang, Y. Li, and R. Nevaia. Muliple arge racking by learningbased hierarchical associaion of deecion responses. TPAMI, 5():898 90, 0. [0] H. Izadinia, I. Saleei, W. Li, and M. Shah. (MP) T: Muliple people uliple pars racker. In ECCV, pages 00, 0. [] H. Jiang, S. Fels, and J. J. Lile. A linear prograing approach for uliple objec racking. In CVPR, 007. [] C.-H. Kuo, C. Huang, and R. Nevaia. Muli-arge racking by online learned discriinaive appearance odels. In CVPR, pages , 00. [] H. Liu, L. J. Laecki, and S. Yan. Robus clusering as ensebles of affiniy relaions. In NIPS, pages, 00. [] H. Liu, X. Yang, L. J. Laecki, and S. Yan. Dense neighborhoods on affiniy graph. IJCV, 98():65 8, 0. [5] A. Milan. Coninuous energy iniizaion racker websie. hp: // [6] A. Milan, S. Roh, and K. Schindler. Coninuous energy iniizaion for uliarge racking. TPAMI, 6():58 7, 0. [7] A. Milan, K. Schindler, and S. Roh. Challenges of ground ruh evaluaion of uli-arge racking. In CVPR Workshops, pages 75 7, 0. [8] H. Pirsiavash, D. Raanan, and C. C. Fowlkes. Globally-opial greedy algorihs for racking a variable nuber of objecs. In CVPR, pages 0 08, 0. [9] G. Shu, A. Dehghan, O. Oreifej, E. Hand, and M. Shah. Par-based uliple-person racking wih parial occlusion handling. In CVPR, pages 85 8, 0. [0] J. Yan, Z. Lei, D. Yi, and S. Z. Li. Muli-pedesrian deecion in crowded scenes: A global view. In CVPR, pages 9, 0. [] B. Yang. hp://iris.usc.edu/people/yangbo/ downloads.hl. [] B. Yang and R. Nevaia. Muli-arge racking by online learning of non-linear oion paerns and robus appearance odels. In CVPR, pages 98 95, 0. [] B. Yang and R. Nevaia. An online learned CRF odel for uliarge racking. In CVPR, pages 0 0, 0. [] T. Yang, S. Z. Li, Q. Pan, and J. Li. Real-ie uliple objecs racking wih occlusion handling in dynaic scenes. In CVPR, pages , 005. [5] A. R. Zair, A. Dehghan, and M. Shah. GMCP-racker: Global uli-objec racking using generalized iniu clique graphs. In ECCV, pages 56, 0. [6] L. Zhang, Y. Li, and R. Nevaia. Global daa associaion for uliobjec racking using nework flows. In CVPR,