Single and Multiple Object Tracking Using a Multi-Feature Joint Sparse Representation

Transcription

1 Sngle and Mulple Objec Trackng Usng a Mul-Feaure Jon Sparse Represenaon Wemng Hu, We L, and Xaoqn Zhang (Naonal Laboraory of Paern Recognon, Insue of Auomaon, Chnese Academy of Scences, Bejng ) {wmhu, wel, xqzhang}@nlpr.a.ac.cn Sephen Maybank (Deparmen of Compuer Scence and Informaon Sysems, Brkbeck College, Male Sree, London WC1E 7HX) sjmaybank@dcs.bbk.ac.uk Absrac: In hs paper, we propose a rackng algorhm based on a mul-feaure jon sparse represenaon. The emplaes for he sparse represenaon can nclude pxel values, exures, and edges. In he mul-feaure jon opmzaon, nose or occluson s deal wh usng a se of rval emplaes. A sparse wegh consran s nroduced o dynamcally selec he relevan emplaes from he full se of emplaes. A varance rao measure s adoped o adapvely adjus he weghs of dfferen feaures. The mul-feaure emplae se s updaed adapvely. We furher propose an algorhm for rackng mul-objecs wh occluson handlng based on he mul-feaure jon sparse reconsrucon. The observaon model based on sparse reconsrucon auomacally focuses on he vsble pars of an occluded objec by usng he nformaon n he rval emplaes. The mul-objec rackng s smplfed no a jon Bayesan nference. The expermenal resuls show he superory of our algorhm over several sae-of-he-ar rackng algorhms. Index erms: Vsual objec rackng, Trackng mul-objecs under occlusons, Mul-feaure jon sparse represenaon 1. Inroducon The ask of vsual objec rackng s o nfer movng objecs saes, e.g. locaon, scale, or velocy, from he observaons n a vdeo. Vsual objec rackng s a key componen n numerous applcaons, such as vsual survellance, vson-based conrol, human-compuer nerfaces, nellgen ransporaon, and augmened realy. An appearance model s essenal for rackng objecs. Changes n llumnaon, vewpon and pose, occlusons, and background cluer make dffcul o consruc a robus appearance model whch s capable of adapng o changes n he envronmen. Many appearance models [1,, 3, 4] have been proposed for objec rackng. These nclude models based on hsograms, kernel densy esmaes, Gaussan mxure models (GMMs), condonal random felds, subspaces, and dscrmnave classfcaon. Sparse represenaon-based objec appearance models [5, 6] have only recenly been appled o vsual rackng. Me e al. [5, 45] proposed a sparse approxmaon-based rackng algorhm usng he L1 -regularzed mnmzaon. An observaon n a frame s sparsely represened n he space spanned by objec emplaes and rval emplaes. The objec emplaes are obaned from he prevous frames. The rval emplaes accoun for he pxels conamnaed by occluson or nose. The convenonal sparse represenaon-based rackng only uses he pxel gray levels n he sequence of frames o consruc he emplae se. The pxel gray level s no always 1

2 represenave enough o dsngush he objec from he background or from oher objecs. I s necessary o consruc a new and more powerful sparse represenaon-based appearance model whch can combne useful feaures, such as colors, exures, or edges. In addon, s sll challengng o use a sparse represenaon-based appearance model o rack mulple objecs under occlusons. In hs paper, we propose a rackng algorhm based on a mul-feaure jon sparse represenaon whch s used o fuse he emplaes consruced from he dfferen mage feaures. A few emplaes, ha are mos represenave for reconsrucng each canddae observaon, are obaned by deermnng he reconsrucon coeffcens of he emplaes of he mul-feaures. The observaon wh he smalles reconsrucon error s chosen as he rackng resul under he parcle flerng framework. We furher exend hs mul-feaure jon sparse reconsrucon-based algorhm o mul-objec rackng wh occluson handlng. The locaon and sze of he occluded pars are nferred usng he rval emplaes. The ask of mul-objec rackng wh occluson handlng s reduced o a smple jon sae Bayesan nference. The man conrbuons of our work are as follows: A mul-feaure jon sparse represenaon-based appearance model s proposed for objec rackng. The varance rao measure n [13] s nroduced o esmae dfferen feaures weghs for calculang he sparse reconsrucon error. The emplaes of he objecs are updaed adapvely usng he rackng resuls o keep he represenave emplaes n he rackng process. We use a sparse wegh consran n he jon sparse represenaon o dynamcally selec he emplaes from he emplae se and esmae he coeffcens of he emplaes. A relavely large pool of emplaes s kep n he emplae se. An acceleraed proxmal graden (APG) algorhm s nroduced o handle he mul-feaure sparse represenaon ask. A mul-feaure jon sparse represenaon-based algorhm s proposed for rackng mul-objecs hrough occlusons. Implc and explc mehods for handlng occlusons are proposed o deermne he un-occluded pars of an objec. The sparse reconsrucon selecs he vsble pars of he objec for machng wh he emplaes. A cross eraon-based mehod s proposed o reduce he compuaonal complexy of fndng he opmal sae n he jon sae space of mul-objecs. We es our algorhms on dfferen vdeo sequences nvolvng heavy occlusons, large changes n appearance and pose, and nosy backgrounds. Wh he above conrbuons, our work sgnfcanly exends he baselne [5]. The res of he paper s organzed as follows. Secon dscusses relaed work. Secon 3 summarzes he sparse represenaon-based appearance model. Secon 4 proposes our mul-feaure jon sparse represenaonbased appearance model. Secon 5 descrbes he Bayesan sae nference for sngle objec rackng. Secon 6 presens our algorhm for rackng mul-objecs wh occluson handlng. Secon 7 shows he expermenal resuls. Secon 8 concludes he paper.. Relaed Work In order o gve he broad conex, we brefly revew he relaed work on appearance model-based rackng

3 and mul-objec rackng..1. Appearance model-based rackng In erms of he consrucon of appearance models, rackng algorhms can be caegorzed no generave model-based [1] and dscrmnave model-based []. The generave model-based mehods [3, 4] descrbe he vsual observaons of a movng objec, and hen rackng s reduced o a search for an opmal sae ha yelds an objec appearance mos smlar o he appearance model. In he dscrmnave models, rackng s vewed as a bnary classfcaon based on an opmal decson boundary whch dsngushes he objec from he background Generave models The useful mehods for consrucng he generave appearance models nclude color hsograms, GMMs (Gaussan mxure models), subspace learnng, and manfold learnng. Color hsograms [37, 38] of mage regons are wdely used o consruc generave appearance models because of he smplcy wh whch hey are compued and her robusness o regon scalng, roaon, and shape varaons. Ther lmaon s ha hey usually gnore he spaal dsrbuon of pxel values. GMM-based appearance models use a mxure of weghed Gaussan dsrbuons o learn a sascal model for colors. Jepson e al. [7] modeled he objec appearance as a mxure of a sable componen, a ransen componen, and an ouler componen. Zhou e al. [8] modeled an objec appearance usng an adapve mxure of Gaussans. Yu e al. [9] proposed a spaal appearance model whch capures he properes of boh local appearance changes and global spaal nformaon abou pxel values. Wang e al. [10] used a spaal-color mxure of Gaussans o model objec appearance. On one hand, GMMs can be appled o ndvdual pxels, and hen he correlaons beween he values of nearby pxels are gnored. On he oher hand, GMMs can be appled o objec mage regons convoluon oupus, and hen he local srucure of mage nensy can be explcly capured. The lmaons of GMMs-based appearance models are ha s necessary bu dffcul o manually se he number of Gaussans and he updang of he GMMs-based appearance models requres a number of hresholds whch have o be se manually, makng auomac updang dffcul. In conras o color hsograms and GMMs-based appearance models whch are pxel-based and hus sensve o global appearance changes and nose, lnear subspace learnng-based appearance models [39] consder he mage as a whole and hen represen he mage as a vecor. Lm e al. [40] presened a human rackng algorhm based on a nonlnear dmenson reducon echnque. Ho e al. [41] presened a vsual rackng algorhm based on lnear subspace learnng. Ross e al. [4] proposed a generalzed rackng algorhm based on an ncremenal vecor subspace learnng mehod. Wu e al. [11] used an onlne ensor decomposon mehod o capure spaal layou nformaon abou appearances for rackng. won and Lee [18] decomposed an observaon model no mulple basc observaon models ha are consruced usng he sparse prncpal componen analyss (PCA). Nguyen e al. [35] proposed a probablsc PCA-based mehod o model mages usng a low-dmensonal jon Gaussan dsrbuon beween pxels. The second-order sascs among pxels were ncluded n he covarance marx of he weghed prncpal componens. Subspace learnng-based mehods 3

4 have he followng lmaons: When he appearance of an objec has large changes, he correspondence of pxels beween he objec and he subspace s no accurae and samples may le beyond he range of he lnear subspace. Ths can cause rackng falure. ernel mehods can be used o descrbe he nonlnear dsrbuon of samples, bu s dffcul o selec approprae kernel funcons n hs applcaon. The reconsrucon error for subspace learnng-based appearance models s usually based on he sum of squared dfferences. Ths makes he appearance models sensve o mage nose and paral occluson. Nonlnear manfold learnng mehods usually consruc appearance models whch depend on local feaures compued from he values of nearby pxels. Porkl e al. [1] proposed a Remannan merc-based objec rackng mehod n whch objec appearances were represened usng he covarance marx of mage feaures. Tuzel e al. [43] proposed an algorhm for deecng people by classfcaon on Remannan manfolds. The lmaons of hese mage covarance marx-based appearance models are ha hey do no drecly model occluson and nose, and drec spaal nformaon abou poson relaons beween pxels s parally los..1.. Dscrmnave models Dscrmnave models selec dfferen dscrmnave feaures and consruc a classfer n he feaure space o dsngush beween he feaures from he objec and he feaures from he background. Avdan [4] consruced offlne a suppor vecor machne (SVM)-based classfer whch was ncorporaed no an opcal flow-based racker. Collns e al. [13] developed an effcen onlne feaure rankng mechansm whch was embedded n a rackng sysem. Avdan [14] combned several weak classfers no a srong classfer o label pxels as objec or background. Grabner e al. [15, 16] adoped onlne boosng o selec dscrmnave local feaures for rackng. Saffar e al. [17] nroduced a random fores algorhm, n whch he decson rees were bul onlne, o selec feaures for rackng. Zhang e al. [51] proposed a rackng algorhm wh an appearance model based on feaures exraced from he mul-scale mage feaure space. Samples of foreground objecs and he background were compressed usng he same sparse measuremen marx. The lmaon of he dscrmnave models s ha hey are dependen on he selecon of suable posve and negave samples whch are used o updae he classfer. An unsuable selecon may cause racker drf. In summary, dfferen appearance models, no maer wheher hey are generave or dscrmnave, have her own mers and lmaons. I s neresng o develop a new robus rackng algorhm whch effecvely fuses mulple appearance cues and drecly models occlusons and nose... Mul-objec rackng wh occluson handlng For mul-objec rackng [48, 49, 50], f close objecs have very smlar appearances and hey are racked separaely usng a sngle objec rackng algorhm, hen he dfferen rackers may lock ono he same objec. To carry ou mulple objec rackng, a jon poseror or a jon lkelhood for all he objecs s usually mananed o explcly model he neracons beween objecs. However, he compuaon of he jon sae space s very expensve. Mulple objec rackng s usually ransformed no a consraned opmzaon problem by defnng a 4

5 one-o-one mappng consran o smplfy he compuaon. Then, opmzaon mehods, such as he Hungaran algorhm or he nework flow algorhm, can be adoped o handle mul-objec rackng problem. In mul-objec rackng, occluson handlng based on hgh level assocaons s crucal. The usual way o ackle occlusons s o explcly model he occluson relaons beween dfferen objecs. Dfferen models have been used. Rasmussen and Hager [30] deduced occluson relaons by analyzng he afflaon of each pxel o he dfferen objecs. Elgammal and Davs [31] segmened people under occluson by ncorporang he occluson relaons of he dfferen deph layers no a lkelhood funcon. Wu e al. [3] appled a Bayesan nework o rack wo faces hrough occluson n whch an exra hdden process for he occluson represenaon was nroduced. Sudderh e al. [33] proposed a hand rackng algorhm n whch he occluson relaons were nferred by usng nonparamerc belef propagaon. The occluson reasonng s complex, and s very dffcul o deduce he occluson relaons. If he deduced occluson relaons are no correc, hen rackng may fal. In conras o he explc modelng of occluson relaons, an alernave s o mplcly handle occlusons by adopng parcular rules. MacCormck and Blake [34] developed a daa assocaon fler based on a probablsc excluson prncple o nfer he objec saes durng occlusons. Nguyen e al. [35] used he spaoemporal conex of each objec o manan he correc deny of he objec durng occlusons. Yang e al. [36] racked mul-objecs by fndng he Nash equlbrum n a game. The lmaon of he mplc mehods s ha s dffcul o defne effecve rules for occluson handlng. I s necessary o develop a mul-objec rackng algorhm whch has a hgh accuracy of occluson handlng whle keepng hgh robusness and whch does no rely on parcular rules for fndng occlusons. 3. Sparse Represenaon-Based Appearance Model Tradonal sparse represenaon-based rackng mehods [5, 7, 8, 9] are appled drecly o he nensy mages, and rval emplaes are used o model occluson and nose. The global appearance of an objec under dfferen condons can be well approxmaed by a low dmensonal subspace spanned by a se of objec emplaes. Therefore, durng rackng, an appearance canddae of an objec can be approxmaed by lnear combnaons of a se of emplaes whch are seleced from he rackng resuls n he prevous frames. A rackng resul s an mage pach whch s normalzed o a parcular sze and sacked o produce a -dmensonal vecor, where equals o he number of he pxels n he normalzed pach. Le { h } n 1 be a se of n emplaes, n.e., H [ h1, h,..., h n]. An appearance canddae m of he objec n a gven mage s approxmaed by a lnear combnaon of { h } n 1 [6]: m wh w h w h w (1) n n where w [ w1, w,..., w ] T n n s a coeffcen vecor and ε s a nose erm. If he appearance of he objec s badly affeced by nose or occluson, hen he componens of ε may be very large. A se of rval emplaes {,,..., } s defned o encode he values of ε. In he -h rval emplae, he -h pxel value s nonzero and 1 5

6 all he oher pxel values are 0. Each rval emplae also s sacked o produce a -dmensonal vecor,.e.,. Le 1 emplaes: u T [,,..., ]. The dscrepancy ε s encoded usng a lnear combnaon of he rval [,,..., ][ e, e,..., e ] T Te () 1 1 where e s he coeffcen of he -h rval emplae and e e1 e [,,..., ] T u. The composon of he objec e emplaes and he rval emplaes for a canddae observaon s shown n Fg. 1, where a rval emplae s shown as a dark square whch conans a sngle whe do correspondng o he nonzero value. The appearance m n (1) s rewren as: [ ] w m H T Bf e (3) ( n) where B [ H T ] and T T T ( n) f w e. Fg. 1. An llusraon of represenaon wh objec emplaes and rval emplaes for a canddae observaon. Vdeos are hghly redundan daa. Much nformaon, such as he appearance of an objec n consecuve frames, s repeve. I s nuve ha he appearance of a racked objec n a vdeo can be sparsely represened by s appearances n he prevous frames. So, an observaon m can be descrbed by a lnear combnaon of objec emplaes H and rval emplaes T, wh he vecor f consraned o be sparse. As a basc sraegy n sparse represenaon, he sparseness of f s descrbed by he L 1 norm of f, whch s defned as f f 1 [5, 6]. Then, he opmal coeffcen vecor f can be found by mnmzng he square of he reconsrucon error for m wh an L 1 regularzaon. Le be he L norm whch s defned as y y for a vecor y. Then, he opmzaon s formulaed by: ˆ arg mn f 1 f m Bf f (4) where s a regularzaon facor and he L norm s used o specfy he reconsrucon error. The mnmzaon n (4) can be carred ou effcenly by lnear programmng [19]. In (4), here s radeoff beween he reconsrucon error and he sparseness. When he occluson s large, o decrease he reconsrucon error, he combnaon of rval emplaes may no longer be sparse n spe of he sparseness regularzaon. Bu, he opmal observaon correspondng o he rackng resul can oban less dense represenaon of he rval emplaes,.e., more un-occluded pxels can ake par n esmang he reconsrucon 6

7 error. Therefore, nroducon of rval emplaes makes he sparse represenaon more robus o dense occlusons [5, 6]. 4. Mul-Feaure Jon Sparse Represenaon-Based Appearance Model In conras wh he radonal nensy mages-based sparse represenaon appearance models for rackng, we propose a mul-feaure jon sparse represenaon-based appearance model whch fuses mulple cues, such as hue, sauraon, nensy, he graden-based edge emplae, and he exure feaure obaned by Gabor flerng [13], n order o acheve more robus rackng. Accordng o he nuon ha f objec appearances are smlar all her feaures are smlar, once an objec emplae s seleced for sparse represenaon of an observaon, all he feaures of he emplae should be vald for he sparse represenaon. So, jon sparse represenaon of mul-feaures can effecvely combne mul-feaures. For each feaure, he sparse represenaon, ogeher wh s occluson or nose handlng, are managed n he same way ha he radonal sparse represenaon-based mehods manage he nensy mage. All feaure emplaes are fused usng he L,1 norm-based jon sparse represenaon whch s a sandard framework for nformaon fuson, wh a sound heorecal bass [5, 53] and a smple mplemenaon Mul-feaure jon sparse represenaon We exrac dfferen ypes of feaure from each objec appearance emplae o form feaure emplaes. Each ype of feaure corresponds o one feaure emplae. The dmenson of each feaure s equal o he number of pxels n an objec appearance emplae. For each feaure k, he feaure emplae se H { h, h,..., h }, where n s he number of he objec appearance emplaes. The k-h feaure vecor k k k k 1 n k H s k m of a canddae observaon s represened usng he followng equaon: n k k k k w 1 m h (5) where w k [ w k 1,..., w k ] T n s he reconsrucon coeffcen vecor for he k-h feaure, and k s he resdual erm. The { w k } k 1 are found usng he leas square regressons wh he L,1 mxed-norm regularzaon [0, 1]: n 1 ˆ k k k W arg mn w m h W (6) W,1 k1 1 where 1 [,,..., n W w w w ]. The,1 L -mxed norm of W s: n n k ( ),1 w 1 k 1 1 W w (7) where vecor 1 T w [ w, w,... w ]. The,1 L mxed-norm regularzaon ncludes he L norm of he coeffcen w for each objec appearance emplae and he L 1 norm of he vecor of he L norm values for all he objec appearance emplaes. The L,1 mxed-norm guaranees jon sparse represenaon, because: The L 1 norm n he L,1 mxed-norm ensures ha he objec appearance emplaes chosen o represen 7

8 an observaon are as few as possble; The L norm n he,1 k L norm ensures ha when w 0, he coeffcens { w k } k 1 of all he feaure emplaes for objec appearance emplae are equal o 0,.e. when he objec appearance emplae s no chosen o represen an observaon, all he feaure emplaes of he objec appearance emplae should also no be chosen o represen he observaon. 4.. Mul-feaure jon sparse represenaon-based appearance model If emplaes dssmlar o he observaon are seleced as nonzero enres for he sparse represenaon, hen here s an ncreased probably ha a background observaon s erroneously chosen as he rackng resul. To make sure ha he observaon les n he span of he represenave emplaes, we defne a wegh for he coeffcen of each emplae so as o favor he selecon of emplaes whch are smlar o he observaon. As he rackng resul n he curren frame s smlar o he rackng resul n he prevous frame, we wegh he objec emplae coeffcens usng he Eucldean dsances from he rackng resul n he prevous frame o he feaure emplaes. We defne a wegh marx k 1 D for objec emplae as: D dag m h, m h,..., m h,..., m h (8) 1 1 k k where k h s he k-h feaure s emplae of he -h appearance emplae and k m s he k-feaure vecor of he rackng resul n he prevous frame. The rval emplaes are added no (6) as specfed n (3). Le f k [( w k ) T,( e k ) T ] T ( k 1,..., ) and e ( j 1,,..., ). Then, 1 ( n ) [,,..., ] 1 T F f f f. Le j [ ej, ej,..., ej ] Equaon (6) becomes: n ˆ 1 k k k F mn m B f D w e j, (9) F k 1 1 j1 where he wegh marx D acs on he feaure emplaes bu no on he rval emplaes. I s seen ha by usng (8) nformaon abou prevous racks s ncorporaed no he opmzaon objecve. In he radonal sparse represenaon-based rackng [5] n whch he coeffcens of he emplaes are no weghed, he sze of he se of objec emplaes s usually requred o be small n order o conrol he compuaonal complexy for handlng he sparse opmzaon problem. Ths excludes many emplaes whch would be useful for he subsequen rackng. Inspred by localy-consraned lnear codng [], we nroduce a hreshold o consran he coeffcens n order o enlarge he sze of he se of he emplaes whou ncurrng a large compuaonal cos for solvng he sparse opmzaon. The wegh marx D n (8) s consraned and redefned as: D k k m h k k f m h max m h. (10) oherwse k k k where ϵ s a hreshold. Ths equaon ensures ha k D grows lnearly wh he dfference k k m h, unl he 8

9 dfference exceeds he hreshold ϵ [46]. Beyond he hreshold, k D s se as nfnely large, whch forces he coeffcen of he emplae o 0. In hs way, many more emplaes can be reaned n he emplae se. Usng he weghs n (10) o smplfy he search for a sparse represenaon s reasonable, as he am n a sparse represenaon s o choose as few emplaes as possble. An approxmaed acceleraed proxmal graden (APG) algorhm [0, 3] s used o effcenly solve he L,1 norm-based opmzaon n (9). Gven he feaure emplae ses k {H } k 1, he algorhm consss of alernae and erave updang of he coeffcen marx F [ f ] and an aggregaon marx k, k 1,.., 1,.., n V [ v ], k, k 1,.., 1,.., n where ndexes an eraon. Each eraon +1 consss of he followng wo seps: 1 a generalzed graden mappng sep o updae he marx F usng he aggregaon marx obaned n he prevous eraon. V an aggregaon sep o updae 1 1 V by lnearly combnng F and F. In he generalzed graden mappng sep, gven 1 V, F s updaed usng he followng wo formulae:, 1,, f k d k ( v k k ), k 1,,..., (11) f max 1,0 f f, 1,,..., n (1) where,, k ( B k ) T m k ( B k ) T ( B k ) v k (13) k k k k T n α s he sep sze, d [1/ D1,1/ D,...,1/ D n,1,...,1] o ensure ha only he coeffcens of he objec emplaes are affeced by he wegh consran, and denoes he elemen-wse produc of wo vecors. In he aggregaon sep, he aggregaon marx 1 1 V s updaed by lnearly combnng F and F : (1 ) V F ( F F ) (14) where s usually se as / ( ). Table 1 summares he opmzaon procedure of he mul-feaure jon sparse represenaon for an observaon. Table 1: The mul-feaure jon sparse represenaon for an observaon 1: Inpu: emplaes { H k }, an observaon k 1 { m k }, he regularzaon parameer λ, and he sep sze value α. k 1 k,0 k,0 : Inalzaon: Inalze f and v ; Se γ 0 = 1 and 0. 3: Repea {Man loop} k, 1 k k, k T k k T k k, 4: f d ( v ( ( B ) m ( B ) ( B ) v )), k =1,,..., 5: 1 1 f max 1,0 1 f f 6: 7: (1 ) V F ( F F ) 8: +1, 9: unl convergence s reached 10: Oupu: F., =1,,, n 9

10 Usng he opmal coeffcens k w for each feaure k obaned by he algorhm shown n Table 1, he k-h feaure vecor k m of he canddae sample 1 k M = [m,..,m,..., m ] s reconsruced by k k Hw. Then, he reconsrucon error R(M) of M s accumulaed over all he feaures: k 1 k k k k R( M) m H w (15) where { k } k 1 are he weghs ha measure he nfluences of dfferen feaures on he reconsrucon error. Usually, k s se as 1/ for smplcy, meanng ha dfferen feaures have he same nfluence on he reconsrucon error Feaure wegh adapaon We nroduce a varance rao [13] o adapvely adjus he weghs { k } for each feaure k, n order o gve a larger wegh o a feaure whch has more ably o dsngush beween he objec and he background. We deermne hs ably for each feaure usng he pxels whn he objec regon and he background pxels lyng nsde a recangle surroundng he objec regon. For each feaure, wo hsograms are used o esmae he dsrbuons of he values of he objec pxels and he values of he parcular background pxels, respecvely. Le { p ( )} 1,.., O and { q ( )} 1,.., O be he L 1 normalzed hsograms of he values of he objec pxels and he values of he background pxels, respecvely, where O s he number of bns and ndexes a bn. The log lkelhood l () for he -h bn s compued by: max( p ( ), ) l ( ) log (16) max( q ( ), ) where ξ>0 s a small value whch s used o ensure ha he numeraor and he denomnaor are no equal o zero. As he varance ( x) of a random varable x equals o he expecaon of x mnus he square of he expecaon of x: ( x) E( x ) ( E( x)), he varance ( lp, ) of { l ( )} 1,..., O wh respec o he hsogram { p ( )} 1,.., O s compued by: O O ( lp, ) E( l ( )) ( E( l( ))) p( ) l ( ) p( ) l( ). (17) 1 1 Smlarly, we compue he varance ( lq, ) of { l ( )} 1,..., O relave o { q ( )} 1,.., O and he varance ( l,( p q ) / ) of { ( )} 1,..., O funcon s defned as: l relave o p q 1,.., ( ) ( ) / O. The varance rao of he log lkelhood ( l,( p q) / ) ( l, p, q) ( l, p) ( l, q ). (18) The denomnaor of (18) s he sum of he whn class varances for he objec and background classes, and he numeraor s he oal varance of he daa from boh he objec and he background. The nuon behnd he varance rao s ha log lkelhood values of pxels on boh he objec and he background are expeced o be 10

11 ghly clusered, and he wo clusers are expeced o be spread apar as much as possble [13]. So, hs varance rao for he feaure can evaluae he feaure s ably o dsngush beween he objec and he background. The wegh k for each feaure k s se o he normalzed varance rao of he correspondng feaure: k j1 k k k ( l, p, q ). (19) j j j ( l, p, q ) 4.4. Updang of objec emplaes Durng rackng, he emplae se s updaed onlne usng he mos recen rackng resuls o capure changes n he objec appearance and keep more represenave emplae n he emplae se, n order ha rackng can be carred ou effecvely even when he sze of he emplae se s no large. Our adapve updang sraegy essenally follows he one proposed n [5]. The dfferences beween our updang sraegy and he sraegy n [5] and he jusfcaons of he dfferences are summarzed below: The emplaes obaned n he frs few frames are kep fxed durng he whole rackng process. Ths can reduce he possbly of rackng drf, as he frs emplaes are usually relable. In [5], he emplaes obaned n he frs frame were reaed equally wh oher emplaes and could be replaced. The wegh of each objec appearance emplae s updaed usng he coeffcens of all he feaures: exp( k w ). The correlaons beween dfferen feaures n each objec appearance n he jon k 1 sparse represenaon are refleced n he wegh, n conras o [5] n whch only he gray feaure s used for he emplae wegh updang. We add a forgeng facor n he esmaon of he wegh of each objec appearance emplae. Namely, he wegh s furher updaed by G g, where [0,1] s a forgeng facor, G s he curren me, and g [1, G] s he me of he emplae beng added no he emplae se. Then, he wegh assgned o he objec appearance emplae decreases over me and capures he mos recen changes. Ths forgeng facor s no used n [5]. The dfference beween he rackng resul M and an objec appearance emplae s esmaed by: k k k m h (0) k 1 If he dfference beween he new rackng resul and he objec emplae wh he lowes wegh n he curren emplae se s larger han a hreshold, hen hs objec emplae s replaced wh he new rackng resul o reflec he changes n appearance. The new emplae s nalzed o have he medan wegh of he curren emplaes. In [5], he dfference beween an observaon and a emplae s defned usng he cosne of he angle beween he feaure vecors for he observaon and he emplae. The cosne canno be drecly appled o he esmaon of smlary beween vecor ses, each of whch consss of mul-feaure vecors wh dfferen weghs. So, we use, nsead, (0) o esmae he smlary beween he ses of 11

12 feaure vecors. Occlusons and nose are consdered n our emplae updang sraegy. If he proporon of an objec s pxels whch are no badly affeced by nose or occluson n he objec, as deermned usng he rval emplaes, are larger han a hreshold, hen he appearance model of he objec s updaed, oherwse he emplae updang s no carred ou. Ths avods updang affeced by large occlusons or nose Remark Usng he weghs n (10), he sandard sze o whch each appearance s normalzed, he APG algorhm, and he sandard sze of he emplae se make praccal o solve he sparse codng for each frame. 5. Bayesan Sae Inference for Sngle Objec Trackng The objec s localzed usng a recangular wndow and s sae a frame s represened usng a 1 1 sx-dmensonal affne parameer vecor z [ x, x, r, c, a, ], where x and x are he D ranslaon parameers and ( r, c, a, ) are he deformaon parameers, correspondng o he roaon angle, he scale, he 1 aspec rao, and he skew drecon, respecvely. Gven an observaon M [ m,... m ], he ask of rackng s o nfer he sae z of he objec. Ths can be represened as a Bayesan poseror probably nference [4]: p( z M ) p( M z ) p( z z ) p( z M ) dz (1) where p( M z ) s he observaon model and p( z z 1) s he dynamc model. A parcle fler [4] approxmaes he poseror probably densy usng a se of weghed parcles. The observaon model p( M z ), whch reflecs he smlary beween M and he emplae se, s esmaed by: k k k k p( M z ) exp m H w () k 1 where η s a scalng facor conrollng he shape of he Gaussan funcon. A Gaussan dsrbuon G( z 1, ) s used o model he sae ranson dsrbuon correspondng o he dynamc model, where he mean vecor z 1 of he Gaussan dsrbuon s he affne parameer vecor esmaed a he prevous frame -1, and he covarance marx s se emprcally. The parcles a he curren me are drawn from he sae ranson dsrbuon G( z 1, ). Durng he samplng, f here s a negave value n he quanes whch are srcly posve, hen he sample s gnored. The wegh of each parcle z s evaluaed usng p( M z ). Maxmum a poseror (MAP) esmaon s used o esmae he sae of he objec a each frame: Observaons assocaed wh he parcles are found, and he observaon whch s mos smlar o he appearance model s aken as he rackng resul. Table summarzes he mplemenaon of he rackng algorhm. 1

13 Table : The rackng Algorhm 1: Inpu he se of he emplaes { H k } whch has been updaed a he prevous frame -1, and he sae z k 1 1 of he objec a frame -1. : Use he dynamc model G( z 1, ) o produce a number of parcles a he curren frame. 3: Use he algorhm shown n Table 1 o esmae he mul-feaure jon sparse represenaon for each parcle z a he curren frame. 4: Evaluae he wegh of each parcle z usng p( M z ) n (). 5: Take he observaon specfed by he parcle wh he larges wegh as he rackng resul. 6: The se of he emplaes s updaed usng he mehod n Secon Oupu he rackng resul a he curren frame and he new se of emplaes. 6. Mul-Objec Trackng wh Occluson Handlng Our algorhm for mul-objec rackng wh occluson handlng s an exenson of our sngle objec rackng algorhm. We ackle non-severe occlusons and severe (or complee) occlusons usng dfferen sraeges for defnng he observaon model. Ths s because when severe or complee occluson happens, here are no enough vsble pars of he objec o provde relable machng beween he observaon and he appearance model. The dynamcal model, whch s a pror model of he objec s moon, s unchanged durng he whole rackng process. The emplaes of each objec are updaed usng he sraegy n Secon 4.4. For smplcy, we focus on descrbng handlng of occlusons beween wo objecs. The mehod s easly exended o more objecs Observaon model for non-severely occluded objec When here s no severe occluson, we use he rval emplaes n he appearance model o consruc he observaon model for each occluded objec. When occluson occurs beween wo objecs A and B, he observaon of one objec may be dependen on A A B he sae of he oher objec. So, he observaon model of objec A can be represened by p( M z, z ). To represen he vsble pars of he objec, we defne a marx [ ] such ha k k 1,..., 1,.., k s se o 0, f he -h pxel s consdered o be occluded, oherwse se o 1. Le be he number of he elemens whose values are 1 n. In order o accuraely esmae he smlary beween he observaon and he appearance model, we defne he observaon model based on he vsble pars of an objec as follows: where ζ s a scalng facor, and We explan wo pons abou (3) and (4): A A B 1 k k k k k p( M z, z ) exp ( m H w ) (3) k 1 k k k k k A exp ( m H w ) ( ) d M k 1. (4) k k k k k k exp ( m H w ) d( ) m k 1 Gven an observaon M, he coeffcens of he emplaes are recovered usng he mul-feaure jon 13

14 sparse represenaon algorhm, and he denfer varables { } are deermned usng he recovered coeffcens of he rval emplaes. So, he denfer varables as well as he coeffcens of he emplaes can be reaed as funcons of M. Then, n (4) s a consan for a gven frame. We use o handle changes n he exen of he vsble pars of he objec n dfferen frames,.e., ake no accoun he area of he occluded poron of he objec. We propose wo ways for esmang he marx [ ]. The frs way s mplc, and does no nvolve k k 1,..., 1,.., reasonng abou he occluson relaons beween objecs. The second way depends on explc reasonng abou occluson relaons. In he mplc way, he marx [ ] s esmaed smply usng he values of he recovered k k 1,..., 1,.., feaure rval emplae coeffcens [ e ] k k 1,..., 1,..,, as hese values nclude he nformaon abou he poson and sze of he occluded pars of he objec. If a value n k k 1,..., [ e ] 1,.., s larger han a hreshold, s consdered ha he correspondng pxel s occluded and he correspondng elemen n k k 1,..., [ ] 1,.., s se o 0, oherwse se o 1. The mer of hs mplc way s ha s smple, and effecve n mos cases. Is lmaon s ha when objecs whch are easly confused wh one anoher also occlude one anoher, he occluson nformaon n he rval emplaes may no be very accurae. In he explc way, we use he nformaon n he rval emplaes o deermne objec occluson relaons. Gven he saes of wo objecs, he observaons correspondng o he saes are specfed. The opmal coeffcens of he emplaes and he rval emplaes of he wo objecs are esmaed, and he pxels n he overlapped regon beween he wo observaons are found. Whn he overlapped regon, he number of he pxels, for whch he coeffcens of he rval emplaes are larger han he predefned hreshold, s calculaed for each objec. I s deermned ha he objec wh he greaer number of such pxels s occluded by he oher objec. In he marx [ ] for he occluded objec, k k 1,..., 1,.., all he elemens whch correspond o he pxels whn he overlapped regon are se o 0. The mer of hs explc way s ha can correc he errors whch arse when objecs whch are easly confused wh one anoher also occlude one anoher. Is lmaon s ha depends on he sably of he occluson relaon reasonng. 6.. Mul-objec rackng wh non-severe occluson handlng Le nferrng z [( z ),( z ) ] be he jon sae of objecs A and B a frame. Gven he observaon M, J A T B T T J z wh occluson handlng s represened as a Bayesan poseror esmaon: where p( z, M ) p( M z, ) p( z z ) p( z, M ) dz (5) J J J J J J s he occluson relaon beween A and B. In our algorhm, he occluded and vsble pars of an objec are auomacally explored durng he sparse 14

15 reconsrucon based on he L,1 norm. By usng he observaon model n (3) o measure he smlary beween he observaon M warped by he objec sae J z and he emplaes, he Bayesan poseror nference for rackng objecs A and B can be smplfed n he followng way: p( z M ) p( M z ) p( z z ) p( z M ) d( z ). (6) J J J J J J Wh hs smplfed nference expresson, he poseror probably p z M can be approxmaed usng a se J ( ) A, B, of weghed parcles n he jon sae space of A and B. Gven a se of parcles { z, z } generaed from he A A B B ranson model p( z z ) and p( z z ), he wegh of each parcle s evaluaed usng he observaon 1 1 lkelhood p M z whch s esmaed as follows: J ( ) p M z p M z z p M z z. (7) J A A B B B A ( ) (, ) (, ) The opmal sae J z corresponds o he parcle whch has he maxmal wegh. If we esmae he weghs of all he parcles n he jon sae space for mul-objecs o fnd he opmal sae of he objecs, he me complexy s ON ( ), where N s he number of parcles per objec and s he number of objecs. When he number of he racked objecs s more han wo, he calculaon s oo me consumng. To ncrease he effcency, a cross eraon s adoped. Le z and A s, z be he esmaed opmal B s, saes for he objecs A and B a he s-h eraon. The opmal saes of objecs A and B a he nex eraon are esmaed usng he followng formulae: zˆ arg max p( M z, z ) p( M z, z ) (8) A A A B B B A, s1 A, s, s, s, s z zˆ arg max p( M zˆ, z ) p( M z, z ˆ ). (9) B A A B B B A, s1 B z, s1, s, s, s1 The cross-eraon of (8) and (9) connues unl convergence. The me complexy of he algorhm decreases o ON ( ) from ON ( ). Ths ensures ha when he number of objecs ncreases, he runme does no quckly ncrease Handlng of severe occlusons When an objec s severely occluded (for example more han 70% percen of he objec s occluded), here are no enough vsble pars of he objec o provde relable machng beween he observaon and he appearance model. If complee occluson occurs, s mpossble o evaluae he observaons usng he appearance model. In he case of severe or complee occlusons, objec moon nformaon n he prevous frames s more relable. In hese crcumsances, we consruc a new observaon model p( M z ), whch akes he moon velocy consran no consderaon, o esmae he poson of he objec. Le 1 1 v x x and v x x 1 be he moon vecors of objec n wo consecuve frames -1 and, where x s he poson of objec a frame. The change n objec moon veloces beween wo consecuve frames s usually very small. So, a parcle whch has smaller changes n moon velocy s gven a 15

16 larger wegh. Then, he lkelhood funcon s defned as follows: v p, 1 1 ( M z ) exp( )exp( v v ) (30) v where, 1 s he angle beween v 1 and v Exenson o more han wo objec rackng r 1 r For ( 3 ) objecs, he observaon model for each objec r s represened as p( M z, z,..., z,..., z ). I s esmaed usng he rgh-hand sde of (3). The key problem s o deermne he marx [ ] for k k 1,..., 1,.., objec r. In he mplc way, he marx [ ] s esmaed smply usng he values of he recovered feaure k k 1,..., 1,.., rval emplae coeffcens k k 1,..., [ e ] 1,..,, as n he case of wo objecs. In he explc way, for each par of objecs whch ncludes objec r, he nformaon n he rval emplaes s used o deermne her occluson relaons, and hen he marx [ ] for objec r s updaed f objec r s occluded. Afer all such pars of objecs are k k 1,..., 1,.., consdered, he marx [ ] J s deermned. Correspondngly, he observaon lkelhood p( M z ) k k 1,..., 1,.., becomes: J r 1 r ( ) p(,...,,..., ) r1 p M z M z z z (31) where J z s he jon sae of he objecs. The formula for esmang he opmal sae of objec r a he nex eraon becomes: r r 1 r1 r, s1 p r, s1, s1, s, s z r1 zˆ arg max ( M zˆ,.., zˆ, z,..., z ). (3) 6.5. Combnaon wh sngle objec rackng When here are no occlusons n he prevous frame, he exen of any occluson n he curren frame s no large and he sngle objec rackng algorhm s robus o rack he objecs. So, n order o reduce he runme, n he curren frame he mul-objec rackng algorhm s only used o rack he objecs whch occluded each oher n he prevous frame. In he nsances n whch objecs are relavely dense or easly confused wh one anoher, even f he objecs are no overlapped, her rackers may be absorbed ono he same objec. We heurscally handle hese nsances. When here s no occluson beween wo objecs A and B a he prevous frame and suddenly here s severe occluson beween hem a he curren frame, he appearance smlary beween objecs A and B a he prevous frame s checked. If he smlary s large, hen s deermned ha he wo rackers are absorbed ono one objec. For objec A or B, he parcle whose correspondng observaon s second o mos smlar o he appearance model of he objec s found. If he smlary beween he second o mos smlar observaon of objec A and he appearance model of objec A s larger han he smlary beween he second o mos smlar observaon of objec B and he appearance model of objec B, hen he rackng resul of objec A n he curren frame s changed o he second o mos smlar observaon of objec A, oherwse he rackng resul of objec B 16

17 s changed o he second-mos smlar observaon of objec B. 7. Expermens We esed our algorhms on varous publcly avalable sequences. Comparsons wh several sae-of-he-ar algorhms were made. Boh qualave analyss and quanave evaluaons were presened o show he effecveness of our algorhms. The parameers were se emprcally accordng o her properes or by referrng o her sengs n he prevous work. To manan he balance beween he reconsrucon and he sparseness, he regularzaon parameer λ n (9) was se o To manan he balance beween he accuracy of he coeffcens and he number of APG eraons, he sep sze α n Table 1 was se o 0.5. To manan he balance beween he accuracy and he speed of rackng, he number of he objec appearance emplaes n he emplae se was se o 60, and he number of parcles was se o 40. The hreshold ϵ n he wegh consran was se o 0.1 n order ha he coeffcens of mos of he emplaes n he emplae se can be drecly forced o 0. The forgeng facor δ for emplae updang was se o 0.95 whch s a value less han bu close o 1. The varances of he affne parameers were se o 5, 5, 0, 0, 0, and 0 whch were also used by he compeng algorhms n [18] and [5] o ensure far follow-up comparson. The hreshold for deermnng wheher he rackng resul was used o updae he emplae se was se o 70%, o ensure ha when mos pars of an objec were no badly affeced by nose or occluson, he rackng resul was used o updae he emplae se. The hreshold used o deermne wheher a pxel s occluded was se o a small value, The hreshold used o deermne wheher an objec appearance emplae s replaced wh he rackng resul was se o 0.3. The scalng facor η n () was se o 30. The scalng facor ζ n (3) was se o For color mage sequences, we used he hue, sauraon, nensy, he graden-based edge emplae, and he exure feaure obaned by Gabor flerng. For gray mage sequences, only he nensy, he edge emplae, and he exure feaure were adoped. The number of possble Gabor feaures s very hgh [47]. To ncrease he effcency of he rackng algorhm, one Gabor fler was chosen for one sequence. Gven a Gabor fler wh fxed parameer values, s feaure vecor has a dmenson equal o he number of pxels n a normalzed mage pach, afer appropraely dealng wh edge pxels. A frequency value was seleced emprcally from {0,, 4, 8, 16, 3}, and an orenaon value was seleced emprcally from {0, π/3, π/6, π/, 3π/4}. As n [5], he racker was nalzed manually n ha he frs objec appearance emplae was manually seleced from he frs frame. Four objec appearance emplaes were consruced by perurbng a few pxels n four possble drecons a he corner pons of he frs emplae a he frs frame. In he rackng afer he frs frame, he rackng resuls were added o he emplae se f he sze of he emplae se was less han a predefned value, oherwse he rackng resuls were used o updae he emplae se. In he followng, he expermenal resuls for sngle objec rackng are shown frs, and hen he resuls for mul-objec rackng wh occluson handlng are shown Sngle objec rackng We compared our algorhm frs wh he sngle feaure sparse represenaon racker, and hen wh several 17

18 sae-of-he-ar rackers Comparson wh sngle feaure sparse represenaon In order o llusrae he srengh of combnng mulple cues, we compared our algorhm wh he ypcal L 1 -regularzed sparse emplae-based racker [5] whch uses he sngle gray cue o consruc objec emplaes. Fve challengng sequences were used for hs comparson. For he frs hree vdeos, he frequency and orenaon of he Gabor fler were se o and π/, respecvely. For he las wo vdeos, hey were se o 4 and 3π/4. The rackng resuls of our algorhm are shown usng red boundng boxes, and he resuls of he L 1 -regularzed algorhm are shown usng green boundng boxes. In he frs sequence, a woman s occluded by wo men walkng across her. The appearances of wo men are very smlar o he woman. Ths makes harder o rack he woman. The rackng resuls are shown n Fg.. The frame numbers are 1, 1, 59, 79, and 85, respecvely. I s seen ha he L 1 -regularzed algorhm fals o rack he woman when here s severe occluson, however he rack s recovered when he background dsurbance s no srong. Our algorhm obans accurae rackng resuls hroughou he sequence. Fg.. The resuls of rackng a woman who s occluded by wo men walkng across her. In he second sequence, a woman s face s occluded by a book n varous frames. The resuls of rackng he face are shown n Fg. 3. The frame numbers are 3, 14, 53, 30, and 333, respecvely. I s seen ha he L 1 -regularzed algorhm loses rack when he occluson s severe, and he rack s recovered when he occluson s reduced. Our algorhm successfully racks he objec hroughou all he frames. Fg. 3. The resuls of rackng a face occluded by a book. In he hrd sequence, a car moves from he lef o he mddle of he scene. Is pose and scale change markedly when moves nearer o he camera and hen urns around. The resuls of rackng he car are shown n Fg. 4. The frame numbers are 0, 80, 173, 196, and 40, respecvely. I s seen ha he L 1 -regularzed racker gradually drfs away and evenually fals o rack he car. Our algorhm successfully racks he car hroughou he vdeo. Fg. 4. The resuls of rackng a car whch undergoes large changes n scale and appearance. 18

19 In he fourh sequence, a boa moves n a lake. Near o he boa n he mage plane, here are ree sumps and houses whch have colors smlar o he boa. The resuls of rackng he boa are shown n Fg. 5. The frame numbers are 50, 158, 59, 310, and 384, respecvely. I s seen ha he L 1 -regularzed racker s dsraced by he background and drfs off he rack n he second half of he vdeo. The reason for hs s ha he L 1 -regularzed algorhm only uses he pxel gray levels, and he gray levels of he boa are smlar o hose n he background. Our algorhm loses rack n some frames, bu he rack s hen recovered. Our algorhm, whch fuses mulple cues, obans much more accurae resuls han he L 1 -regularzed algorhm. Fg. 5. The resuls of rackng a boa n a lake. In he ffh sequence, a car moves agans a nosy background. As s a dark ngh, he color of he car s smlar o he background. The resuls of rackng he car are shown n Fg. 6. The frame numbers are 0, 80, 171, 11, and 44, respecvely. I s seen ha boh our algorhm and he L 1 -regularzed algorhm rack he car accuraely. In hs gray sequence, he gray levels alone supply suffcen nformaon for rackng. The shape feaure and he exure feaure are no dscrmnave. However, he fuson from he shape and exure cues does no degrade he rackng performance, because hey are gven low weghs. Fg. 6. The resuls of rackng a car runnng n a dark ngh. From he above rackng resuls, s concluded ha our algorhm s more robus han he L 1 -regularzed algorhm. In mos cases, he mul-feaure jon sparse represenaon-based rackng algorhm ouperforms he sparse represenaon-based rackng algorhm [5] ha only uses he pxel gray levels Comparson wh sae-of-he-ar algorhms Anoher fve challengng sequences were used o compare our algorhm wh he followng fve sae-of-he-ar algorhms: he L 1 -regularzed sparse emplae-based racker (LRST) [5] whch s a baselne for our algorhm he vsual decomposon racker (VDT) [18] whch uses he sparse PCA o fuse mulple cues he ncremenal subspace racker (IST) [4] whch s a ypcal generave appearance model-based racker he mulple nsance learnng (MIL) racker [5] whch s a ypcal dscrmnave model-based racker he onlne Adaboos racker (OBT) [15] whch s also a ypcal dscrmnave model-based racker. The compeng algorhms use smlar confguraons o our algorhm. For example, he same number of parcles s used for he parcle fler-based compeng algorhms (LRST, VDT, IST, and OBT). For he frs 19

20 sequence, he frequency and orenaon of he Gabor fler were se o 4 and 3π/4, respecvely. For he second sequence, hey were se o and π/. For he hrd sequence, hey were se o 8 and π/6. For he fourh sequence, hey were se o and π/. For he ffh sequence, hey were se o and π/6. In he followng, he resuls of our algorhm are shown usng red boundng boxes. The resuls of he MIL racker are shown usng lgh blue boundng boxes. The resuls of he L 1 -regularzed racker are shown usng green boundng boxes. The resuls of he vsual decomposon racker are shown usng yellow boundng boxes. The resuls of he onlne Adaboos racker are shown usng pnk boundng boxes. The resuls of he ncremenal subspace-based racker are shown usng blue boundng boxes. In he frs sequence, a woman walks from he rgh o he lef, whle parally occluded by cars. Some objecs n he background have appearances smlar o he woman. The resuls of rackng he woman are shown n Fg. 7. The frame numbers are 4, 41, 141, 06, and 309, respecvely. Boh our algorhm and he vsual decomposon racker successfully and accuraely rack he woman hrough all he frames. The MIL racker keeps he rack of he woman, bu he resuls are no accurae. The oher algorhms fal o rack he woman. Fg. 7. The resuls of rackng a woman who s parally occluded by cars. In he second sequence, a grl changes her head pose by urnng her head from full face o look away from he camera, undergong large changes n appearance. In he mddle of he vdeo, he grl s face s severely or even almos compleely occluded by a man s head. The resuls of rackng he grl s head are shown n Fg. 8. The frame numbers are 56, 11, 30, 37, and 438, respecvely. Our algorhm, he MIL racker, and he vsual decomposon racker successfully rack he grl s head hrough large appearance changes and occlusons. The ncremenal subspace racker keeps rack a he begnnng, bu gradually loses he rack when he changes n appearance become large. The oher algorhms canno accuraely rack he grl s head. Fg. 8. The resuls of rackng a grl s head whch undergoes large changes n appearance and severe occlusons. In he hrd sequence, a pedesran wh a small apparen sze walks from he rgh o he lef. He s nally seen from he sde and hen from he back, and hus undergoes gradual bu evenually large changes n appearance. The oher pedesrans, who have appearances smlar o hm, occlude hm or are occluded by hm from me o me. The resuls of rackng hs pedesran are shown n Fg. 9. The frame numbers are 17, 88, 305, 373, and 455, respecvely. I s seen ha only he L 1 -regularzed racker and he onlne Adaboos racker fal o 0

21 rack he pedesran. All he oher four algorhms keep he rack of he pedesran. However, n some frames he resuls of he MIL racker and he ncremenal subspace racker are no accurae. Fg. 9. The resuls of rackng a pedesran wh a small apparen sze and gradual pose changes. In he fourh sequence, a pedesran moves n a nosy and cluered oudoor envronmen. There are background elemens, such as cars and rees, whch have colors smlar o he pedesran. The resuls are shown n Fg. 10. The frame numbers are 6, 8, 5, 78, and 93, respecvely. I s seen ha he ncremenal subspace racker and he vsual decomposon racker fal o rack he pedesran even a he begnnng. The MIL racker, he L 1 -regularzed racker, and he onlne Adaboos racker keep he rack n more han half of he sequence, bu fnally ransfer he rackng o a car n he background. Our algorhm successfully racks he pedesran hrough all he frames. Fg. 10. The resuls of rackng an oudoor pedesran n a nosy and cluered envronmen. In he ffh sequence, a deer runs n a nosy background. Oher deer, whch are very smlar n appearance, pass by. The resuls of rackng he head of he deer are shown n Fg. 11. The frame numbers are 15, 9, 35, 53, and 66, respecvely. I s seen ha he vsual decomposon racker and he ncremenal subspace racker fal o rack he head a he begnnng of he sequence. The MIL racker, he L 1 -regularzed racker, he onlne Adaboos racker, and our algorhm drf off rack n some frames due o background dsurbances, bu he rack s recovered when he background dsurbances cease. Fg. 11. The resuls of rackng he head of a deer agans a cluered background Quanave evaluaons Quanave evaluaons were made for he sx algorhms usng he above fve vdeos. In each frame, four benchmark pons were manually labeled correspondng o he four corners of a boundng box whch conans he racked objec. We calculaed he roo mean square error (RMSE) of he four corner pons n he boundng box beween he rackng resuls and he ground ruhs. Fgs. 1, 13, 14, 15, and 16 show he RMSE curves of our algorhm and he fve compeng algorhms for he fve sequences respecvely, where he x-coordnae s he 1

22 frame number and he y-coordnae s he RMSE n each frame. Table 3 lss he mean of he RMSEs for all he frames n each of he fve sequences. I s seen ha he performance of our algorhm s comparable wh ha of he vsual decomposon racker and ha ouperforms he oher four algorhms. Fg. 1. The RMSE curves of he sx algorhms for sequence 1. Fg. 13. The RMSE curves of he sx algorhms for sequence. Fg. 14. The RMSE curves of he sx algorhms for sequence 3. Fg. 15. The RMSE curves of he sx algorhms for sequence 4. Fg. 16. The RMSE curves of he sx algorhms for sequence 5. Table 3: The mean of he RMSEs for all he frames n each of he fve sequences Trackers Onlne L MIL-based 1 Vsual Incremenal Sequences Adaboos regularzed decomposon subspace Our Vdeo Vdeo Vdeo Vdeo Vdeo From he above rackng resuls, we draw he followng concluson. The algorhms fusng mulple cues,.e. he vsual decomposon racker and our algorhm, are more accurae han he algorhms whch only use a

23 sngle cue. Our algorhm can effecvely fuse mulple cues n a sparse represenaon for vsual rackng. As he MIL racker deals effecvely wh he drf problem by usng mulple nsances, ouperforms he oher hree algorhms whch only use he sngle cue. 7.. Mul-objec rackng In order o valdae he effecveness of our algorhm for rackng mulple objecs, fve dfferen vdeos, wo conanng faces, wo conanng pedesrans, and one conanng vehcles, were used o compare our algorhm wh hree sae-of-ar algorhms. For he frs four vdeos, he frequency and orenaon of he Gabor fler were se o and π/, respecvely. For he las vdeo, hey were se o 16 and π/3. The frs example corresponds o he second vdeo n Secon In he mddle of he sequence, he man s face gradually dsappears from he scene and hen appears n he scene agan. The ncremenal subspace-based algorhm n [4] s a ypcal sngle objec rackng algorhm, wdely used for comparsons. We exended he ncremenal subspace-based algorhm o rack mul-objecs by rackng mul-objecs ndependenly, whou any occluson handlng. We compared our mul-objec rackng algorhm wh he ncremenal subspace algorhm [4] o show he necessy of occluson handlng. The resuls of rackng he wo faces are shown n Fg. 17, where he frs row shows he resuls of our algorhm and he second row shows he resuls of he ncremenal subspace algorhm. The frame numbers are, 15,, 5, and 8, respecvely. I s seen ha he ncremenal subspace-based racker does no effecvely handle he occlusons produced by he dsappearance of he face of he man. The reason for hs s ha, for he ncremenal subspace algorhm, he occluded pars of an objec are also used for machng, bu he nvsble pars canno be reconsruced by he subspace based-machng. On he conrary, our observaon model adops he vsble pars for machng, whle gnorng he occluded pars. Ths s he key of success of our algorhm n rackng he wo faces. So, occluson handlng s necessary for rackng mul-objecs durng occlusons. Fg. 17. The resuls of rackng he faces of a grl and a man: (a) Our algorhm; (b) The ncremenal subspace algorhm In he second example, he faces of wo men occlude each oher and undergo changes n appearance. We compared our mul-objec rackng algorhm wh he followng wo sae-of-he-ar mul-objec rackng algorhms wh occluson handlng: Yang s algorhm [36], whch uses a game-heorec analyss o mplcly handle occlusons Zhang s algorhm [44], whch use speces compeon o decompose he rackng of mul-objecs wh 3

24 occlusons no he rackng of a se of un-occluded objecs. The resuls of rackng he wo faces are shown n Fg. 18, where he frs row shows he resuls of our algorhm, he second row shows he resuls of Zhang s algorhm, and he hrd row shows he resuls of Yang s algorhm. The frame numbers are 8, 14, 17, 04, and 37, respecvely. I s seen ha Zhang s algorhm drfs off he rack for he occluded objec o some exen. Yang s algorhm fals o rack he occluded objec durng and afer he occluson. Our algorhm effecvely racks boh he faces. The changes n he appearances of he faces are effecvely handled usng our emplae updang mechansm. Fg. 18. The resuls of rackng wo men s faces whch occlude each oher and undergo changes n appearance: (a) Our algorhm; (b) Zhang s algorhm; (c) Yang s algorhm. (a) (b) (c) Fg. 19. The resuls for rackng hree pedesrans under occlusons: (a) Our algorhm; (b) Zhang s algorhm; (c) Yang s algorhm. The vdeo n he hrd example s from he open PETS004 daabase. In hs vdeo, here are muual occlusons beween hree pedesrans. Two men urn around o face he camera and her appearances change. A woman urns around o face away from he camera. Fg. 19 llusraes resuls of our algorhm, Zhang s algorhm, and Yang s algorhm for rackng all hree pedesrans. The frame numbers are 5, 5, 45, 65, and 88, 4

25 respecvely. I s seen ha our mehod successfully racks all he pedesrans. The changes n he appearances of he pedesrans are successfully ackled hrough our emplae updang sraegy. Occlusons are effecvely handled. However, boh he compeng algorhms ransfer he rackng from he man marked by a green wndow o he women marked by a red wndow. Boh he algorhms lose he rack of he man marked by a blue wndow. A quanave evaluaon of he rackng accuracy was conduced o furher demonsrae beer performance of our mul-objec rackng algorhm. The evaluaon s comprsed of he followng wo aspecs: he number of successfully racked frames (rackng s consdered o be successful f he esmaed boundng box s cener s n he ground ruh box) he RMSE (roo mean square error) beween he esmaed poson and he ground ruh. Table 4 shows he resuls of quanave comparsons beween our algorhm, Zhang s algorhm, and Yang s algorhm for rackng he hree pedesrans (persons A, B and C) n he hrd sequence. I s seen ha he mean rackng error of Yang s algorhm for person A s very large. Ths s because Yang s algorhm quckly loses he rack of hs person. I s apparen ha he resuls of our algorhm are more accurae han he resuls of Zhang s algorhm and Yang s algorhm. Table 4: Quanave comparsons beween our algorhm, Zhang s algorhm, and Yang s algorhm for he hrd sequence Algorhms Our algorhm Zhang s algorhm Yang s algorhm Person A (Blue) 89/90 75/90 5/90 Successfully Person B (Green) 90/90 1/90 18/90 racked frames Person C (Red) 90/90 86/90 40/90 Person A (Blue) RMSE Person B (Green) Person C (Red) (a) (b) (c) Fg. 0. The rackng resuls for he fourh sequence: (a) Our algorhm; (b) Zhang s algorhm; (c) Yang s algorhm. In he fourh example, hree pedesrans (persons A, B, and C) occludng each oher are racked. The vdeo s from he PETS daase n 006. Fg. 0 llusraes he rackng resuls of our algorhm, Zhang s algorhm, and Yang s algorhm, where person A s racked usng a blue wndow, person B s racked usng a green wndow, and person C s racked usng a red wndow. The frame numbers are 10, 6, 38, 50, and 63, respecvely. I s 5

26 seen ha boh Zhang s algorhm and Yang s algorhm fal o accuraely rack persons B and C durng occlusons. The compeng algorhms do no deal effecvely wh he occlusons beween persons B and C. Our algorhm successfully racks all he hree pedesrans by effecvely handlng he occlusons beween he hree pedesrans. Table 5 shows quanave resuls for our algorhm, Zhang s algorhm, and Yang s algorhm for he fourh sequence. I s apparen ha he resuls of our algorhm are he mos accurae. Table 5: Quanave resuls of our algorhm, Zhang s algorhm, and Yang s algorhm for he fourh sequence Algorhms Our algorhm Zhang s algorhm Yang s algorhm Person A 89/89 89/89 89/89 Successfully Person B 89/89 76/89 57/89 racked frames Person C 89/89 58/89 89/89 Person A RMSE Person B Person C For he fourh mul-objec rackng example, wo more pedesrans were racked, besdes he hree pedesrans who were racked n he Fg. 0. The explc occluson handlng was used. The resuls are shown n Fg. 1. I s seen ha he fve pedesrans are correcly racked. Fg. 1. The resuls of our algorhm for rackng fve pedesrans. In he ffh example, mul-vehcles were racked n a crowded raffc nersecon scene. Fg. shows he resuls of our algorhm for rackng egh vehcles n he scene, where wo pars of smlar vehcles, whch are easly confused wh each oher, move close o each oher. The frame numbers are 3475, 3500, 355, 3550, and 3600, respecvely. I s seen ha all he egh vehcles are successfully racked. Fg.. The resuls of our algorhm for rackng egh vehcles n a dense raffc nersecon scene Compuaonal complexy and runme The compuaonal complexy for our sngle objec rackng algorhm s O( n N), where s he number of eraons requred o oban he mul-feaure jon sparse represenaon for an observaon. The compuaonal complexy for our mul-objec rackng algorhm s O( nn ). As measured on an Inel Core (3.4GHz/L3) compuer, he runme of our sngle objec rackng algorhm for each frame n all he examples s less han 0.8 seconds and he runme of our mul-objec rackng algorhm s less han 5 seconds per frame n he case of occlusons. As he number of objecs ncreases, he runme does no quckly ncrease. I s shown ha he compuaonal complexy of he proposed mehod s no 6

27 very hgh. Currenly, our algorhm s unable o work n real-me. We expec furher research on hs problem. Alhough mul-feaure fuson s appled by our algorhm, our sngle objec rackng algorhm s faser han he L 1 -regularzed sparse emplae-based racker (LRST) because he APG algorhm s used n our algorhm. The speed of our sngle objec rackng algorhm s comparable o he speed of he VDT [18], bu slower han he speeds of he IST, he MIL racker, and he OBT. The speed of our mul-objec rackng algorhm s comparable o he speed of Yang s algorhm, bu slower han he speed of Zhang s algorhm. 8. Concluson In hs paper, we have proposed a new rackng algorhm based on a mul-feaure jon sparse represenaon. In our algorhm, mulple cues have been successfully fused ogeher n he sparse represenaon framework. The varance rao has been nroduced o adap he weghs of dfferen feaures. The emplae se has been updaed adapvely usng he rackng resuls. By usng a sparse wegh consran, a large number of emplaes can be kep n he emplae se. We have furher proposed an algorhm for rackng mul-objecs under occlusons usng a mul-feaure sparse reconsrucon. The machng beween he observaons and he emplaes s based on he vsble pars of he occluded objecs. The expermenal resuls have valdaed he effecveness of boh our sngle objec rackng algorhm and our mul-objec rackng algorhm. The man lmaons of our mehod are ha many parameers are se emprcally and he semanc correcons beween he dfferen feaures are no modeled. References 1. L. Cazzan, M. Gupa, and A. oppal, Generave Models for Smlary-Based Classfcaon, Paern Recognon, vol. 41, no. 7, pp , July M. m, Dscrmnave Sem-supervsed Learnng of Dynamcal Sysems for Moon Esmaon, Paern Recognon, vol. 44, no , pp , Oc B.S. Manjunah, J.R. Ohm, V. Vasudevan, and A. Yamada, Color and Texure Descrpors, IEEE Trans. on Crcus and Sysems for Vdeo Technology, vol. 11, no. 6, pp , June D.A. Ross, J. Lm, R. Ln, and M. Yang, Incremenal Learnng for Robus Vsual Trackng, Inernaonal Journal of Compuer Vson, vol. 77, no., pp , May X. Me and H. Lng, Robus Vsual Trackng and Vehcle Classfcaon va Sparse Represenaon, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 33, no. 11, pp , Nov J. Wrgh, A. Yang, A. Ganesh, S. Sasry, and Y. Ma, Robus Face Recognon va Sparse Represenaon, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 31, no., pp. 10-7, Feb A. Jepson, D. Flee, and T. El-Maragh, Robus Onlne Appearance Models for Vsual Trackng, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 5, no. 10, pp , Oc S. Zhou, R. Chellappa, and B. Moghaddam, Vsual Trackng and Recognon Usng Appearance-Adapve Models n Parcle Flers, IEEE Trans. on Image Processng, vol. 13, no. 11, pp , Nov T. Yu and Y. Wu, Dfferenal Trackng Based on Spaal Appearance Model, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, pp , H. Wang, D. Suer,. Schndler, and C. Shen, Adapve Objec Trackng Based on an Effecve Appearance Fler, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 9, no. 9, pp , Sep Y. Wu, J. Cheng, J. Wang, and H. Lu, Real-Tme Vsual Trackng va Incremenal Covarance Tensor Learnng, n Proc. of IEEE Inernaonal Conference on Compuer Vson, pp , F. Porkl, O. Tuzel, and P. Meer, Covarance Trackng Usng Model Updae Based on Le Algebra, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, pp , R. Collns, Y. Lu, and M. Leordeanu, Onlne Selecon of Dscrmnave Trackng Feaures, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 7, no. 10, pp , Oc S. Avdan, Ensemble Trackng, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 9, no., pp , Feb

28 15. H. Grabner, M. Grabner, and H. Bschof, Real-Tme Trackng va On-lne Boosng, n Proc. of Brsh Machne Vson Conference, pp , H. Grabner, C. Lesner, and H. Bschof, Sem-Supervsed On-lne Boosng for Robus Trackng, n Proc. of European Conference on Compuer Vson, pp. 1-8, A. Saffar, C. Lesner, J. Sanner, M. Godec, and H. Bschof, On-lne Random Foress, n Proc. of IEEE Inernaonal Conference on Compuer Vson Workshops, pp , J. won and.m. Lee, Vsual Trackng Decomposon, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, pp , S.J. m,. oh, M. Lusg, S. Boyd, and D. Gornevsky, A Mehod for Large-Scale 1-Regularzed Leas Squares, IEEE Journal on Seleced Topcs n Sgnal Processng, vol. 1, pp , X.T. Yuan and S. Yan, Vsual Classfcaon wh Mul-Task Jon Sparse Represenaon, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, pp , June J. Zhang, A Probablsc Framework for Mul-Task Learnng, Techncal Repor, Yu, T. Zhang, and Y. Gong, Nonlnear Learnng Usng Local Coordnae Codng, n Proc. of Annual Conference on Neural Informaon Processng Sysems, pp. 1-9, P. Tseng, On Acceleraed Proxmal Graden Mehods for Convex-Concave Opmzaon, SIAM Journal of Opmzaon, May 008. hp:// 4. M. Isard and A. Blake, Condensaon: Condonal Densy Propagaon for Vsual Trackng, Inernaonal Journal of Compuer Vson, vol. 9, no. 1, pp. 5-8, B. Babenko, M.-H. Yang, and S. Belonge, Robus Objec Trackng wh Onlne Mulple Insance Learnng, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 33, no. 8, pp , D.L. Donoho and X. Huo, Uncerany Prncples and Ideal Aomc Decomposon, IEEE Trans. on Informaon Theory, vol. 47, no. 7, pp , Nov B. Lu, L. Yang, J. Huang, P. Meer, L. Gong, and C. ulkowsk, Robus and Fas Collaborave Trackng wh Two Sage Sparse Opmzaon, n Proc. of European Conference on Compuer Vson, pp , F. Chen, Q. Wang, S. Wang, W. Zhang, and W. Xu, Objec Trackng va Appearance Modelng and Sparse Represenaon, Image and Vson Compung, vol. 9, no. 11, pp , Q. Wang, F. Chen, W. Xu, and M.-H. Yang, Onlne Dscrmnave Objec Trackng wh Local Sparse Represenaon, n Proc. of IEEE Workshop on Applcaons of Compuer Vson, pp , C. Rasmussen and G. Hager, Probablsc Daa Assocaon Mehods for Trackng Complex Vsual Objecs, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 3, no. 6, pp , June A. Elgammal and L. Davs, Probablsc Framework for Segmenng People under Occluson, n Proc. of Inernaonal Conference on Compuer Vson, vol., pp , Y. Wu, T. Yu, and G. Hua, Trackng Appearances wh Occlusons, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, Madson, vol. 1, pp , June E. Sudderh, M. Mandel, W. Freeman, and A. Wllsky, Dsrbued Occluson Reasonng for Trackng wh Nonparamerc Belef Propagaon, n Proc. of Annual Conference on Neural Informaon Processng Sysems, pp , J. MacCormck and A. Blake, A Probablsc Excluson Prncple for Trackng Mulple Objecs, Inernaonal Journal of Compuer Vson, vol. 39, no. 1, pp , H. Nguyen, Q. J, and A. Smeulders, Spao-Temporal Conex for Robus Mularge Trackng, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 9, no. 1, pp. 5-64, Jan M. Yang, T. Yu, and Y. Wu, Game-Theorec Mulple Targe Trackng, n Proc. of IEEE Inernaonal Conference on Compuer Vson, pp. 1-8, Nummaro, E. oller-meer, and L.V. Gool, An Adapve Color-Based Parcle Fler, Image and Vson Compung, vol. 1, no. 1, pp , Jan P. Perez, C. Hue, J. Vermaak, and M. Gangne, Color-Based Probablsc Trackng, In Proc. of European Conference on Compuer Vson, vol. 1, pp , Y. L On Incremenal and Robus Subspace Learnng, Paern Recognon, vol. 37, no. 7, pp , H. Lm, V.I. Moraru, O.I. Camps, and M. Sznaer, Dynamc Appearance Modelng for Human Trackng, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, vol. 1, pp , J. Ho,. Lee, M. Yang, and D. regman, Vsual Trackng Usng Learned Lnear Subspaces, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, vol. 1, pp , S. Avdan, Suppor Vecor Trackng, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 6, no. 8, pp , Aug O. Tuzel, F. Porkl, and P. Meer, Human Deecon va Classfcaon on Remannan Manfolds, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, pp. 1-8, June X. Zhang, W. Hu, W. Qu, and S. Maybank, Mulple Objec Trackng va Speces-Based Parcle Swarm Opmzaon, IEEE Trans. on Crcus and Sysems for Vdeo Technology, vol. 0, no. 11, pp , Nov

29 45. X. Me, H. Lng, Y. Wu, E. Blasch, and L. Ba, Mnmum Error Bounded Effcen l1 Tracker wh Occluson Deecon, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, pp , June J.B. Tenenbaum, V. de Slva, and J.C. Langford, A Global Geomerc Framework for Nonlnear Dmensonaly Reducon, Scence, vol. 90, no. 5550, pp , Dec Manjunah and W. Ma, Texure Feaures for Browsng and Rereval of Image Daa, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 18, no. 8, pp , Aug A.A. Bu and R.T. Collns, Mul-arge Trackng by Larangan Relaxaon o Mn-Cos Nework Flow, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, Oral presenaon, June J. Lu, P. Carr, R.T. Collns, and Y. Lu, Trackng Spors Players wh Conex-Condoned Moon Models, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, Oral presenaon, June R.T. Collns, Mularge Daa Assocaon wh Hgher-Order Moon Models, n Proc. of IEEE Conference on Compuer Vson and Paern Recognon, pp , June Zhang, L. Zhang, and M.-H. Yang, Real-Tme Compressve Trackng, n Proc. of European Conference on Compuer Vson, pp , F. Ne, H. Huang, X. Ca, and C. Dng, Effcen and Robus Feaure Selecon va Jon l,1 -Norms Mnmzaon, n Proc. of Advances n Neural Informaon Processng Sysems, pp , J. Lu, S. J, and J. Ye, Mul-Task Feaure Learnng va Effcen l,1 -Norm Mnmzaon, n Proc. of Conference on Uncerany n Arfcal Inellgence, pp , 009. Acknowledgmens Ths work s parly suppored by he 973 basc research program of Chna (Gran No. 014CB349303), he Naonal 863 Hgh-Tech R&D Program of Chna (Gran No. 01AA01504), and he Naural Scence Foundaon of Bejng (Gran No ). Wemng Hu receved he Ph.D. degree from he deparmen of compuer scence and engneerng, Zhejang Unversy n From Aprl 1998 o March 000, he was a posdocoral research fellow wh he Insue of Compuer Scence and Technology, Pekng Unversy. Now he s a professor n he Insue of Auomaon, Chnese Academy of Scences. Hs research neress are n vsual moon analyss, recognon of web objeconable nformaon, and nework nruson deecon. We L receved hs B.Sc degree n auomaon from Zhejang Unversy, Chna, n 006. He receved he Ph.D. degree a he Naonal Laboraory of Paern Recognon, Insue of Auomaon, Chnese Academy of Scences, Chna, n 01. Now, he s workng n Albaba Group Company. Hs research neress nclude compuer vson and paern recognon. Xaoqn Zhang receved he B.Sc degree n elecronc nformaon scence and echnology from Cenral Souh Unversy, Chna, n 005 and PhD degree from he Insue of Auomaon, Chnese Academy of Scences, Chna, n 010. He s currenly a lecure n Wenzhou Unversy, Chna. Hs research neress are n vsual rackng, moon analyss, and acon recognon. Sephen Maybank receved a BA n Mahemacs from ng's college Cambrdge n 1976 and a PhD n compuer scence from Brkbeck college, Unversy of London n Now he s a professor n he School of Compuer Scence and Informaon Sysems, Brkbeck College. Hs research neress nclude he geomery of mulple mages, camera calbraon, vsual survellance ec. 9