A Bayesian algorithm for tracking multiple moving objects in outdoor surveillance video

Transcription

1 A Bayesan algorhm for racng mulple movng obecs n oudoor survellance vdeo Manunah Narayana Unversy of Kansas Lawrence, Kansas manu@u.edu Absrac Relable racng of mulple movng obecs n vdes an neresng challenge, made dffcul n real-world vdeo by varous sources of nose and uncerany. We propose a Bayesan approach o fnd correspondences beween movng obecs over frames. By usng color values and poson nformaon of he movng obecs as observaons, we probablscally assgn racs o hose obecs. We allow for racs o be los and hen recovered when hey resurface. The probablsc assgnmen mehod, along wh he ably o recover los racs, adds robusness o he racng sysem. We presen resuls ha show ha he Bayesan mehod performs well n dffcul racng cases and compare he probablsc resuls o a Eucldean dsance based mehod. 1. Inroducon The ably o rac a parcular obec or obecs n successve frames s an mporan sep n obec racng and classfcaon applcaons. In many racng applcaons, wheher n he vsble or non-vsble specrum, mulple arge obecs are o be analyzed a each me sep. Fndng relable correspondences beween obecs n one me sep o obecs n he nex s a nonrval as gven he non-deermnsc naure of he subecs, her moon, and he mage capure process self. I can be a dffcul as n some cases, especally n he presence of nosy measuremens from he sensors, occluson from obecs n he feld of vew, and changes n orenaon and drecon of movemen of he obecs. In he wor dscussed n hs paper, we acle he ssue of real-me mulple obec correspondence n oudoor saonary camera scenes. Whle our applcaon s n he vsble specrum and uses he color nformaon from obecs, he mehod could be exended o he non-vsble specrum as well by usng approprae specral nformaon. We begn wh he deecon of movng pxels usng subracon from a movng average bacground model. The movng pxels are clusered no regons, whch we refer o as blobs, so ha pxels belongng o a Donna Haveramp Unversy of Kansas Lawrence, Kansas dhaveram@u.edu sngle obec are grouped ogeher. Once movng regons have been denfed, he nex as s o generae racs of hese obecs over successve frames. Ths s essenally a correspondence as, he goal of whch s o fnd a mach for each blob or obec n he prevous frame o one of he blobs or obecs n he curren frame. We use Bayesan nference based on he color and poson nformaon of he movng obecs. When he nformaon from a frame does no suppor he presence of an obec, we allow for he correspondng rac o be declared los. If he obec resurfaces n subsequen frames, he sysem reassgns he rac o he obec. Thus, he Bayesan mehod s able o handle occluson, dsappearance of obecs, sudden changes n obec velocy, and changes n obec color profle and sze. 2. Prevous Wor Snce movng obecs are ypcally he prmary source of nformaon n survellance vdeo, mos mehods focus on he deecon of such obecs. Common mehods for moon segmenaon nclude average bacground modelng [1], Gaussan bacground modelng [2], and opcal flow mehods [3]-[5]. Smple bacground subracon s a wdely used mehod for he deecon of movng obecs. However, he bacground mage s no always nown and ofen needs o be auomacally generaed by he sysem [1]. In our mehod, we use an average bacground model. Tradonally, Bayesan racng mehods nvolve generang a reference model of he obec beng raced, hen loong for he bes mach for he model n each frame by usng predcve approaches le Kalman flers[6][7], and samplng-based approaches le Condensaon[8] and he Jon Probablsc Daa Assocaon Fler(JPDAF)[7][9]. Insead, we perform moon segmenaon o deec obecs for racng. Ths leads o a fundamenal dfference n he way poseror probably s calculaed for he JPDAF versus our mehod. In JPDAF, he poseror probably of assocang each observaon o each rac s calculaed usng he Bayesan formula on he rac model equaons. In oher words, JPDAF calculaes he probably of blob-o-rac 1

2 assocaon, gven he rac models. We approach he assgnmen dfferenly by calculang he probably of blob-o-rac assocaon, gven he prevous frame s blob nformaon. Thus, we do no need o manan rac models for each rac. We probablscally assgn racs o blobs by usng he color values and poson nformaon of he segmened blobs as observaons n he Bayes formula. 3. Moon Segmenaon We use a movng average model where he average nensy value for each pxel over a wndow of N frames (ypcally 150) s regarded as he bacground model. Movng obecs do no conrbue much o he average nensy value [10]; hus, he model becomes a reasonable esmae of he bacground. If I ( y, s he nensy level a coordnaes Y=y (column) and X=x (row) of he h frame n he sequence and bg ( y, s he average bacground model value a (y, for frame, hen = + ( N / 2) bg ( y, 1/ N I ( y, (1) = ( N / 2) The segmened oupu s: seg ( y, = 1, f bg ( y, I ( y, > T (2) = 0, oherwse where T s a predeermned hreshold, ypcally 10. Followng segmenaon, regon growng s used o locae each movng obec. An envelope s esablshed around each obec of neres by use of morphologcal eroson and dlaon flers. Addonal nose s flered by gnorng blobs less han S pxels (ypcally 20) n sze. 4. Obec Tracng We perform obec racng by fndng correspondences beween blobs n he prevous frame and blobs n he curren frame. In he case of a noseless envronmen and perfec segmenaon of blobs, he as s sraghforward because a one-o-one correspondence can be usually be found for a blob n wo successve frames. In realy, however, here s ofen a grea deal of uncerany n he fnal segmenaon. Falure o deec small ye vald blobs due o her sze, occluson of vald blobs by oher obecs n he scene, change n obec orenaon, enry (or ex) of obecs n he scene, and camera er are he man reasons for uncerany. These facors mae correspondence a dffcul problem o solve, parcularly for vdeos ha are sho oudoors. Our goal s o develop a correspondence scheme ha s robus o hese sources of uncerany General racng approach For each moon blob n he frs frame, a new rac s creaed. In subsequen frames, machng blobs are sough for each rac n he prevous frame. Mached racs are updaed wh poson and color nformaon from he machng blob. We allow racs o be declared los f any of he facors menoned above causes he algorhm o fal o deec a suable mach for a rac. The rac s no deleed bu s ep alve for a few frames n hopes ha a mach wll be found. A rac s deleed only f a suable mach s no found for L (ypcally 5) consecuve frames. If here s any blob n he new frame ha canno be mached o an exsng rac, hen a new rac s generaed. A ls of currenly acve racs s mananed for each me sep/frame The correspondence algorhm Commonly, a dsance-based mach marx s used o solve he correspondence problem, le n [10]. To faclae rac correspondence, we desgned a Bayesan nference mehod o fnd machng blobs n consecuve frames. We also calculae he probably ha a rac s los n he curren frame. The basc dea behnd our approach s ha gven an obec (blob) n he curren frame, here s some probably ha a blob of smlar color and poson wll be observed n he nex frame. Usng Bayes formula, we can fnd he probably of a blob belongng o a rac from he prevous frame, gven he blob s color and poson observaons. The sub-problem can be formulaed hus: T =,,..., s he se of racs n If { } u frame (-1) and O { o o,... o } = 1, 2 s he se of v movng blobs n frame, wha s he bes possble mach beween he racs and blobs? Here, s he h rac, 1 u -1 s he oal number of racs n frame (-1), s he h blob found n frame and v s he oal number of movng blobs n frame. In oher words, wha s he probably of assgnmen of 1 rac o blob for all and? An ncrease n he 1 probably of assgnmen of rac o a blob should auomacally reduce he probably of assgnmen 1 of rac o oher blobs o, where m. If no 1 m suable mach for s found, hen he probably ha s los should be made hgh. By loong a he observaons for all elemens 1 1 T and o O, we updae a belef marx 2

3 whch conans he probably of mach for each rac o each canddae blob. The probablsc newor ha s used o calculae hs belef marx s gven n Fg Mehod and updang formulae In he Bayes newor, R, G, and B are he color observaons. R s he dfference beween mean Red values, G s he dfference beween mean Green values, and B s he dfference beween mean Blue values of he blob and 1 rac. Togeher R, G, and B form he color observaon c: { R = r, G = g B b} c =, = Y and X are he dfferences beween poson of blob and predced poson of rac 1 n y and x coordnaes, respecvely. A smple esmae based on curren velocy and curren poson of rac s used as he predced poson for he rac. Togeher, Y and X form he poson observaon d: { Y = y X x} d =, = Dependng on he sae of he color and poson observaon of blob, he poseror probably of a 1 rac beng assgned o blob o s calculaed. Though R, G and B values are no ndependen n naure, for smplcy, we assume ndependence among R, G, and B observaons. The assumpon of ndependence does no affec he resuls adversely. Thus, p ( c) = R = r) G = g) B = b) (3) Smlarly, he Y and X ndependence assumpon leads o p ( d) = Y = y) X = (4) For ease of noaon, we now refer o he even rac assocaed wh blob rac 1 1 as Assgn. The even ha s no assocaed wh curren canddae blob s called NoAssgn. NoAssgn mples ha rac has been assocaed wh anoher blob m ) or has been declared los. o m 1 (where From (3) and (4), we can say c = p ( R = r p ( G = g (5) p ( B = b p ( d = Y = y X = x (6) Also, we se p ( c No = 0.1 (7) p ( d No = 0.1 (8) These are probables ha a gven observaon s seen n 1 blob whou a correspondng rac exsng n he prevous frame. I may be noed ha hese numbers are small, whch reflecs ha he probably of fndng a blob of a gven color and poson whou a correspondng rac n he prevous frame s low. The pror probably for he assgnmen s: 1 o ) = = 1/( v + 1) (9) where v s he number of movng blobs n frame. 1 Noe: The facor ( v +1) mples ha he rac has an equal pror probably of beng assocaed wh any of he v blobs of frame or of beng declared as los. Fg. 1. Probablsc newor for rac assgnmen 3

4 c) s gven by: c) = c (10) + c NoAssgn ) NoAssgn ) Smlarly, d) s gven by: d) = d (11) + d No No where NoAssgn ) = 1 The poseror probably of a rac beng assgned o a blob gven he blob s color observaon s, by Bayes formula, p ( Assgn c) = c (12) c) Smlarly, he poseror probably of a rac beng assgned o a blob gven he blob s poson observaon s p ( Assgn d) = d (13) d) Belef Marx and updae 1 The Belef Marx s a u rows by ( v +1) columns marx. The Belef Marx conans he probables of each rac beng assgned o each blob and also of beng declared as los, hence he ( v +1) columns. Denoed by BP, Belef s he probably of assgnmen of rac o blob ;.e., Assgn ). For each frame, he followng procedure s followed o updae he belef marx: 1.Inalze he Belef Marx o 1 /( v + 1) 1 BP (0) = 1/( v + 1), {1,2,3, K, u } (14) {1,2,3, K, v } Where u -1 s he number of racs n frame -1, v s he number of movng blobs n frame and BP (0) s he nal value of Belef Marx a row and column. 2. Ierae hrough all racs and blobs and: 1 1 For each rac For each blob T o O BP (n) a. Calculae based on he color observaon of where BP (n) sands for he value of he Belef Marx a row and column due o observaon from he n h blob (whch s ). Noe ha he value of n s always equal o he value of. b. Normalze or updae oher belefs BP m, m c. Recalculae (n) based on poson BP observaon of d. Normalze or updae oher belefs, m BP m 3. Mach racs and blobs based on he updaed Belef Marx The Belef calculaon based on he color observaon s: BP = c BP (15) c) where (n) s he Belef BP a he end of he BP observaon from he n h blob (whch s ). As noed before, he value of n s always equal o value of. Equaon 15 s obaned by replacng Assgn c) and p ( wh BP (n) and BP, respecvely, n he poseror equaon (12). Ths follows from he fac ha BP (n) s he poseror probably afer he color observaon from o, and BP s he pror before he color observaon from. Smlarly, he Belef calculaon based on he poson observaon s: BP = d BP (16) d) Afer each blob normalze he Belef Marx row so ha observaon, s necessary o v 1 + m= 1 BP m = 1 We propose he followng formula for updae For all m, Δ NoAssgn ) Belefmnew = Belefmold 1 + NoAssgn ) old (17) Ths formula ensures ha all Belefs are alered proporonally whle mananng he probably v 1 + = 1 requremen BP = 1. 4

5 Usng he followng formal defnons, Belef Belef m old m new m old m new Assgn ) ( n ) NoAssgn ) = 1 BP NoAssgn ) old = 1 BP Δ NoAssgn ) ( n 1) BP( n) = ΔBP equaon (17) becomes m m Δ BP 1 (1 BP old ) BP m new m old (18) Gven he belef marx, a sample of whch s llusraed n Fg. 2a, he bes possble assgnmen s deermned. The marx rows correspond o acve racs from he prevous frame and columns correspond o deeced blobs n curren frame. Fg. 2b shows he resulng rac assgnmens for a sngle frame. Blob 1 s assgned o rac 12, blob 2 o rac 7. Tracs 3 and 11, whch are acve from prevous frames, are declared los n frame PDF s used for R, G, and B; X and Y By manually observng dfference n color and poson n obec racs over a few hundred consecuve frames n he daa, approprae Probably Dsrbuon Funcons were deermned for color and poson probably. The PDF s are shown n Fg Resuls Samples from he racng resuls are shown n fgures 4 and 5. Fg. 4 shows racs 03 and 07 for wo cars hrough an occluded regon. Car 03 s compleely occluded n frame Also, n frames 0210 and 0225, Cars 07 and 03 ge ncorrecly segmened by he bacground segmenaon mehod no wo obecs each. The Bayesan mehod s capable of recoverng from such errors as can be seen from resuls n frame 0285 where boh cars are correcly denfed. Successful racng of cars n a crowded regon s shown n Fg. 5. Obecs 41, 42, 43 and 45 are correcly denfed n successve frames despe her close spaal proxmy. I may be noed ha here s also a sudden change n velocy of obec 41 beween frames 1650 and The algorhm s robus o hs change because uses a combnaon of color and poson for rac correspondence. Erroneous segmenaon of car 41 n frame 1650 (as wo obecs, 41 and 46) and of car 45 (as wo obecs, 45 and 47) s correced by frame Fg. 2. (a) Belef marx (b) Tracs resulng from Belef marx for frame 240 of vdeo sequence We compare he resuls of he Bayesan mehod o a basc Eucldean dsance based mehod smlar o approach used n [10]. The Bayesan belef marx and he Eucldean dsance marx for frame 0275 are presened for analyss n Fg. 6. Each value n he Eucldean dsance marx s he sum of he Eucldean dsances n color (R, G, and B) and poson (y and x coordnaes) values beween a blob of he curren frame and a rac from he prevous frame. The smaller he dsance value, he beer he mach beween he blob and he rac. Frame 0275 (Fg 6b) has wo deeced obecs and hree acve racs from prevous frames. Frame 0265 (Fg 6a) shows he earler rac placemens. Car 12, whch s deeced and raced n frame 0265 s no deeced n frame 0275 due o shorcomngs of he segmenaon algorhm. Also, a new car eners he scene n frame In he belef marx, blobs 1 and 2 are he blobs on he lef sde of frame 0275 and blob 3 s he blob on he rgh sde of frame From he Bayesan belef marx, we can nfer ha blobs 1 and 2 are bes assgned o racs 07 and 03 respecvely. Trac 12 s declared los and a new rac (14) s generaed for blob 3. The Eucldean dsance marx would assgn blobs 1 and 2 correcly, bu would ncorrecly assgn blob 3 o rac 12. From he frames shown, we can see ha hs would be an error. For every rac, he 5

6 Bayesan mehod gves he probably of he rac beng los. I s no possble o calculae he chances of a rac beng los from he Eucldean dsance marx. 6. Conclusons and fuure wor Whle Bayesan reasonng and newors have been used by many researchers n he area of vdeo survellance, hs paper descrbes a drec way of usng Bayesan nference for solvng he correspondence problem n racng. I s drec n he sense ha he color and poson of he new blobs are he observaons, whle he assgnmen of blobs o nown racs s he hypohess. Wh he provson for allowng racs o be declared as los, we add robusness o he sysem and are able o pc up racs of obecs whch may have dsappeared momenarly due o occluson or oher reasons. Though we currenly wor n RGB space, he mehod may be easly expanded o oher observaon spaces such as HSV. I could also be adaped hrough expermenaon o IR magery usng nensy raher han color and addng obec sze as an observaon for fndng correspondence. In hyperspecral daa, Prncpal Componens Analyss could be done o choose he bes bands, and he PDF s of hese bands would be used n place of RGB PDF s for he Bayesan nference. The Bayesan correspondence mehod s a generc approach and no specfc o color vdeo daa. We plan o adap o oher magng modales n he fuure. In ours wor hus far, he PDF s used for color and poson observaons were arrved a by manual observaon of he sequence. In he fuure, hese PDF s may be learn auomacally over he course of he vdeo. Also, n our survellance vdeo, here s sgnfcan change n obec sze and deph from one par of he scene o anoher. We are currenly worng on auomacally learnng hese varaons based on rac moon values. References [1] Guohu L, Jun Zhang, Hongwen Ln, D. Tu, and Maoun Zhang, A movng obec deecon approach usng negraed bacground emplae for smar vdeo sensor, n Proceedngs of he 21s IEEE Insrumenaon and Measuremen Technology Conference, IMTC 04, May 2004, vol. 1, pp [2] C. Sauffer and W. E. L. Grmson, Learnng paerns of acvy usng real-me racng, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 22, no. 8, pp , Aug [3] J. L. Barron, D. J. Flee, and S. Beauchemn, Performance of opcal flow echnques, Inernaonal Journal of Compuer Vson, vol. 12, no. 1, pp , [4] B. K. P. Horn and B. G. Schunc, Deermnng opcal flow, Arfcal Inellgence, 17, pp , [5] Wang, J. Y. A. and Adelson, E. H., Spao-emporal segmenaon of vdeodaa, n Proceedngs of SPIE on Image and Vdeo Processng II, 2182, pp , San Jose, Feb Fg. 3. (a) PDF for color observaon, R (The same dsrbuon apples for color observaons G, B) (b) PDF for poson observaon, Y (The same dsrbuon apples for poson observaon X) [6] R.E. Kalman, A new approach o lnear flerng and predcon problems, Trans. Amercan Socey of Mechancal Engneers, Journal of Basc Engneerng, vol. 82, pp , [7] Y. Bar-Shalom and T. E. Formann, Tracng and Daa Assocaon. Academc Press, [8] M. Isard and A. Blae, Condensaon - condonal densy propagaon for vsual racng, Inernaonal Journal of Compuer Vson, vol. 29, no. 1, pp. 5 28, [9] C. Rasmussen and G. D. Hager, Probablsc Daa Assocaon Mehods for Tracng Complex Vsual Obecs, IEEE Trans. on Paern Analyss and Machne Inellgence, vol. 23, no. 6, pp , Jun [10] S. Inlle, J. Davs, and A. Bobc, Real-me closedworld racng, IEEE CVPR, pp ,

7 Fg. 4. Cars hrough an occluded regon Fg. 5. Crowded cars secon of vdeo 7

8 Fg. 6. Comparson of Bayesan belef marx and Eucldean dsance based marx for frame 0275 Trac numbers assgned for (a) Earler frame 0265 (b) Curren frame 0275 (c) Bayesan belef marx for frame 0275 (d) Eucldean dsance marx for frame