Effective Appearance Model and Similarity Measure for Particle Filtering and Visual Tracking

Transcription

1 Effective Appearance Mode and Simiarity Measure for Partice Fitering and Visua Tracking Hanzi Wang David Suter and Konrad Schinder Institute for Vision Systems Engineering Department of Eectrica and Computer Systems Engineering Monash University Cayton Vic. 38 Austraia {hanzi.wang d.suter Abstract. In this paper we adaptivey mode the appearance of objects based on Mixture of Gaussians in a joint spatia-coor space (the approach is caed SMOG). We propose a new SMOG-based simiarity measure. SMOG captures richer information than the genera coor histogram because it incorporates spatia ayout in addition to coor. This appearance mode and the simiarity measure are used in a framework of Bayesian probabiity for tracking natura objects. In the second part of the paper we propose an Integra Gaussian Mixture (IGM) technique as a fast way to extract the parameters of SMOG for target candidate. With IGM the parameters of SMOG can be computed efficienty by using ony simpe arithmetic operations (addition subtraction division) and thus the computation is reduced to inear compexity. Experiments show that our method can successfuy track objects despite changes in foreground appearance cutter occusion etc.; and that it outperforms severa coorhistogram based methods. Introduction Visua tracking in unconstrained environments is one of the most chaenging tasks in computer vision because it has to overcome many difficuties arising from sensor noise cutter occusions and changes in ighting background and foreground appearance etc. Yet tracking objects is an important task with many practica appications such as smart rooms human-computer interaction video surveiance and gesture recognition. Generay speaking methods for visua tracking can be roughy cassified into two major groups: deterministic methods and stochastic methods. In deterministic methods (for exampe the Mean Shift (MS) tracker []) the target object is ocated by maximizing the simiarity between a tempate image and the current image. The ocaization is impemented by iterative search. These methods are computationay efficient but they are sensitive to background distraction cutter occusion etc. Once they ose the target object they can not recover from the faiure on their own. This probem can be mitigated by stochastic methods which maintain mutipe hypotheses in the state space and in this way achieve more robustness. For exampe the Partice Fiter (PF) [2 3 4] has been widey appied in visua tracking in recent years.

2 A partice fiter tracks mutipe hypotheses simutaneousy and weights them according to a simiarity measure (i.e. the observation ikeihood function). This paper is essentiay concerned with devising and cacuating this ikeihood function/simiarity measure. Visua simiarity can be measured using many features such as intensity coor gradient contour texture or spatia ayout. A popuar feature is coor [ ] due to its simpicity and robustness (against scaing rotation partia occusion and non-rigid deformation). Usuay the appearance of a region is represented by its coor histogram and the distance between the normaized coor histograms of two regions is measured by the Bhattacharyya distance [2 4]. Despite its popuarity the coor histogram aso has severa disadvantages: ) The spatia ayout information of a tracked object is competey ignored (see figure (a)). As a resut a tracker based on coor histograms is easiy confused when two objects with simiar coors but different spatia distributions get cose to each other. An ad-hoc soution is to manuay spit the tracked region into severa sub-regions (e.g. [4 7]). 2) Since the appearance of the target object is reduced to a goba histogram the simiarity measure (e.g. the Bhattacharyya coefficient) is not discriminative enough (see Fig. ) [8]. 3) For a cassica coor histogram based partice fiter the construction of the histograms is a botteneck. The computation is quadratic in the number of sampes. In order to overcome the disadvantages of coor histograms we describe a Spatia-coor Mixture of Gaussians (caed SMOG) appearance mode and propose a SMOG-based simiarity measure in Sect. 2. The main advantage of SMOG over coor histograms and genera Gaussian Mixtures is in that both the coor information and the spatia ayout information are utiized in the objective function of SMOG. Therefore the SMOG-based simiarity measure is more discriminative. When SMOG and the SMOG-based simiarity measure are used in partice fiters one major botteneck is the extraction of the parameters (weight mean and covariance) of SMOG for each partice. In Sect. 3 we propose an Integra Gaussian Mixture (IGM) technique as a fast way to extract these parameters and which aso requires ess memory storage than the integra histogram [9]. In Sect. 4 experiments showing the advantages of our method over other popuar methods are provided. We summarize the paper in Sect SMOG for Partice Fiters 2. A Brief Review of the Partice Fiter Denoting by X t and Y t the hidden state and the observation respectivey at time t. The goa is to estimate the posterior probabiity density function (pdf) p(x t ) of the target object state given a avaiabe observations up to time t: Y :t ={Y i i= t}. Em-

3 poying the first-order Markovian assumption p(x t X : t ) = p(x t X t ) the posterior distribution of the state variabe can be formuated as foows: p (X Y ) L(Y X ) p(x X )p(x Y ) dx () t : t t t t t t : t t Given the dynamic mode p(x t X t ) and the observation ikeihood mode L (Y t X t) the posterior pdf distribution in Eq () can be recursivey cacuated. The partice fiter approximates the posteriori distribution p(x t Y :t ) based on a finite set of random partices and associated weights ( j) ( j) {X } M t W. If we draw partices from an importance density i.e. ( j) ( j) ( j) X ~ q (X X Y t j = : ) the weights of new t t t t partices become: ( j) ( j) ( j) ( j) L(Y t X )p(x X ) t t t Wt (2) ( j) ( j) q(x X Y : ) t t t Then the state estimate of the object at each frame can be obtained by either the mean state or a maximum a posteriori (MAP) estimate []. The observation ikeihood function L (Y t X t ) pays an important roe in the partice fiter. It determines the weights of partices and thereby coud significanty infuence the performance []. The ikeihood function mainy affects the partice fiter by the foowing ways: ) It affects the way partices are re-samped. Re-samping is necessary to decrease the number of ow weighted partices and to increase the ones with more potentia partices. Partices are re-samped according to their weights. 2) It affects the state estimate ˆX t of the target object. Two popuar ikeihood function categories are: contour-based modes (e.g. [2]) and coor-based modes (e.g. [ 2 4 6]). Athough the contour-based mode can accuratey describe the shape of a target it performs poory in cutter and the time compexity is high. In the coor-based mode a coor histogram (due to its robustness to noise rotation and partia occusion etc.) is frequenty empoyed with the Bhattacharyya coefficient as a simiarity measure. However coor histogram has some imitations as we show next. 2.2 Limitations of Coor-histogram Based Simiarity Measure We iustrate the main disadvantage of the coor histogram based simiarity measure: it acks information about the spatia ayout of the target object and is thus not discriminative enough. Denote by ( u) φo = { φ } and ( ) t Ot u=... m φ { u O = φ O } respectivey the m-bin normaized ν ν u=... m coor histograms of target mode O t and the target candidate O ν the Bhattacharyya coefficient (i.e. the simiarity measure) between the reference region and candidate region is:

4 m ( u) ( u) ( O O ) O O ρ φ φ = φ φ (3) t ν t v u= Simiaity Measure Simiaity Measure Simiaity Measure Y (a) X Transation (b) Transation (c) Scae (d) Fig.. Coor-histogram based simiarity measure. The score of the simiarity measure over (b) x-transation; (c) y-transation; and (d) scaing. (see text beow and compare with Fig. 2) In Fig. we track a face comprising pixes within a red rectange region in a video sequence from Target candidates are generated by transating the rectange from -2 to 2 horizontay or verticay and by scaing the rectange by a factor of.2 (the smaer green rectange inside the target mode) to 2 (the arger green rectange inside the target mode) in steps of.2. We use 8x8x8 coor histogram bins. From Fig. we can see that the simiarity measure by Eq. (3) obtains very simiar scores for different target candidates and does not discriminate we between different candidate regions. 2.3 SMOG: A Joint Spatia-Coor Appearance Mode Both the appearance mode and the simiarity measure are very important to the performance of partice fiters. The coor histogram as described above is one popuar appearance mode. Other popuar modes for foreground and/or background appearance incude: the Gaussian [3] the kerne density [4 5] and the MOG (Mixture of Gaussians) based appearance mode [ ]. For exampe [3] represented humans by bobs and modeed each bob by a Gaussian mode. The kerne density based mode is robust to noise and does not require the cacuation of parameters (such as weights mean and covariance of the Gaussian mode) but it is computationay expensive and requires a arge storage space. It is aso not trivia to update the appearance changes. The disadvantage of the genera MOG-based mode is that it treats each pixe independenty without using any spatia information. Moreover it requires setting the number of Gaussians and a earning rate. Despite these imits it is popuar because () it can mode the muti-moda distribution of the appearance; (2) it is computationay efficient; (3) it is easy to adapt to the changes of the appearance; and (4) it does not require a arge storage space. We mode the appearance of an object with a joint spatia-coor mixture of Gaussians. We refer to this approach as SMOG. We denote by S i =(x i y i ) and j C={C} i i j=... respectivey the spatia feature (i.e. the 2D coordinates) and the coor d feature with d coor channes (in RGB coor space C i={r ig ib i } and d=3) at pixe x i. Thus we can write the features of x i as the Cartesian product of its position and

5 coor: x i = (S ic i). We assume that the spatia feature (S) and the coor feature (C) are independent to each other. For the mean and the covariance of the th mode of the S C S C Gaussian Mixtures we have µ = ( µ µ )and Σ = ( Σ Σ ). The estimated density t t t t t t at the point x i in the joint spatia-coor space can be written as: S T S S C T C C exp (S ) ( ) (S ) exp (C ) ( ) (C ) k i µ t Σt i µ t i µ t Σt i µ t 2 2 po( xi) = ω t S /2 d /2 C /2 = 2 π Σt (2 π) Σt (4) 2.4 SMOG-Based Simiarity Measure We mode the appearance of a target object O t by SMOG with k modes. We initiaize S C S C the parameters of SMOG for a target object { ω Ot O t O t O t O t= µ t= µ t= Σt= Σt t } by a K- = =... k means agorithm foowed by a standard EM agorithm. Once we obtain the parameters of the target object we either update these parameters in an exponentia forgetting way or keep the parameters (if we detect that it is occuded by other objects) in the foowing frames (t=23 ). At time t we sampe M partices (i.e. target candidates O v ) and evauate the ikeihood function in Eq. () for each partice. The parameters of each target candidate ν Oν Oν Oν Oν { ω µ µ Σ Σ } are cacuated by: O S C S C = t t t t t... k. Cacuate the Mahaanobis distances between pixes {x i } in the target candidate O v ={ x i } i= to each mode of SMOG of the target object O t in coor space: D (C µ Σ ) = (C µ ) ( Σ ) (C µ ) (5) 2 C O t C O t C O t T C O t C O t i t t i t t i t 2. Labe the pixes satisfying AY( D = k 2.5) with the number of the mode to which the Mahaanobis distance is the east. For other pixes abe them with zero. LB( xi) = argmin D (6) O S C S C 3. Cacuate the parameters ν Oν Oν Oν Oν { ω µ µ Σ Σ } of the target candidate by: = t t t t t... k k O ν t = ( LB( xi) ) ( LB( xi) ) i= = i= Oν S Oν C Oν t t t i i i i= i= ω δ δ (7) µ = ( µ µ ) = xδ( LB( x ) ) δ( LB( x ) ) O S O C O O T O ν ν ν ν ν Σ t = ( Σt Σ t ) = ( xi µ t ) ( xi µ t ) δ( LB( xi) ) δ( LB( xi ) ) i= i= where δ is the Kronecker deta function. The covariance matrix is taken to be a diagona matrix for simpicity. One shoud normaize the coordinate space first so that the coordinates of pixes in the target candidate (and target object) are within the range [ ]. S C Let Λt and Λ be respectivey the spatia and the coor simiarity measure between the th mode of the target candidate O v and the th mode of the target object O t t.

6 The SMOG-based simiarity measure (as compared to the coor-histogram based simiarity measure in Eq. (3)) between two regions (O v and O t ) in the joint spatiacoor space is defined as: k S C Λ ( Ot Ov) = Λt Λ (8) t = where S S Oν S Ot T ˆ S S Oν S Ot Λ t = µ t µ t Σt µ t µ t 2 and C Oν O Λ min( t t = ωt ωt ). exp ( ) ( ) ( ) The ikeihood function in our method is given by: L(Y t X t) exp 2 ( Λ( Ot Ov) ) 2σ b where σ b is the observation variance. with ˆ S S Oν S O ( Σ ) ( ) ( t t = Σ t + Σ t ) (9) Simiaity Measure Simiaity Measure Simiaity Measure X Transation Y Transation Scae (a) (b) (c) Fig. 2. The score by the SMOG-based simiarity measure over (a) x-transation; (b) y- transation; and (c) scae. We repeat the experiment in Fig. using SMOG. As shown in Fig. 2 the SMOGbased simiarity measure (Eq. (8)) is more discriminative than the coor-histogram based simiarity measure in Eq. (3). Recenty Birchfied et a. [2] proposed a method (Spatiograms) which captures the spatia information of the genera histogram bins and appied it to the Mean Shift (MS) tracker. The spatia mean and covariance of each bin is computed. In contrast we consider the spatia ayout and coor distribution of each mode of SMOG. The number of the Gaussians (normay k is set within the range from 3 to 7 in our case) is much ess than the number of the histogram bins. SMOG is aso more efficient in estimating density distribution of the data and in computation and requires ess storage space to buid up an integra Gaussian mixtures image (as described in Sect. 3) than the integra histogram method [9]. 2.5 Updating the Parameters of SMOG We dynamicay mode the object appearance by updating the parameters of SMOG through a earning rate α. The assumption made here is that in the temporay neighboring frames (e.g. frame t and frame t-) the appearance (incuding both spatia and coor distributions) of an object does not change dramaticay.

7 Simiar to [] and [7] we assume that the past appearance is exponentiay forgotten and new information is graduay added to the appearance mode. To hande occusion where image outiers exist we use a heuristic way: we update the appearance ony if the score of the simiarity measure is arger than a threshod T u. When occusion is decared (i.e. the score is ess than T u ) we stop updating the appearance mode. 2.6 Choosing the Coor Space t=2 t=34 t=52 Fig. 3: Tracking resuts empoying RGB as coor feature (in the first row) and rgi as coor feature (in the second row). We empoy the normaized coor space in our method. The normaized chromaticity coordinates of (r g b) can be written as: r=r/(r+g+b); g=g/(r+g+b); b=r/(r+g+b). The intensity information is aso expoited. Thus we use (r g I) as the coor feature in our method. In Fig. 3 we show an experiment iustrating the advantage of (r g I) over (R G B) coor space in deaing with iumination changes. (r g I) coor space shows more robustness to the iumination change. In contrast the method empoying (R G B) achieved ess accurate resuts and ost the target at the end. Fig. 4 shows the adaptation of the proposed method to the appearance changes by updating the appearance mode in subsection 2.5. Our method succeeds in adaptation to appearance changes throughout the sequence.

8 Fig. 4: The appearance of the tracked target changes with time increasing. 3 Integra Gaussian Mixture for Higher Computationa Efficiency To efficienty cacuate the simiarity measure Λ ( Ot Ov) (in Eq. (8)) we need to cacuate Oν { S Oν S Oν ω µ Σ } for each target candidate. One possibe way which is =... k usuay used in the coor-histogram based partice fiters (such as [2 4]) is to randomy sampe a partice and generate a target candidate and then cacuate the parameters corresponding to the candidate region. This is computationay inefficient because partices may have many overapped regions and the same operator for each possibe region can be repeated many times. To overcome this inefficiency integra methods expoiting rectange features were introduced by Vioa et a. [22] and more recenty were deveoped by Poriki [9]. In [22] a grey-eve image is converted to integra image format (i.e. the vaue of each pixe is the sum of vaues of a pixes to the eft and above of the current pixe). In [9] integra histogram is constructed by a recursive propagation of an aggregated histogram in a Cartesian data space. We propose an Integra Gaussian Mixture (IGM) technique as a fast and efficient way to extract the parameters of SMOG for each partice. To cacuate the parameters 2 2 of the th mode of a target candidate we need to cacuate ( n µ x µ y σx σ y ) i.e. the number of pixes whose abe is the spatia mean and variance vaues in x and y coordinates. We can write these quantities in the foowing form: n = δ ( LB( x ) ) i i= x = x i ( LB( xi) ) n; y = y i ( LB( xi) ) n i= i= y = x i ( LB( xi) ) n x ; y = y i ( LB( xi) ) n y i= i= µ δ µ δ σ δ µ σ δ µ and we have ω = n 2 k σ S S x n; µ = ( µ x µ y ); Σ = () 2 = σ y ()

9 The procedure of the IGM can be described as foows:. Predict the region R that incudes a partices (i.e. target candidates) in the 2D image. 2. Labe each pixe xi in R by step and 2 in subsection Generate a GM image whose i th pixe is given by x i = { x i } =... k where 2 2 x = δ ( LB( x ) )(x x y y ). i i i i i i 4. Buid an IGM image where each pixe is the sum of vaues of a pixes of the GM image to the eft and above of the current pixe. 5. Cacuate the parameters of each target candidate by four tabe ookup operations which are simiar to [22]. 6 6 Traditiona coor histogram based method Our IGM based method Traditiona coor histogram based method Our IGM based method CPU Time (S) 3 CPU Time (S) umber of Partices umber of Partices (a) (b) Fig. 5: The computationa time v.s. the number of partices for the coor histogram based method and the proposed method. Candidate region size in (b) is twice as that in (a). We find that once the IGM is buit the cacuation of the ikeihood function is very fast. Fig. 5 gives a rough estimation of the computationa time (in MATLAB code) to evauate the ikeihood function for partices. From Fig. 5 we can see that the cacuation of the coor histogram based simiarity measure in Condensation is computationay expensive and wi be affected by both the number of partices and the size of target candidate regions. When we doube the region size (Fig. 5 (b)) of the candidate region (Fig. 5 (a)) the computationa time of the coor histogram based Condensation increased by about 6%. In contrast both the number of partices and the size of the target candidate regions have much ess infuence on the computationa compexity of the proposed method: the processing time is about to 2 times ess than the coor histogram based Condensation. 4 Experiments We test the effectiveness of our method using a number of video sequences with different environments and conditions. We compare with two popuar coor histogram based methods: the Mean Shift tracker and Condensation. ote: we empoy the (r g I) coor space for a three methods. For the Mean Shift tracker and the Conden- Some demo video sequences of our method can be obtained from

10 sation tracker we use 6x6x6 coor histogram bins. For both Condensation and our method we empoy a random wak dynamic mode (number of partices M=2). t= t=8 t=2 t=2 t=25 Fig. 6: Tracking resuts of the face sequence with the MS tracker (first row) Condensation (second row) and our method (third row). In Fig. 6 the human face is moving to the eft and right very quicky. The iumination on the face aso changes. The background scene incudes cutter and materia of simiar coor to the face. As we can see in Fig. 6 the Mean Shift tracker fais to track the face very soon; the resuts of Condensation are not accurate and Condensation even fais to track the face in some frames because the coor histogram based simiarity measure is not discriminative enough (section 2.2). In comparison our method which considers both coor and spatia information of the target object never oses the target and achieves the most accurate resuts. t= t=3 t=4 t=53 t=74 Fig. 7: Tracking resuts of the gir sequence with the MS tracker (first row) Condensation (second row) and our method (third row). The tracked face is shown in the upper-right window.

11 t= t=53 t=57 t=6 (a) (b) t=3 to t=44 Fig. 8: Tracking resuts of the soccer sequence with three methods (a): the MS tracker (first row) Condensation (second row) and our method (third row). The tracked body is aso shown in the upper-right window; (b) tracking resuts with occusions by our method. t= t=8 t=49 t=73 (a) a (b) Fig. 9: (a) Tracking resuts of the footba sequence with the MS tracker (first row) Condensation (second row) and our method (third row); (b) the target appearance changes (frames from 2-77).

12 Fig. 7 and Fig. 8 show situations where two humans with very simiar coors get cose to each other and one occudes the other. In Fig. 7 when the man s face gets cose to and occudes the gir s face the resuts of both the MS tracker and Condensation are greaty infuenced. In Fig. 8 (a) because the coor histogram based simiarity ignores the spatia information both the MS tracker and Condensation break down when two payers with simiar coors but different spatia distributions get cose to each other. In contrast our method works we in both cases. Fig. 8 (b) shows that our method can sti effectivey track the human body even if it is amost competey occuded by another payer. ext we test the adaptation of our method to appearance changes. In Fig. 9 a particuary chaenging (with high cutter) video sequence is used. The head of a payer is tracked even though it moves fast and the appearance of the head changes frequenty (incuding occusion burring and changes in the spatia and coor distributions of the appearance). Fig. 9 shows that our method has successfuy tracked the target and adapted to the changes of the target appearance. 5 Concusion We have described an effective appearance mode (SMOG) in a joint spatia-coor space and a new simiarity measure based on SMOG. The SMOG appearance mode and the SMOG-based simiarity measure consider both the spatia distribution and the coor distribution of objects: they utiize richer information than the genera coor histogram based appearance mode and simiarity measure. We aso propose an Integra Gaussian Mixture (IGM) technique which greaty improves the computationa efficiency of our method. Thus the number of partices and the size of target candidate region can be greaty increased without significant change in the processing time of the proposed method. We have successfuy appied the SMOG appearance mode and the SMOG-based simiarity measure to the task of visua tracking in the framework of partice fiters. Our tracking method can effectivey hande cutter iumination changes appearance changes occusions etc. Comparisons show that our method outperforms popuar methods such as the genera coor histogram based MS tracker and Condensation. Acknowedgements We thank Dr. Chunhua Shen for his vauabe comments and the ARC for support (grant DP45246).

13 References. Comaniciu D. V. Ramesh and P. Meer Kerne-based Object Tracking. IEEE Trans. Pattern Anaysis and Machine Inteigence (5): p ummiaroa K. E. Koer-Meierb and L.V. Goo An Adaptive Coor-Based Partice Fiter. Image and Vision Computing 23. 2: p Isard M. and A. Bake Condensation-Conditiona Density Propagation for Visua Tracking. Internationa Journa of Computer Vision (): p Perez P. et a. Coor-Based Probabiistic Tracking. European Conference on Computer Vision. 22. p Shen C. A.v.d. Henge and A. Dick. Probabiistic Mutipe Cue Integration for Partice Fiter Based Tracking. Internationa Conference on Digita Image Computing - Techniques and Appications. 23. p McKenna S.J. et a. Tracking Groups of Peope. Computer Vision and Image Understanding 2. 8: p Pérez P. J. Vermaak and A. Bake Data Fusion for Visua Tracking with Partices. Proceedings of the IEEE (3): p Yang C. R. Duraiswami and L. Davis. Fast Mutipe Object Tracking via a Hierarchica Partice Fiter. Internationa Conference on Computer Vision. 25. p Poriki F. Integra Histogram: A Fast Way to Extract Histograms in Cartesian Spaces. Computer Vision and Pattern Recognition. 25. p Zhou S. R. Cheappa and B. Moghaddam Visua Tracking and Recognition Using Appearance-Adaptive Modes in Partice Fiters. IEEE Transactions on Image Processing 24. : p Lichtenauer J. M. Reinders and E. Hendriks. Infuence of the Observation Likeihood Function on Partice Fitering Performance in Tracking Appications. IEEE Internationa Conference on Automatic Face and Gesture Recognition. 24. p Isard M. and A. Bake. ICODESATIO: Unifying Low-eve and High-eve Tracking in a Stochastic Framework. European Conference on Computer Vision p Wren C.R. et a. Pfinder: rea-time tracking of the human body. IEEE Trans. Pattern Anaysis and Machine Inteigence (7): p Egamma A. et a. Background and Foreground Modeing using on-parametric Kerne Density Estimation for Visua Surveiance. Proceedings of the IEEE 22. 9(7): p Yang C. R. Duraiswami and L.S. Davis. Efficient Mean-Shift Tracking via a ew Simiarity Measure. Computer Vision and Pattern Recognition. 25. p McKenna S.J. Y. Raja and S. Gong Tracking Coour Objects Using Adaptive Mixture Modes. Image and Vision Computing : p Stauffer C. and W.E.L. Grimson. Adaptive Background Mixture Modes for Rea-time Tracking. Computer Vision and Pattern Recognition p Han B. and L. Davis. On-Line Density-Based Appearance Modeing for Object Tracking. Internationa Conference on Computer Vision. 25. p Wu Y. and T.S. Huang Robust Visua Tracking by Integrating Mutipe Cues Based on Co- Inference Learning. Internationa Journa of Computer Vision (): p Khan S. and M. Shah. Tracking Peope in Presence of Occusion. Asian Conference on Computer Vision. 2. p Birchfied S. and S. Rangarajan. Spatiograms versus Histograms for Region-Based Tracking. Computer Vision and Pattern Recognition. 25. p Vioa P. and M. Jones Robust Rea-Time Face Detection. Internationa Journa of Computer Vision (2): p