A Rao-Blackwellized Parts-Constellation Tracker

Transcription

1 A Rao-Blackwellized Pars-Consellaion Tracker Gran Schindler and Frank Dellaer College of Compuing, Georgia Insiue of Technology {schindler, Absrac We presen a mehod for efficienly racking objecs represened as consellaions of pars by inegraing ou he shape of he model. Pars-based models have been successfully applied o objec recogniion and racking. However, he high dimensionaliy of such models presen an obsacle o radiional paricle filering approaches. We can efficienly use pars-based models in a paricle filer by applying Rao-Blackwellizaion o inegrae ou coninuous parameers such as shape. This allows us o mainain muliple hypoheses for he pose of an objec wihou he need o sample in he high-dimensional spaces in which parsbased models live. We presen experimenal resuls for a challenging biological racking ask. 1. Inroducion We are ineresed in racking insecs in video, a ask complicaed by he fac ha insecs exhibi non-rigid moion. Like oher racking arges, such as people, insecs are physically composed of muliple pars ha flex and bend wih respec o each oher. We would like o model his flexible moion, which is hypohesized o improve he performance of our racker and increase he richness of he daa ha can be used for subsequen analysis. As such, we adop a model ha incorporaes an objec s individual pars, modeling he join configuraion of he pars as a whole, and modeling he appearance of each par individually. We show how o efficienly incorporae such a model ino a paricle filer by reaing he shape analyically and sampling only over pose, a process commonly known as Rao-Blackwellizaion. We use Expecaion-Maximizaion (EM) o learn appearance and shape parameers for hese models and perform racking wih a Rao-Blackwellized paricle filer. We adop he framework of [5] o model insecs as flexible consellaions of pars. Though pars-based models have a long hisory of use in compuer vision, a powerful probabilisic formulaion for modeling objecs composed of flexible pars was firs offered by Burl, Weber, and Perona [2] and laer exended by Fergus, Perona, and Zisserman [5]. In heir formulaion, each par has a locaion, appearance, and relaive scale, and he shape of an objec is represened as Figure 1: Pars-consellaion model of a bee. We learn a join shape disribuion on par configuraions, as well as an appearance model for each par. The mean appearance and pose of each par are shown above. Ellipses show individual par covariances. By inegraing over shape, we can efficienly incorporae such a model ino a paricle filer. he relaive locaion of is pars. We apply his framework o he problem of racking moving objecs in video. Oher pars-based mehods have been used for racking as well. A pars-based mehod for racking loose-limbed people in 3D over muliple views is presened in [15], which makes use of boom-up par-deecors o deec possible par locaions in each frame. Our mehod akes a relaed approach, bu uses an image regisraion echnique based on he wellknown Lucas-Kanade algorihm [10] for locally regisering par emplaes. In conras o [15], we are racking a arge across a single view conaining many oher idenical arges. Rao-Blackwellizaion, as applied o paricle filers, is a mehod o analyically compue a porion of he disribuion over he sae space, so as o reduce he dimensionaliy of he sampled space and he number of samples used. Rao-Blackwellized paricle filers (RBPFs) have previously been applied o several esimaion problems, including insec racking. In [9], an RBPF was used o incorporae subspace appearance models ino paricle filer racking of insecs. In [14], he auhors inegrae over he 2D arge posiions and sample over measuremen arge assignmens o rack people based on noisy posiion measuremens from IR sensors. In [6], de Freias uses an RBPF for faul deecion where Kalman filers are applied over coninuous parameers and samples obained over discree faul saes. And finally, in [12], he auhors inegrae over landmark locaions in a roboics applicaion where he goal is o localize a robo while simulaneously building a map of he environmen.

2 2. A Bayesian Filering Approach Bayesian filering is a radiional approach o he arge racking problem in which, a ime, we recursively esimae he poserior disribuion P (X Z 1: ) of some sae X condiioned on all measuremens Z 1: up o ime as: P (X Z 1: ) P (Z X ) (1) X 1 P (X X 1 )P (X 1 Z 1: 1 ) We call P (Z X ) he measuremen model and P (X X 1 ) he moion model. When applying a Bayes filer o he problem of parsbased racking, he sae X = (Y, S ) has wo componens: he pose Y of he arge and he shape S describing he configuraion of pars. The measuremens Z 1: = I 1: are images I observed a ime in a video sequence. By analogy o equaion (1), if we waned o compue he poserior disribuion P (Y, S I 1: ) on boh pose Y and shape S, he corresponding Bayes filer would be compued by inegraing over boh he pose Y 1 and he shape S 1 a he previous ime sep 1: P (Y, S I 1: ) = kp (I Y, S ) Y 1 S 1 P (Y, S Y 1, S 1 )P (Y 1, S 1 I 1: 1 ) By inegraing over he curren shape S on boh sides of equaion 2 we obain a marginal filer on he pose Y alone : P (Y I 1: ) = k P (I Y, S ) S Y 1 S 1 P (Y, S Y 1, S 1 )P (Y 1, S 1 I 1: 1 ) In our model, we will assume ha (a) he moion of he arge is independen of shape, and (b) ha here is no emporal coherence o he shape. Taking ino accoun hese independence assumpions he join moion erm facors ino he produc of a simpler moion model P (Y Y 1 ) and a shape model P (S Y ): P (Y, S Y 1, S 1 ) P (Y Y 1 )P (S Y ) Thus he final form of our exac marginal Bayes filer is: P (Y I 1: ) = k P (I Y, S )P (S Y ) S Y 1 (2) S 1 P (Y Y 1 )P (Y 1, S 1 I 1: 1 ) (3) We describe a Mone Carlo approximaion of his Bayes filering disribuion in Secion The Pars-Consellaion Model To fully specify he above Bayes filer in equaion (3), we need o define an appearance model P (I Y, S ), a shape model P (S Y ), and a moion model P (Y Y 1 ). Here, we describe our appearance and shape models in more deail. The moion model does no depend on shape, and is hus no specific o our approach Appearance Model If we define he image I as he union of foreground and background image regions F (Y, S) and B(Y, S), whose posiion and exen have a funcional dependence on boh pose Y and shape S, he appearance model facors as: P (I Y, S) = P (F (Y, S), B(Y, S) Y, S) = P F (F (Y, S))P B (B(Y, S)) Here P F and P B are disribuions on he foreground and background models, respecively. This facorizaion is valid if we assume no overlap beween foreground and background regions in he image. Similar o he approach aken by [2] and [5], we can arrive a a formulaion of he image likelihood purely in erms of F, he foreground region of ineres, by muliplying boh sides of his expression by a consan: = P F (F (Y, S))P B (B(Y, S)) P B(F (Y, S)) P B (F (Y, S)) = P B (F (Y, S), B(Y, S)) P F (F (Y, S)) P B (F (Y, S)) P F (F (Y, S)) P B (F (Y, S)) Finally, we break he foreground F ino muliple regions F n corresponding o he individual pars of he model, obaining a produc of likelihood raios P (I Y, S) n P Fn (F n (Y, S)) P B (F n (Y, S)) where each par of he foreground F n is evaluaed according o a differen foreground disribuion P Fn Shape Model Shape is modeled as a join Gaussian disribuion P (S Y ) on par posiions and is parameerized by θ shape = {µ shape, Σ shape }. For example, if here are N pars and each par has boh a locaion and an orienaion, hen Σ shape is a full 3N 3N covariance marix. This is similar o he shape model from [5]. The shape model is condiioned on pose Y simply because he mean µ shape is defined wih respec o he arge s curren posiion. (4)

3 By subsiuing his approximae poserior (5) ino formula (3) for he exac marginal Bayes filer, we obain he following Mone Carlo approximaion o he marginal filer: P (Y I 1: ) k P (I Y, S )P (S Y ) S i w (i) 1 P (Y Y (i) 1 )α(i) 1 (S 1) Figure 2: Learned parameers for he foreground image models of he pars of a bee. The lef column shows he mean pixel values of each par, while he righ column shows he pixel variance. 4. A Rao-Blackwellized Paricle Filer Our Bayes filering disribuion can be approximaed wih a paricle filer [7, 8, 3] in which he poserior P (Y 1, S 1 I 1: 1 ) is represened by a se of weighed paricles. Using a radiional paricle filer, we would sample a pose Y from he moion model P (Y Y 1 ), sample a shape S from he shape model P (S Y ), and hen weigh his join sample using he appearance model P (I Y, S ). However, hese join samples on pose and shape live in such a high-dimensional space ha approximaing he poserior disribuion requires an inracably large number of paricles, poenially making a pars-based model infeasible. In a Rao-Blackwellized paricle filer (RBPF) [13] we analyically inegrae ou shape and only sample over pose. Thus, we can achieve he same performance as a radiional paricle filer wih vasly fewer paricles. As in [9], we approximae he poserior P (Y, S I 1: ) over he sae {Y, S } wih a se of hybrid paricles {Y (j), w (j), (S )} M j=1, where w(j) is he paricle s imporance weigh and each paricle has is own condiional disribuion (S ) on shape S : (S ) = P (S Y (j), I 1: ) P (I Y (j), S )P (S Y (j) ) The hybrid samples consiue a Mone Carlo approximaion o he poserior erm P (Y, S I 1: ) as follows: P (Y, S I 1: ) P (Y I 1: )P (S Y, I 1: ) (5) j w (j) δ(y (j), Y ) (S ) Thus, he Rao-Blackwellized paricle filer proceeds hrough he following seps: Algorihm 1 Rao-Blackwellized Pars-Consellaion Filer 1. Selec a paricle Y (i) 1 from he previous ime sep according o weighs w (i) Sample a new paricle Ŷ (j) P (Y Y (i) 1 ) from he moion model 3. Calculae he poserior densiy (S ) on shape S : (S ) = P (I Y (j), S )P (S Y (j) ) 4. Compue he imporance weigh w (j) (see Secion 4.1 below) Compuing Imporance Weighs = S (S ) The imporance weigh compuaion involves an inegraion over shape S, bu i is racable because (S ) is a Gaussian. The inegral of any funcion q(x) = k exp { 1 2 µ Σ} x 2 proporional o a Gaussian is { q(x) = k exp 1 } x x 2 µ x 2 Σ = k 2πΣ where µ x 2 Σ = (µ x) T Σ 1 (µ x) is he squared Mahalanobis disance from x o µ wih covariance marix Σ. Noe ha he consan k is equal o q(µ), as { q(µ) = k exp 1 } 2 µ µ 2 Σ = k exp { 12 } 0 = k Hence, wih µ he mode of q(x), we have q(x) = q(µ) 2πΣ (6) x Observe ha if (S ) is a produc of all Gaussian erms, hen i is iself Gaussian. Thus, he only consrain on our model is ha he shape model, foreground P Fn (.) and background P B (.) models be normally disribued.

4 We find he mode of (S ), which we denoe by S, by opimizaion. We use an inverse-composiional varian of he Lucas-Kanade algorihm [10, 1] which opimizes he shape by regisering emplaes for each par (he means of he foreground disribuions P Fn (.)) o he measured image I. See secion 6 for an explanaion of he assumpions underlying his approach. Finally, we apply he above propery of Gaussians (6) o compue our imporance weigh: w (j) = S (S ) = (S ) 2πΣ = P (I Y (j) n, S )P (S Y (j) ) 2πΣ P Fn (F n (Y (j), S )) P B (F n (Y (j), S )) P (S Y (j) ) 2πΣ 5. Learning Model Parameers For a par-based model wih N pars, we mus learn he shape parameers θ S = {µ S, Σ S }, and appearance parameers θ I = {θ F1,..., θ FN, θ B } (see figure 2). Given a se of raining images I 1:T, we use expecaion-maximizaion (EM) [4, 11], saring from an iniial esimae for he parameers θ = {θ S, θ I }. Here, we assume pose Y is given and rea shape S as a hidden variable. E-Sep The goal of he E-sep is o calculae he poserior disribuion P (S Y, I, θ) on shape for each raining image given he curren esimae of he parameers θ. Noe ha his disribuion is almos idenical o he condiional disribuion α(s) on shape from he RBPF. Thus, in he E-sep, we essenially creae a se of hybrid paricles each wih given pose Y and disribuion α (S θ) defined wih respec o he curren parameer esimaes: P (S Y, I, θ) n P Fn (F n (Y, S) θ Fn ) P B (F n (Y, S) θ B ) P (S Y, θ S) (7) M-Sep In he M-sep, we maximize he expeced logposerior Q(θ; θ old ) wih respec o θ o obain a new se of parameers θ new. θ new = argmax θ Q(θ; θ old ) We define S as he opimal shape for raining image I according o equaion 7. This opimal shape is obained using image regisraion as explained before in Secion 4.1. We compue he expeced log-poserior Q(θ; θ old ): = E[log P (I Y, S, θ I ) + log P (S Y, θ S )] P (S Y,I,θold ) = [ [log P Fn (F n (Y, S ) θ Fn ) n log P B (F n (Y, S ) θ B )] + log P (S Y, θ S )] Inuiively, afer we find he se of opimal shapes S1:T, finding he θ ha maximizes Q(θ; θ old ) is equivalen o calculaing he mean and covariance direcly from he shapes and appearances defined by S1:T. Noe ha he background parameers θ B are no updaed during EM. 6. Experimenal Resuls Figure 3: Tracking a dancing honey bee in an acive hive is a difficul ask, furher complicaed by he non-rigid shape of he bee and he complex waggle porion of he dance. We used he pars-based Rao-Blackwellized paricle filer o rack a dancing honey bee in an acive hive (see figure 3), a difficul ask ha is of ineres o biologiss sudying honey bee behavior. There are over 120 bees in frame hroughou he video and hey ofen collide and/or occlude each oher. The waggle porion of a bee s dance involves complex moion ha is no easily modeled, while he urning porion of he dance bends a bee s body in a way ha is difficul o model wih only a single emplae. We represen he pose of a bee as a vecor Y = {x, y, θ} including 2D posiion (x, y) and roaion θ. The cener of roaion of each arge is 20 pixels from he fron of is head. We model a bee as consising of N = 3 pars of size 30x20 pixels each. Each par has is own pose {x n, y n, θ n } wih respec o he pose Y of he enire bee, such ha he shape S of bee composed of N pars is represened by a 3 N dimensional vecor. Therefore, for our ask, he shape model is a 9-dimensional Gaussian on join shape configuraion S, he moion model is a 3-dimensional Gaussian on change in pose Y, and he appearance model pus a 3-dimensional (R,G,B) Gaussian on every pixel of each foreground region F 1:N, while he background B is modeled wih a single 3- dimensional Gaussian on he average color of a region. The parameers of he shape, appearance, and moion models were learned from a raining daa se consising of

5 Tracking Mehod Paricles Failures Mean Translaion Error Mean Roaion Error Mean Time/Frame Lucas-Kanade pixels 0.42 rad 0.25 s 1-Par PF pixels 0.12 rad 0.89 s 3-Par RBPF pixels 0.27 rad 0.85 s Table 1: The pars-based RBPF using only 80 paricles fails less han half as many imes as wo oher rackers ha do no model shape, including a emplae-regisraion mehod and a single-emplae paricle filer ha uses 540 paricles. Paricles Failures Mean Translaion Error Mean Roaion Error Mean Time/Frame pixels 0.26 rad 0.97 s pixels 0.20 rad 1.60 s pixels 0.19 rad 2.98 s pixels 0.18 rad 5.64 s pixels 0.13 rad s Table 2: Incorporaing a pars-consellaion model ino a radiional paricle filer wihou Rao-Blackwellizaion requires many more samples and sill does no achieve he performance level of he RBPF. 807 frames of hand-labeled bee poses. The shape and appearance models were learned simulaneously by applying EM o a subse of 50 frames of raining daa. The moion model was learned from he incremenal ranslaions and roaions beween successive frames of he enire raining se. We esed our racker on a video sequence (810 frames a 720x480 pixels) for which we have hand-labeled ground ruh daa. All ess were run on a 2.8 GHz Penium 4 processor. We say a failure occurred when he racked posiion differs from he ground ruh posiion by more han half he bee s widh (15 pixels). For hese experimens, when he racker fails, i is reiniialized a he ground ruh posiion for he curren frame and resumes racking. Firs, we compared our pars-based RBPF racker agains wo oher racking mehods (see Table 1). The firs racker (which avoids paricles compleely) uses an ieraive imagealignmen mehod based on Lucas-Kanade. A single bee emplae is aligned o he image in each frame, saring from he aligned locaion of he previous frame. The second racker is a radiional paricle filer wih a single-emplae appearance model in which no opimizaion occurs. This paricle filer samples over pose, bu shape is no modeled a all. Using a pars-consellaion model decreased he number of racking failures by a facor of 2, from 50 o 24. For comparison, we show he performance in Table 2 of a pars-based model which does no rea shape analyically and insead samples over boh shape and pose. Even afer repeaedly doubling he number of paricles o 1600, he racking performance does no improve much beyond he resuls of he non-shape-based paricle filer in Table 1. Because join samples on shape and pose live in a 12- dimensional space, an exremely large number of paricles (and processing ime) would be needed o mach he performance of our 80-paricle RBPF racker. Only wih Rao- Blackwellizaion do pars-consellaion models become efficien enough o be of use in a paricle filer. In furher experimens, we recorded only 3 failures for an 80-paricle RBPF using a more compuaionally expensive per-pixel background model (see Figure 4), while a 500-paricle single-par PF using his model fails roughly wice as ofen. The remaining failures occurred during he waggle porion of he dance, suggesing ha a more sophisicaed moion model is necessary o furher reduce he number of racker failures. Noe ha in heory, we should joinly align all pars of he model simulaneously wih respec o boh he join shape disribuion and he individual par appearance models. In our implemenaion, we iniialize he shape a he mean of he join shape disribuion, bu we are opimizing he appearance of each par individually. Because he appearance model ofen dominaes he shape model, his is a reasonable approximaion which is suppored by he experimens. 7. Conclusion The pars-based RBPF racker presened here reduced racker failures by a facor of 2. We used somewha naive appearance and moion models, in par, so ha we could isolae and observe more clearly he specific advanages of a pars-based model for difficul racking asks. Only wih he addiion of more sophisicaed appearance models (e.g. subspace models) and moion models (e.g. swiching linear dynamic sysems) would we expec he racker o perform perfecly. Wha we have demonsraed is ha pars-consellaion models can be beneficial for some racking asks, and ha

6 Rao-Blackwellizaion enables he efficien use of pars-based models for paricle filering. References X Y T pi 0-2pi (a) x vs. y Frame Number (b) From op o boom: x,y,hea vs. ime Tracker Paricles Failures Trans. Err. Ro. Err. Time/Frame 1-Par PF pixels 0.08 rad 2.01 s 3-Par RBPF pixels 0.13 rad 2.03 s (c) Tracker performance wih a more expensive per-pixel background model. Figure 4: A pars-based RBPF wih 80 paricles recorded 3 failures over he course of a bee dance (c). We plo he racker s performance agains ground ruh (a) in a 2D view and (b) in a ime series view. The ground ruh rajecory is ploed in blue, he rajecory reurned by he racker is ploed in red, and failures are indicaed wih black dos and ick marks in (a) and (b) respecively. Observe ha all of he remaining failures occur during he waggle porion of he bee dance. [1] S. Baker, F. Dellaer, and I. Mahews. Aligning images incremenally backwards. Technical Repor CMU-RI-TR-01-03, CMU Roboics Insiue, Pisburgh, PA, Feb [2] M.C. Burl, M. Weber, and P. Perona. A probabilisic approach o objec recogniion using local phoomery and global geomery. Lecure Noes in CS, 1407:628, [3] J. Carpener, P. Clifford, and P. Fernhead. An improved paricle filer for non-linear problems. Technical repor, Deparmen of Saisics, Universiy of Oxford, [4] A.P. Dempser, N.M. Laird, and D.B. Rubin. Maximum likelihood from incomplee daa via he EM algorihm. Journal of he Royal Saisical Sociey, Series B, 39(1):1 38, [5] R. Fergus, P. Perona, and A. Zisserman. Objec class recogniion by unsupervised scale-invarian learning. In IEEE Conf. on Compuer Vision and Paern Recogniion (CVPR), June [6] N. Freias. Rao-Blackwellised paricle filering for faul diagnosis. IEEE Trans. Aerosp., [7] N.J. Gordon, D.J. Salmond, and A.F.M. Smih. Novel approach o nonlinear/non-gaussian Bayesian sae esimaion. IEE Procedings F, 140(2): , [8] M. Isard and A. Blake. Conour racking by sochasic propagaion of condiional densiy. In Eur. Conf. on Compuer Vision (ECCV), pages , [9] Z. Khan, T. Balch, and F. Dellaer. A Rao-Blackwellized paricle filer for EigenTracking. In IEEE Conf. on Compuer Vision and Paern Recogniion (CVPR), [10] B. D. Lucas and Takeo Kanade. An ieraive image regisraion echnique wih an applicaion in sereo vision. In Sevenh Inernaional Join Conference on Arificial Inelligence (IJCAI-81), pages , [11] G.J. McLachlan and T. Krishnan. The EM algorihm and exensions. Wiley series in probabiliy and saisics. John Wiley & Sons, [12] M. Monemerlo, S. Thrun, D. Koller, and B. Wegbrei. Fas- SLAM: A facored soluion o he simulaneous localizaion and mapping problem. In AAAI Na. Conf. on Arificial Inelligence, [13] K. Murphy and S. Russell. Rao-Blackwellised paricle filering for dynamic Bayesian neworks. In A. Douce, N. de Freias, and N. Gordon, ediors, Sequenial Mone Carlo Mehods in Pracice. Springer-Verlag, New York, January [14] D. Schulz, D. Fox, and J. Highower. People racking wih anonymous and ID-sensors using Rao-Blackwellised paricle filers. In Proceedings of IJCAJ, [15] Leonid Sigal, Sidharh Bhaia, Sefan Roh, Michael J. Black, and Michael Isard. Tracking loose-limbed people. In IEEE Conf. on Compuer Vision and Paern Recogniion (CVPR), pages , 2004.