Tracking of Multiple Objects Using Optical Flow Based Multiscale Elastic Matching

Transcription

1 Trackng of Mutpe Objects Usng Optca Fow Based Mutscae Eastc Matchng ngzh Luo And Suchendra M. Bhandarkar Department of Computer Scence, The Unversty of Georga Athens, Georga , USA Abstract A nove hybrd regon-based and contour-based mutpe object trackng mode usng optca fow based eastc matchng s proposed. The proposed eastc matchng mode s genera n two sgnfcant ways. Frst, t s sutabe for trackng of both, rgd and deformabe objects. Second, t s sutabe for trackng usng both, fxed cameras and movng cameras snce the mode does not rey on background subtracton. The eastc matchng agorthm expots both, the spectra features and contour-based features of the tracked objects, makng t more robust and genera n the context of object trackng. The proposed eastc matchng agorthm uses a mutscae optca fow technque to compute the veocty fed. Ths prevents the mutscae eastc matchng agorthm from beng trapped n a oca optmum unke conventona eastc matchng agorthms that use a heurstc search procedure n the matchng process. The proposed eastc matchng based trackng framework s combned wth Kaman fter n our current experments. The mutscae eastc matchng agorthm s used to compute the veocty fed whch s then approxmated usng B- spne surfaces. The contro ponts of the B-spne surfaces are used drecty as the trackng varabes n a Kaman fterng mode. The B-spne approxmaton of the veocty fed s used to update the spectra features of the tracked objects n the Kaman fter mode. The dynamc nature of these spectra features are subsequenty used to reason about occuson. Expermenta resuts on trackng of mutpe objects n rea-tme vdeo are presented. 1 Introducton and Background Mutpe object trackng n dynamc scenes s chaengng n severa aspects. The frst chaenge arses from occusons, whch ncudes mutua occuson between foreground objects and occuson caused by background objects. When occuson occurs, some objects are partay or totay nvsbe. Ths makes t dffcut to accuratey ocaze the occuded object and track t contnuousy over severa mage frames. The second chaenge s the formuaton of an object mode that s capabe of handng object deformaton. The object mode shoud be abe to capture the most mportant and reevant nformaton about the object and factate fast and reabe trackng. The abty to dea wth occuson depends, to a great extent, on the object mode. The thrd chaenge s to meet the rea tme constrants of most trackng appcatons n the rea word. Fast and accurate object ocazaton over tme s the utmate objectve of a trackng system. Generay speakng, there exst three broad categores of object modes n the context of trackng: contour-based modes [1], [5], [7], [8], [23], regon-based modes [2], [3], [4], and feature pont-based modes [9], [10], [22]. The contour-based mode does not encode any coor or edge nformaton wthn the nteror of the object. The contour nformaton by tsef s not enough to hande genera nstances of occuson. In the absence of any spectra nformaton, feature pont-based trackng methods are easy dstracted by nosy feature ponts n the background and are, by ther very nature, mted to objects rch n feature ponts. A regon-based object mode s more sutabe when occuson s present snce t encodes spectra nformaton. Occuson handng s another mportant ssue that arses n mutpe object trackng systems and s cosey ntertwned wth the choce of the object mode. In the case of contour-based modes, the robustness of the occuson reasonng s hghy dependent on the quaty of object segmentaton and typcay, ony smpe cases are we handed. Koer et a. [7] propose a depth-based occuson reasonng scheme based on a geometrc mode whch computes the projecton of the movng object n the 3-D word onto the mage pane. However, ther method s ony appcabe n stuatons where the object moves n the vertca drecton n the mage pane and hence not vad for more genera trackng scenaros. Aso, regon-based object modes that rey prmary on coor/gray eve hstograms of the movng regons are not we suted to hande occuson snce no object shape nformaton s avaabe. McKenna et a. [3] use smpe correspondence anayss to do trackng when the 1

2 tracked objects are ndependent of each other. In the event of occuson, ther technque can ony compute a statstca probabty that a pxe of a gven coor beongs to a specfc object whch s not usefu for the purpose of accurate object ocazaton. Eastc matchng has been used wdey for deformabe object recognton [6] and trackng [15]. Eastc matchng s potentay we suted for trackng of deformabe objects. Snce eastc matchng expots both, the spectra nformaton wthn and the spata coherence constrant amongst the object pxes, t can be easy ntegrated wth a regon-based object mode. Another advantage of eastc matchng s that the trackng resuts are not dependent on the accuracy of the background subtracton used to extract the movng objects. Ths makes t possbe to track movng objects wth a movng camera. However, most exstng eastc matchng methods use a heurstc search agorthm wthn the matchng procedure [6, 15, 16, 17] and are hence prone to get trapped n a oca optmum. Optca fow methods expot the mage gradent nformaton to compute the veocty fed, makng t possbe to avod a oca optmum wthn the eastc matchng procedure. However, most optca fow methods use ony gray scae mages and the veoctes at each mage pxe are computed ndependenty, thus resutng n a nosy veocty fed. In a homogeneous mage regon, the mage gradent vaue s a constant (cose to zero) resutng n ambguous vaues for the veocty fed. Aso, most optca fow agorthms ensure ony oca consstency of the veocty fed snce the mage gradent s computed wthn a oca wndow. Very few optca fow agorthms usng mutpe-channe (mut-spectra) mages have been reported n the research terature. Markandey et a. [19] and Goand et a. [18] have proposed optca fow agorthms that use mages wth 2 and 3 channes respectvey. In ths paper, the optca fow agorthm s generazed to use mages wth an arbtrary number of channes. The generazed optca fow computaton procedure s used wthn a hybrd regon- and contourbased eastc matchng scheme. Snce the spata coherence constrant s mposed n the eastc matchng procedure, the ambgutes n the computaton of the veocty fed n a ocay homogenous regon of the mage s resoved. In order to ncrease the range of consstency of the optca fow agorthm, two schemes are adopted. The frst scheme empoys an teratve mutscae Lucas-Kanade agorthm for optca fow computaton usng a pyramda mage structure. The second scheme uses a Kaman fterng agorthm to predct the veocty fed whch s subsequenty refned by performng a search n the neghborhood of the predcted ocaton. The ncorporaton of mutscae eastc matchng wthn the Kaman fterng framework has two advantages: (a) the nherent mtaton of the near Kaman fter whch assumes that the trackng varabes have Gaussan dstrbutons s ad- Fgure 1. The trackng scheme dresed, and (b) changes n object sze n the mage (scae changes) are effectvey deat wth. The overa system s depcted n Fgure 1. Gven an nta object poston and ts veocty fed, the Kaman fterng agorthm predcts the new veocty fed for the next stage. An teratve eastc matchng agorthm uses the predcted nta veocty and the computed optca fow to determne the new veocty fed for the object. The teratve eastc matchng agorthm frst maps the predcted veocty fed to a manuay chosen eve n the pyramd, and determnes the actua veocty at eve usng eastc matchng. Then the veocty computed at eve s mapped to eve 1 and used as the nta veocty at eve 1. Ths procedure s performed teratvey unt the veocty at eve 0 s obtaned. An occuson reasonng and oca tunng procedure s performed at each eve to generate and verfy the occuson hypothess at each pxe ocaton and refne the veocty fed. The new veocty fed s then used to update the contour tempate and object mode. The new veocty fed s approxmated usng B-spne surfaces, whch are then used to update the status of the Kaman fter. The updated Kaman fter s used to predct the veocty fed for the new stage. 2 The Mutscae and Mutresouton Object Mode In order to compute a mutscae representaton of the movng objects, each of the orgna mages s encoded as L eves of Gaussan pyramd and Lapacan pyramd. The mage at eve n the Gaussan pyramd s denoted by I. The Gaussan pyramd mage at eve + 1 s computed as I +1 (x,y) = G(x,y;σ) I (2x,2y), where G(x,y;σ) s the Gaussan smoothng operator, s the convouton operator and σ s the scae parameter (σ = 1 s used n our case). Ths procedure, termed as the REDUCE functon [21], smoothy sampes the orgna mage wth a sampng nterva of ength 2 aong the x axs and y axs. Its nverse functon, defned as Î (x,y) = G(x,y;σ) I +1 (x/2,y/2), sampes the mage at eve + 1 back to. The Lapacan 2

3 pyramd mage at eve s gven by P = I Î. The pyramd mage at the frst eve ( = 0) has the same sze as the orgna mage. If the sze of the orgna mage s (W,H), where W s the mage wdth and H the mage heght, the mage sze at eve n the pyramd s (W/2,H/2 ). An RGB coor mage s encoded as a sx-channe mage at each eve, where three of the mages (R,G and B) are obtaned from the Gaussan pyramd and the other three mages (R,G and B) are obtaned from the Lapacan pyramd. Object O () at eve s represented by a network of ponts O () = ({j },K,L, {Tj }), where 1 j N and N s the number of ponts used to represent the object. K s the connectvty matrx wth dmenson N N such that kj = 1 f ponts and j are connected, and kj = 0 otherwse. In practce, k j = 1 f pont s one of the neghborng ponts of pont j. The matrx K s symmetrc and k = 0. L s the connectvty matrx for the boundary ponts. If and j are contour ponts and are connected to each other wthn a wndow of predetermned sze, then j = 1, ese j = 0. The matrx L s aso symmetrc. {Tj } s the object contour tempate at eve. Each contour pont n {j } has one and ony one correspondng pont n contour tempate {Tj }, and T j s not vad f j s not a contour pont. In our current work, the boundary tempate mode s obtaned usng B-spne contour fttng as descrbed n Secton 3. For each pont j, c j = c( j ) represents the feature vector assocated wth pont j and (Σ2 j ) = Σ 2 (j ) represents the covarance matrx of the feature vector at pont j. Snce c j s a vector comprsng of the vaues from each of the sx channes and t s 6-dmensona. Both c j and (Σ 2 j ) are temporay varyng vaues and are updated onne. When there s nsuffcent tempora nformaton for a pont, (Σ 2 j ) s ntazed to a defaut vaue (Σ 2 0). An onne updatng scheme for c j and (Σ2 j ) s gven n Secton 4.4. V = {vj } s the veocty fed assocated wth the object where vj = (v x(j ),v y(j )) s the veocty at pont j. Instead of computng the veocty for each pont of the object, the desgned object mode dynamcay adjusts the sampng ntervas of the ponts used for veocty computaton n order to baance the computatona oad. Gven N 0, the desred number of ponts used for computaton, and N, the tota number of ponts beongng to the object, the sampng nterva used to obtan the ponts for veocty computaton s determned as max( N/N 0,1). Smpe nterpoaton s used to determne the veocty for the ponts whch are not samped. The contour tempate Tj s used to gude the search process. 3 Contour Tempate The contour tempate s pre-traned for the purpose of object trackng. Note that the object contour s mpcty ncuded n the object mode. Gven the contour tranng set {C }, each contour C s frst normazed to a specfed sze (eg for face trackng) as shown n Fgure 2(a)(1). The contours n the tranng set are obtaned manuay from the tranng vdeos. The spata dstrbuton of contour ponts s equazed usng nterpoaton such that the resutng contour ponts are unformy dstrbuted aong the contour. The startng pont on the contour s chosen to be agned wth centrod pont of the contour, as shown n Fgure 2(a)(2) where O s the centrod pont and S s the chosen starng pont. Let C = (c,0,...,c,n 1 ) be the contour obtaned from C after normazaton, equazaton and agnment. A vector of B-spne contro ponts {T } = (t,0,...,t,m 1 ) s obtaned for each contour C n the tranng set as shown n Fgure 2(a)(3) where the pont marked n red represents the frst contro pont. M = 10 s used n the face trackng experment. Fgure 2(a)(4) depcts the contour restored usng the B-spne contro ponts. The contro pont vectors {T } are modeed as a mxture of K Gaussan dstrbutons for a predetermned vaue of K. The correspondng K custers of contro ponts are determned usng the K-means agorthm. The dstance between the contro pont vectors T and T j s measured usng the Eucdean metrc d(t, T j ) = T T j. As shown n Fgure 2(b), 352 faces are randomy extracted from the vdeo data for tranng n the face trackng experment. The contro ponts obtaned from the normazed faces are shown n Fgure 2(b)(1). Wth the K-means agorthm, 3 custers are obtaned from the 352 contro pont vectors. The 3 contours generated from the 3 custer centers of the contro pont vectors are shown n Fgure 2(b) (2) (3) and (4) where the number wthn each contour represents the custer cardnaty. Each custer s represented as a Gaussan dstrbuton N(T k,σ k ). Gven an object contour C obtaned va eastc matchng, ts contour tempate s determned usng the foowng procedure. The contour C s frst normazed, equazed and agned, and ts B-spne contro pont vector T s determned usng the tranng procedure descrbed above. Q contro pont vectors are randomy generated from each of the K custers resutng n a tota of KQ contro pont vectors. The random generaton functon uses the Gaussan dstrbuton of each custer. Among the KQ contro pont vectors, the one at mnmum dstance to T s chosen as the contour tempate. The contro vaue for each pont n C s computed and used to compute ts correspondng ponts wth the contour tempate. These ponts are scaed back to ther orgna sze and used n the eastc matchng agorthm n the next step. 3

4 (a) The B-spne contour (b) Contour custerng Fgure 2. Contour tempate 4 Pyramda Eastc Matchng Mode Inspred by the pyramda mpementaton of Lucas- Kanade mage regstraton agorthm [20], a mutpe scae eastc matchng mode s desgned for regon-based trackng. However, n the orgna agorthm [20] ony grayscae mages are used, each feature pont s tracked ndependenty and mage nterpoaton s used for veocty computaton, whch ncurs hgh computaton oad. In the proposed scheme, the orgna agorthm s generazed to use mages wth any number of channes. In partcuar, 6-channe mages are used as descrbed n the object mode. The mposton of the spata coherence constrant suppresses the random nose n the veocty fed computaton, and the use of mutpe scaes ensures that an optma veocty fed can be obtaned. Instead of usng mage nterpoaton, a oca tunng agorthm s used to obtan sub-pxe accuracy. The pyramda eastc matchng agorthm can be generazed as foows. Gven object O (t) at tme t and eve, ts nta veocty fed (V 0 ), and the new mage I (x,y;t+1), the objectve of eastc matchng agorthm s to refne the veocty fed V of the tracked object(s), such that the energy functon defned n equaton (1) s mnmzed. n whch E = ǫ (v ) (1) =1 ǫ (v ) = g [c j I ( j + (v0 ) + v )]2 j O( ) } {{ } feature matchng + γ j ( + (v0 ) + v T ) ( j + (v0 j ) + vj T j ) 2 } {{ } contour constrants + β k j (v0 ) + v (v0 j ) v j 2 } {{ } veocty constrants (2) where = g( ) s the occuson hypothess for pont of the gven object such that g = 0 f s occuded, and g = 1 otherwse, and (v0 j ) = (vx 0 j,vy 0 j ) s the nta veocty at pont j. In order to avod the need for mage nterpoaton, each vaue n the nta veocty (V 0 ) s actuay ts nearest nteger vaue. O(x ) s the set of pxes n a wndow centered at pont and v j s the ncrementa veocty. The frst part n equaton (2) measures the feature match of each pont of the object wth the new mage, whch requres that the correspondng mage pont has a smar feature vector n order to mnmze the energy functon. The second part n equaton (2) defnes the contour constrant, whch requres that the neghborng contour ponts have smar dspacement vaues from the contour tempate n order to mnmze the energy functon. In cases where the contour tempate s not avaabe, Tj can be smpy set to (0,0) for a j, whch s equvaent to a smoothness constrant mposed on the object contour. The thrd part n equaton (2) mposes spata coherence on the veocty fed, whch requres the veocty of neghborng pxes to be cose to each other n order to mnmze the energy functon. The parameter β contros the eastcty of the object. Equaton (1) s mnmzed when E / (V ) τ = 0, whch s equvaent to: E (v = 2g )τ [cj I (j + (v0 ) + v I ( )] j + (v0 ) + v ) j O( ) (v )τ + 4γ j ( j + T T j + (v0 ) (v j 0 ) + v v j )τ + 4β kj (v v j + (v0 ) (vj 0 ) ) τ (3) Note that the above equaton s obtaned under the assumpton that K and L are symmetrc. By usng Tayor expanson, I ( j + (v0 ) + v ) I ( j + (v0 ) ) + I v j (v ) τ (4) n whch, Iv j = (Ix j,iy j ) s the gradent vector at ocaton j + (v0 ). Ix = [I (x + 1,y) I (x 1,y)]/2 and Iy = [I (x,y + 1) I (x,y 1)]/2. Based on these gradent equatons, the Tayor expanson n equaton (4) s vad when vx 1 and vy 1. For convenence, et δij = c j I (j + (v0 ) ). Equaton (3) can be rewrtten as: E (v = g )τ (I v ) τ I j v (v j )τ g δi j (I v ) τ j j O( ) j O( ) + 2γ j ( j + T T j + ((v0 ) (vj 0 ) + v v j )τ + 2β kj ((v )τ (vj )τ + ((v 0 ) ) τ ((vj 0 ) ) τ ) (5) The dervaton of equaton (5) takes advantage of the fact that I ( j + v 0 + v )/ (v )τ = (I v j ) τ and 4

5 I v j (v )τ (I v j ) τ = (I v j ) τ I v j (v )τ. Equaton (5) s equvaent to equaton (6) and equaton (7) gven beow: E 2 vx = [2k β + 2 γ + g 2 Ix ]v j x j O( ) + g Ix I j y v j y 2β k j v x j j O( ) g N δij I x + 2β k j j ((v0 x ) (v x 0 ) ) j j O( ) + 2γ j (x x j + Tx Tx j + (v0 x+ ) (v x 0 ) ) (6) j E 2 v y = [2k β + 2 γ + g 2 Iy ]v j y j O( ) + g Ix I j y v j x 2β k j v y j j O( ) N g δij I y + 2β k j j ((v0 y ) (v y 0 ) ) j j O( ) + 2γ j (y y j + Ty Ty j + (v0 y+ ) (v y 0 ) ) (7) j By ettng a E / v x = 0 and E / v y = 0, a system of near equatons descrbng the ncrementa veocty fed V can be obtaned n the form of A(V ) τ = b, whch can be soved wth the LU decomposton method. The matrx A s gven by: (Iv ) τ (I 1 v ) + 2(βk γ 1 )I (βk 1,N + γk 1,N )I 0 2(βk 21 + γk 21 )I (βk 2,N + γk 2,N )I (βk N,1 + γk N,1 )I 0... (Iv ) τ (I N v ) + 2(βk N N + γ N )I 0 (8) ( 1 0 where I 0 = 0 1 Vector b s gven by: ), k = j k j and = j j. b = g δij (I v ) τ 2 βk j j (((v0 ) ) τ ((vj 0 ) ) τ ) j O( ) 2 γ j ( j + T T j + ((v0 ) (v j 0 ) ) τ ) (9) The veocty fed V 0 s ntazed under one of the foowng stuatons: when the object trackng procedure s ntazed, the veocty fed s assumed to be a 0. The pyramda eastc matchng agorthm starts at a gven eve (eg. = 2) n the pyramd to compute the veocty fed. When the veocty eve at eve s known, t can be mapped to eve 1. The veocty v at pont (x,y) maps to pont (2x,2y) and ts vaue at eve 1 s 2v. The thrd stuaton s when the veocty fed n prevous frames (t t 0 ) s known, n whch case the Kaman fter s used to predct the veocty fed at tme t Usng Images wth Any Number of Channes The aforementoned pyramda eastc matchng agorthm assumes that a snge channe mage s used. It s easy to show that t can be extended to nput mages wth any number of channes. Suppose the mutpe channe mage s gven by I (x,y) = ( 1 (x,y),..., D (x,y)), where D s the number of channes. For a smpe grayscae mage, D = 1. In the case of an RGB mage, D = 3. As mentoned n the descrpton of the proposed object mode, we use feature mages wth D = 6 n our experments. For mutpe channe mages, equaton (2) can be rewrtten as: ǫ (v ) = g D α d [c N j,d I d ( j + v )]2 + β k j v v j 2 j O( ) d=1 + γ j + (v0 ) + v T ( j + (v0 j ) + vj T j ) 2 (10) n whch, α d s the coeffcent of dmenson d. Its dfferenta equaton s gven by: E 2 (v = g D d )τ α d ((Iv ) (v j )τ δ(ij d ) )((Iv d ) ) τ j d=1 j O( ) + 2γ j ( j + T T j + ((v0 ) (v j 0 ) + v v j )τ ) + 2β kj (v v j + (v0 ) (vj 0 ) )) τ (11) 4.2 Loca Tunng Note that equaton (5) s obtaned by assumng the Tayor expanson gven n equaton (4) s vad. That s, the ncrementa veocty v shoud be sma at each pont of the object (eg. v x < 1 and v y < 1). In other words, the nta veocty needs to be cose to the actua veocty. By usng a pyramda mage structure, the above condton s met f the eastc matchng agorthm starts at a hgher eve n the pyramd where the veoctes aong the axs and Y axs are sma enough, or f the gven nta veocty s cose enough to the actua veocty. In addton to these methods, an teratve oca tunng agorthm s desgned to adjust the veocty ocay, such that the fna ncrementa veocty at each pont s sma enough. Ths oca tunng agorthm s especay usefu when ony some of the object ponts have veocty vaues arger than 1 aong the axs or the Y axs. The oca tunng agorthm s descrbed beow. (1) For pont, assume every other v j s fxed for j, then v can be obtaned by sovng the bnary near equatons (6) and (7). (2) If the ncrementa vaue v x > 1, then the new vaue vx 0 = vx+sgn(v 0 x ). If the ncrementa vaue v y > 1, then the new vaue vy 0 = vy 0 + sgn(v y ) where sgn(x) = 1 f x > 0, and sgn(x) = 1 f x < 0. 5

6 (3) Repeat steps (1) and (2) unt a the ncrementa vaues v x < 1 and v y < 1, or the maxmum number of teratons s met. The fna veocty for each pont s v 0 + v. When the occuson hypothess changes at some of the ponts, the oca tunng agorthm s aso used to recompute the veocty at those ponts. 4.3 Occuson Reasonng When occuson exsts, t s necessary to update the occuson hypothess for each pont when ts veocty s avaabe. A reasonabe assumpton about occuson s that an object s occuded graduay nstead of suddeny. It s aso assumed that an occuded object becomes unoccuded graduay nstead of suddeny. Therefore, ony those ponts near the object boundary or the boundary of the occuded area are chosen as canddates to be updated. In order to decde whether a pont s occuded or not, the Mahaanobs dstance between the feature vector assocated wth the countour pont c(x,y) and the feature vector assocated wth the correspondng mage pont I(x+v x,y+v y ) s computed as d = [c(x,y) I(x+v x,y+v y )]Σ 1 [c(x,y) I(x+v x,y+ v y )] τ. If d s above a certan threshod, ths pont s cassfed as occuded, otherwse t s cassfed as unoccuded. When mutpe objects correspond to the same pont n the mage, ony one object can be vsbe at ths mage pont. The object wth the mnmum dstance d to ths pont n the mage s cassfed as the vsbe object at ths mage pont. After the occuson reasonng s performed, the resutng nformaton s fed back to the eastc matchng agorthm, and the tunng agorthm s used to adjust the veocty fed ocay. At most two teratons are needed for the occuson hypothess to be updated and the resutng nformaton fed back to eastc matchng agorthm. 4.4 Adaptaton of the Object Tempate As an object moves, ts shape and coor features change dynamcay. Thus, the object shape mode and the coor features of the ponts on the object need to be updated at each teraton. For a pont whch s not occuded, ts feature pont s updated as: c(, t + 1) = c(, t) + ρ[i( + v ; t + 1) c(, t)], Σ 2 ( ; t + 1) = Σ 2 ( ; t) + ρ[(i k ( + v ; t + 1) c k ( ; t))(i k ( + v ; t + 1) c k ( ; t)) τ Σ 2 (, t)], where ρ s a gven earnng rate. 5 Veocty Fed Approxmaton Usng B- spne Surfaces The eastc matchng agorthm yeds a mappng whch mnmzes an energy functon that takes nto account both, feature smarty and shape dstorton durng trackng. The computed mappng determnes the dspacement of each pont aong the and Y axes,.e. the veocty of each pont. However, ths mappng may aso yed some nosy and fase matches that do not refect the actua moton of the object. Hence B-spne surfaces are used to smooth the veocty fed and suppress the effect of the nosy and fase matches. Provded the veocty n V = {v k } = {(v k,x,v k,y )} s known, the foowng procedure s used to determne the N N Y B-spne contro ponts n order to approxmate V. For each pont ocaton k = (x k,y k ) of the object, the correspondng B-spne contro parameter (û k, ˆv k ) s estmated as: (û k, ˆv k ) = ((N m + 1) x k /W, (N Y n + 1) y k /H) (12) where W and H are the wdth and heght of the object respectvey. The estmated veocty component aong the axs ˆv k,x s expressed n terms of (û k, ˆv k ) as foows: m 1 n 1 ˆv k,x (û k, ˆv k ) = d x =0 j=0 k,j N m (u1 k )Nn j (v1 k ) (13) k where the d,j s are the contro ponts whch determne the assocaton of a gven pont on the B-spne surface wth the contro parameters (u,v), N m (u) and Nj n(v) are the bass functons aong the u and v axes respectvey, and n and m are the orders of the B-spne (m = n = 4 n our case). Here k = + u0 k, j k = j + vk 0, (u0 k,v0 k ) = ( û k, ˆv k ), and (u 1 k,v1 k ) = (û k u 0 k, ˆv k vk 0 ). Equaton (13) can be further generazed as ˆv k,x (û k, ˆv k ) = N NY =0 j=0 dx,j B k(,j), where B k (,j) = N m 1 (û 1 û 0 k )Nn 1 k j ûk(ˆv 1 0 k ), f û k < û k + m 1 and ˆv k j < ˆv k + n 1; B k (,j) = 0 otherwse. For each pont assocated wth an object, mnmzaton of the foowng objectve functon s used to determne the vaues of the L = N N Y contro ponts: E = v k,x ˆv k,x (û k, ˆv k ) 2 (14) k=1 where (û k, ˆv k ) are the estmated contro parameters computed usng equaton (12), and N s the tota number of ponts of the object. The mnmzaton entas sovng a system of equatons gven by E/ d x = 0, whch, n turn, can be represented by equaton Ad x = b x, where A s gven by: Nk=1 B k (0, 0)B k (0, 0)... Nk=1 B k (0, 0)B k (N, N Y ) Nk=1 B k (0, 1)B k (0, 0)... Nk=1 B k (0, 1)B k (N, N Y ) Nk=1 B k (N, N Y )B k (0, 0)... Nk=1 B k (N, N Y )B k (N, N Y ) and b x = ( N k=1 B k(0,0)v k,x, N k=1 B k(0,1)v k,x,..., N k=1 B k(n,n Y )v k,x ) τ. The system of equaton can be soved usng LU decomposton. The vaue of N and N Y s usuay sma for rgd object trackng. In our experments, 4 4 contro ponts can approxmate the veocty fed n the mage pane resutng 6

7 from 3-D movement of a panar object (transaton, rotaton or a combnaton of the two) wth neggby sma mean squared error (MSE). Athough we have not examned the MSE resutng from the approxmaton, usng 4 4 contro ponts, of the veocty fed n the mage pane resutng from 3-D movement of a 3-D object, our experments on human trackng yed very good resuts. Snce an eastc object wth restrcted deformaton can be usuay approxmated by a rgd object wth artcuated moton, the resutng veocty fed can st be approxmated usng 4 4 contro ponts. More contro ponts are necessary ony for modeng compex and/or abrupt moton and deformaton of a hghy eastc object. To further reduce the computatona compexty, the vaues of N m and N n can be precomputed and stored n a ookup tabe. (a) Zoom n (c) Rotaton (b) Zoom out (d) Iumnaton change Fgure 3. Zoomng, rotaton, umnaton change 6 Veocty Estmaton Mode (a) Face scang (b) Face occuson A Kaman fter s used to estmate/predct the veocty fed of an object. Note that the term veocty fed, n the Kaman fter agorthm, actuay denotes the contro pont vaues resutng from the B-spne approxmaton of the veocty fed. The canonca Kaman fter used n ths paper can be descrbed usng the foowng equatons: ˆV d,k+1 = ˆV + d,k + q k (15) Z k = ˆV d,k + v k (16) where V d s the estmated/predcted veocty fed, and Z k s the actuay measured veocty fed. Equaton (15) represents the pror estmaton of V d whereas equaton (16) descrbes the near reaton between the estmated V d and the actuay measured veocty fed Z k. Varabes q k and v k represent random nose n the pror estmaton and actua measurement of the veocty fed respectvey. Both q k and v k are modeed as Gaussan whte nose wth dstrbutons N(0, Q) and N(0, R) respectvey. 7 Expermenta Resuts The proposed trackng agorthm has been apped to varous trackng scenaros. In our current experment, the objects are ntazed manuay by abeng ther contours over the frst mage. Fgure 3(a) shows the snapshots of the trackng of an eraser n a vdeo whe the camera s zoomng n. Fgure 3(b) shows the snapshots of the trackng of an eraser n a vdeo whe the camera s zoomng out. Fgure 3(c) shows the snapshots whe the tracked object s rotatng n the mage pane and Fgure 3(d) shows the snapshots whe the scene s subject to goba change n umnaton. Fgure 3 shows that the proposed trackng agorthm can hande arge changes n object sze (.e., sgnfcant scae changes) (c) Two face trackng Fgure 4. Face Trackng and hande object rotaton and scene umnaton change due to the adaptve nature of the object tempate and the robustness of the features used. Experments are aso conducted on face trackng. Fgure 4(a) shows the trackng resut when the tracked face exhbts arge scae changes n the vdeo. Fgure 4(b) shows the trackng resut n the presence of occuson thus demonstratng that the occuson reasonng s very robust n handng occusons. Fgure 4(c) shows the trackng resuts on two faces where one face occudes another. The varous trackng resuts can be vewed n the vdeo accompanyng ths paper. 8 Concusons A nove hybrd regon-based and contour-based mutpe object trackng mode usng eastc matchng s proposed. The eastc matchng agorthm expots both, the spectra features and contour-based features of the tracked objects, makng t more robust and genera n the context of object trackng. The proposed eastc matchng agorthm uses a mutscae optca fow computaton agorthm to compute the veocty fed. Ths prevents the mutscae eastc matchng agorthm from beng trapped n a oca optmum unke conventona eastc matchng agorthms that use a heurstc search procedure n the matchng process. The proposed trackng framework can be vewed as a generazaton of the tradtona near Kaman fter where the 7

8 mutscae eastc matchng agorthm s used to compute the veocty fed whch s then approxmated usng B-spne surfaces. The contro ponts of the B-spne surfaces are drecty used as the trackng varabes n a Kaman fterng mode. The B-spne approxmaton of the veocty fed s used to update the spectra features of the tracked objects n the Kaman fter mode. The dynamc nature of these spectra features are subsequenty used to reason about occuson. Expermenta resuts on trackng of mutpe objects n rea-tme vdeo are presented. Expermenta resuts show that the proposed agorthm s very effcent n handng occusons and changes n scae and umnaton. However, t s observed that a snge object hypothess s not suffcent to hande a possbe trackng scenaros. Wthout a sutabe foreground or background mode, ths scheme s not sutabe for ong term trackng. Future work w ntegrate the optca fow based eastc matchng mode wth a foreground object detecton agorthm and partce fterng agorthm to acheve robust and consstent trackng. References [1] P. L, T. Zhang, and A.E.C. Pece, Vsua contour trackng based on partce fters, Img. Vs. Comput., Vo. 21, 2003, pp [2] M. Isard and J. MacCormck, BraMBLe: A Bayesan Mutpe-Bob Tracker, Proc. ICCV, Vancouver, Canada, Vo. 2, Juy 2001, pp [3] S.J. McKenna, S. Jabr, and Z. Durc, A. Rosenfed, H. Wechser, Trackng Groups of Peope, CVIU, Vo. 80, 2000, pp [4] K. Nummaro, E. Koer-Meer, and L.V. Goo, An adaptve coor-based partce fter, Img. Vs. Comput., Vo. 21, No. 1, 2003, pp [5] A. Bake, R. Curwen, and A. Zsserman, A framework for spato-tempora contro n the trackng of vsua contours, IJCV, Vo. 11, No. 2, 1993, pp [6] M. Lades, J. Vorbruggen, J. Buhmann, J. Lange, C.V.D. Maburg, and R. Wurtz, Dstorton Invarant Object Recognton n the Dynamc Lnk Archtecture, IEEE Trans. Comp., Vo. 42, No. 3, 1992, pp [7] D. Koer, J.W. Weber and J. Mak, Robust Mutpe Car Trackng wth Occuson Reasonng. Proc. ECCV Stockhom, Sweden, 1994, pp [8] D. Terzopouos and R. Szesk, Trackng wth Kaman Snakes, n Actve Vson, MIT Press, Cambrdge, MA, USA, 1993, pp [9] W.J. Ruckdge, Locatng Objects Usng the Hausdorff Dstance, Proc. ICCV, Massachusetts USA. June, 1995, pp [10] S. Mak, G. Roth, and C. McDonad, Robust Corner Trackng for Rea-Tme Augmented Reaty, Proc. Vson Interface, Cagary, Aberta, Cananda, May, 2002, pp [11] V. Lepett, J. Pet, and P. Fua, Pont Matchng as a Cassfcaton Probem for Fast and Robust Object Pose Estmaton. Proc. IEEE Conf. CVPR, Washngton DC, USA, June Vo. 2, pp [12] D. Chung, W.J. MacLean, and S. Dcknson, Integratng Regon and Boundary Informaton for Improved Spata Coherence n Object Trackng, Proc. IEEE Conf. CVPR, Washngton DC, USA, June [13] D. Comancu, V. Ramesh, and P. Meer, Kerne-Based Object Trackng, IEEE Trans. on Pattern Anayss and Machne Integence, May 2003, Vo. 25, No. 5, pp [14] J. Suvan, A. Bake, M. Isard, and J. MacCormck, Object Locazaton by Bayesan Correaton, Proc. ICCV, Corfu, Greece, September 1999, Vo.2, pp [15] R. Bajcsy, and S. Kovacc, Mutresouton Eastc Matchng, Computer Vson, Graphcs, and Image Processng, Apr 1989, Vo. 46, Issue 1, pp [16] C. Chuang, and C. Kuo, Waveet Deformabe Mode for Shape Descrpton and Mutscae Eastc Matchng, SPIE s Symposum on Vsua Communcatons and Image Processng, Cambrdge, MA, Nov [17] M. Tang, and S. Ma, A Fast Agorthm of Mutresouton Eastc Matchng, The 10th Scandnavan Conference on Image Anayss, Lappeenranta, Fnand, June [18] P. Goand, and A.M. Brucksten. Moton from coor. Center for Integent Systems TR #9513, Computer Scence Department, Technon, Hafa, August [19] V. Markandey, and B.E. Fnchbaugh. Mutspectra constrants for Optca Fow Computaton. Proc. 3rd Int. Conf. on Computer Vson. Osaka, Japan, December pp [20] J. Bouguet, Pyramda Impementaton of the Lucas-Kanade Feature Tracker: Descrpton of the agorthm, OpenCV Documentaton, [21] P.J. Burt, and E.H. Adeson, The Lapacan Pyramd as a Compact Image Code, IEEE. Trans. on Communcatons, Apr 1983, Vo. COM-31, No.4, pp [22] K. Shafque, and M. Shah, A Nonteratve Greedy Agorthm for Mutframe Pont Correspondence, IEEE. Trans. PAMI, Januray 2005, Vo. 27, No. 1, pp [23] A. Ymaz,. L, and M. Shah, Contour-Based Object Trackng wth Occuson Handng n Vdeo Acqured Usng Mobe Cameras, IEEE. Trans. PAMI, Novemebr 2004, Vo. 26, No. 11, pp