Object Tracking using Ane Structure for Point. Correspondences. Subhashis Banerjee. Department of Computer Science and Engineering

Transcription

1 Object Trackin usin Ane Structure or Point Correspondences Gurmeet Sinh Manku Pankaj Jain y Amit Aarwal Lalit Kumar Subhashis Banerjee Department o Computer Science and Enineerin Indian Institute o Technoloy New Delhi suban@cseiitdernetin Abstract A new object trackin alorithm based on ane structure has been developed and it is shown that the perormance is better than that o a Kalman lter based correlation tracker The alorithm is ast, reliable, viewpoint invariant, and insensitive to occlusion and/or individual corner disappearance or reappearance Detailed experimental analysis on a lon real imae sequence is also presented 1 Introduction Trackin and computation o structure and motion have been treated in the past as two separate problems On one hand, traditional approaches to trackin have been based primarily on intensity correlation matchin, with the more sophisticated approaches employin recursive Kalman lterin [, ] or temporal consistency Little attempt has been made to impose structural constraints on the eatures bein tracked [, 8] On the other hand most approaches to structure and motion parameter estimation have assumed that reliable correspondences are already available [, 9] The successes o these approaches are crucially dependent on the correctness o the assumed correspondences Reid and Murray [, ] have developed a real-time object tracker which uses a constant imae velocity Kalman lter to establish the point correspondences across rames They use the point correspondences obtained usin their Kalman lter based correlation matcher to compute the D ane structure o the xation point and the ane camera projection equations in each rame, and use these to localize the xation point in each rame The method is ast, requirin O(n) oper- Present address: Dept Computer Science, University o Caliornia, Berkeley, USA y Present address: Dept Computer Science, Stanord University, Stanord, USA ations to track n corners in each rame, and the xation point can also be located with reasonable accuracy In this paper we urther improve the Kalman lter based trackin alorithm Our work attempts to uniy eature trackin and structure computation and provide simultaneous solutions to both, each o which aids the other We use the Kalman lter based method proposed by Reid and Murray to obtain the initial correspondences and subsequently impose the structure based constraints to obtain better correspondence results Such an approach improves upon the perormance o the Kalman lter based correlation tracker with minimal overheads and enhances the quality o correspondences makin the structure and motion computations more accurate Usin the structure and motion we localize the xation point in each rame more accurately, and the aze demand enerated is smooth and immune to occlusion or disappearance o the corners We show that usin our approach it is possible to obtain sinicantly smaller averae per-pixel error in the localization o points on the imae than what is obtained usin the Reid-Murray alorithm Such improved correspondence results may be useul not only or locatin xation points or trackin but also or any other application that uses structure, like object reconition, video indexin etc Further, we compute the basis (a- ne camera projection parameters) in each rame rom a lare number o corners and the method does not depend on the accuracy o the detection o ew particular corners In each rame we compute the correspondin basis and consequently no chane o basis is required The computation required in each rame is directly proportional to the number o corners tracked (O(n)), and consequently the alorithm can be implemented in realtime We assume that the object in view is underoin a- ne motion, which is more eneral than the customary

2 assumption o riid transormations [] We use the ane camera projective model [, ] and recover structure up to an arbitrary ane transormation The ane camera model is valid when the eld o view is small and the depth o the object is small compared to the viewin distance, which is expected in any oveal trackin application [, ] When the perspective eects are small, it is convenient to assume the parallel projection model o the ane camera which explicitly models the ambiuities Further, it is possible to detect when the ane camera approximation ails to hold In what ollows we describe our new approach to eature trackin In Sec, we describe the Kalman lter based corner trackin alorithm [] which we use to obtain the initial matches or our alorithm and discuss its limitations In In Sec, we describe our new alorithm or trackin usin ane structure In Sec we present trackin results on a lon real imae sequence and compare the perormance with the Kalman lter based trackin alorithm o [] Kalman Filter based Trackin (Reid and Murray) Kalman lters comprise a class o linear unbiased minimum-error covariance sequential state estimation alorithms A constant imae-velocity Kalman lter is used or each corner [] to be tracked The state vector or any corner, x(k) is iven by a x 1 matrix [x(k); y(k); _x(k); _y(k)] T State transition is modeled by x(k + 1) = Fx(k) + v(k)? u(k); where v(k) are iid Gaussian zero-mean, temporally uncorrelated noise with covariance Q, u(k) is the camera's displacement (eo-motion) rom time step k? 1 to k, and F = 1 t 1 t 1 1 Measurement is noisy and iven by a x 1 vector z(k) = Hx(k)+w(k), where w(k) are iid Gaussian, zero-mean, temporally uncorrelated noises with covariance R, and 1 H = 1 The covariance matrix R can be constructed by notin that the localization error in corner detection has the same variance as that o the Gaussian mask used by the corner detector The position o each corner is predicted in the new rame usin the Kalman lter prediction equation [1] and all corners ound within a search area whose size depends on the predicted covariance matrix P are crosscorrelated with the corner in the previous rame The best match, above a certain minimum threshold (typically ) is selected and the Kalman lter's state vector and state covariance is updated usin the standard equations [1] For each corner unmatched in the previous rame (Kalman lter not initialized) the position in the next rame is predicted usin the eo-motion o the camera I a match is ound successully by crosscorrelation then the Kalman lter or the corner is initialized The method yields several alse matches across rames This is because matchin is based on the correlation o intensities in a window around the corner which is inherently unreliable Moreover, the method tracks each corner individually, without reard to the interrelation o the eatures amon themselves, ie, the structure Invokin the assumptions o an ane camera model and ane motion o the object, it is possible to impose some constraints on the corners collectively The motivation or imposin such constraints is to be able to reject alse matches reported by the Kalman lter and also orce some more matches, thereby improvin the overall perormance and reliability Matchin usin Ane Structure We assume that the object in view is underoin a- ne motion and that the imaes in the successive views are ormed by ane projection In the ane projection model [] all projection rays are parallel and the epipoles are at innity resultin in parallel epipolar lines Given n point correspondences in two or more views, it is possible to determine the ane structure o the n point conuration [] and the ane structure is invariant to ane motion See [] or an exposition o the ane multiple views eometry We derive the ollowin constraint or matchin based on ane multiple views eometry: The point matches between the k th and the (k + 1) th views must be consistent with the ane structure computed up to the k th view In what ollows we briey describe the overall structure o our new alorithm and then describe some o the crucial procedures used in more detail 1 Overall alorithm Track mode = KALMAN or each successive imae in the sequence do Match usin Kalman lters; (Sec ) I Track mode = KALMAN (Comment: Kalman lter based trackin) Gaze Point = center o mass o matched corners; I Initialization o basis (Sec ) is successul

3 then set Track mode to AFFINE; else (Comment: Ane structure based trackin) I basis computation (Sec ) is unsuccessul then Set Track mode to KALMAN; Gaze Point = center o mass o matched corners; continue with the next rame; Compute ane structure;(sec ) Reject outliers usin ane structure; (Sec ) I Re-computation o basis is unsuccessul then Set Track mode to KALMAN; Gaze Point = center o mass o matched corners; continue with the next rame; Compute ane structure; Force matches usin ane structure; (Sec ) Locate the Gaze point usin its ane structure; (Sec ) Update Kalman lters or all matched corners; Initialization o basis and structure We use the Tomasi and Kanade [9] procedure to initialize the basis and the ane structure o the corners which have a match history o at least F rames by constructin a F P measurement matrix W iven below We use F = in our experiments W = x 1 1 x 1 : : : x 1 P y1 1 y 1 : : : y 1 P x 1 x : : : x P y1 y : : : y P where (x j i ; yj i ) are the centered coordinates o the i th corner in the j th rame Carryin out a sinular value decomposition o W yields its actorization into W = e 1 e e e 1 e e 1 : : : P 1 : : : P 1 : : : P (1) The rows o the rst matrix ive the ane bases in the F views and the second matrix ives the invariant ane structure o the P points The actorization o the observation matrix is the best rank approximation possible in a least square sense I the residual error is lare suestin a rank situation it indicates that the ane projective model is invalid A rank situation suests deeneracy - planar object or planar motion Basis Computation Consider all corners in the previous rame which have been matched and or which ane structure is known Compute the basis matrix h usin least squares methods such that h 11 h 1 h 1 h h 1 h h 1 h = x N 1 y1 N x N y N The oodness o t provides us with an estimate o how closely the observed matches ollow the theoretical model I time is at a premium then the re-computation o basis may be carried out only i the oodness o t turns out to be really bad The least squares procedure also returns the matrix h, which contains the variance o the correspondin elements o h Note that with the above procedure we always obtain a correspondin basis in each rame with respect to an invariant set o D ane coordinates Consequently no chane o basis is ever required Rejection o Outliers usin Ane structure Out-lier rejection is done by considerin all corners in the previous rame that have been matched and whose ane structure is known and predictin the position o correspondin corners in the current rame by projectin the ane structure I the discrepancy between the predicted and the matched position is more than what is predicted by h then we label the corner as unmatched For a corner with ane structure (,, ) the predicted position is: x = h 11 + h 1 + h 1 + h 1 y = h 1 + h + h + h Forcin o Matches usin Ane structure Consider all corners in the previous rame that have not been matched in the current rame but whose ane structure is known We predict the position o those corners in the current rame by a projection mechanism similar to that used in rejection o outliers and search or the match in the window dened about the predicted positions usin correlation matchin In addition, or all those corners whose ane structure is known rom the past rames and which have not been matched in

4 the previous rame, we orce matches by projectin the ane coordinates in the current rame and searchin or corners in the present rame in the neihborhood o the predicted position determined by h and the variance o the ane structure This procedure helps in recoverin corners which ail to appear in one or two rames and retain the structure inormation Ane Structure Computation Ane structure computation is similar to computation o basis except we have the h matrix and compute the ane structure For all the matched corners the ane structure (,, ) can be computed by least squares such that h 11 h 1 h 1 h 1 h h h 11 h 1 h 1 h 1 h h = x? h 1 y? h x? h 1 y? h where the primed variables reer to the previous rame Gaze Point Computation Determination o the aze point by xatin on the centroid o all the matched points results in jitter in the aze demand This is because the centroid tends to shit due to disappearance and re-appearance o eature points Instead, as suested in [], we determine the aze point by determinin the position o a xed point in the new rame usin the computed ane basis Let the invariant ane coordinates o the aze point be (,, ) and let h be the basis matrix The new aze point is then iven by x aze = h 11 + h 1 + h 1 + h 1 y aze = h 1 + h + h + h In our experiments, we observed that the errors involved in the computation o basis are minimum in the ourth component o the basis ie its oriin We have thereore set all the components o the ane coordinates o the aze point to zero which corresponds to the center o mass chosen in Tomasi Kanade actorization (Sec ) 8 Computational cost Consider the problem o trackin n corners usin the above alorithm Kalman lterin (Sec ), computin ane structure (Sec ), rejectin outliers usin ane structure (Sec ) and orcin matches usin ane structure (Sec ) are all O(n) operations and the least square computation o ane structure involves a xed size basis matrix The only non-linear steps in the above alorithm are the initialization o structure and basis usin the Tomasi-Kanade actorization (Sec ) and the basis computation (Sec ) These require least square solutions with matrices whose sizes are proportional to n However, these processes can also be made o constant time by usin only a xed number o corners We have ound in our experiments that choosin a xed number () o corners with maximum match ae or these computations does not aect the perormance o the overall alorithm Consequently, the overall trackin alorithm can be made to work in linear time Further, note that in our alorithm it is not necessary to handle the deenerate (D) and the nondeenerate (D) cases separately in all the procedures mentioned above Results We have implemented the corner trackin alorithm in real-time (at Hz) usin a SUN machine or rame rabbin and corner detection [] and a Pentium (1 M Hz) or trackin and robot control We have carried out extensive experimentation on some imae sequences In what ollows we present the trackin results in a 1 rames o-line sequence obtained at Hz The sequence consists o acial motion includin nod, shake and curl The sequence contains instances o both deenerate and non-deenerate motion In Fi 1 we present the trackin results in six rames Some statistics contrastin the perormance o the Kalman lter based alorithm aainst our method have been plotted in Fi 1 to The ane structure based alorithm nally matches rouhly the same number o corners in each rame as the Kalman lter alorithm (Fi ) But in each rame about 8-1% o the corners matched by the Kalman lter are rejected by the various staes o out-lier rejection and a similar number o new matches are ound by orcin matches usin ane structure This increases the quality o matchin as is evident rom the comparison o the averae per-pixel error in structure based prediction usin both methods (Fi ) Consequently, the aze point can be located more accurately In particular, we have observed that the quality o aze xation is sinicantly better than the Kalman lter based alorithm when there is an abrupt chane in the motion and the Kalman lters ail to predict accurately Also, the structure based method o aze xation, as suested in [], enerates a smoother aze demand as compared to center o mass trackin (Fi ) and results in a smaller camera speed (Fi ) In Fi we plot the D bases across the entire sequence to demonstrate that our method computes correspondin bases across the rames and no chane o basis is ever required The correspondin averae variance o the invariant ane coordinates o the point set is iven in Fi

5 Kalman Filter Based Trackin Aine Structure based Trackin 1 Total RMS Pixel Error Fiure : Averae per-pixel error in prediction usin structure and basis Fiure 1: Frame numbers 1, 1 and 1 o the sequence The ures on the let show the matched corners ater Kalman lterin The ures on the riht show the matches ater the entire alorithm The tail with each corner shows its displacement (scaled by a actor o ) The bi rectanle is the ovea under view Gaze 1 1 Aine Structure based Trackin (X) Aine Structure based Trackin (Y) Centre o Mass Trackin (X) Centre o Mass Trackin (Y) Fiure : X and Y coordinates o the Gaze point Total Number o Matches Kalman Filter Based Trackin Aine Structure based Trackin Gaze Demand 1 1 Aine Structure based Trackin Centre o Mass Trackin Fiure : Total number o matches Fiure : Camera speed

6 Fiure : The correspondin basis computed in Frame numbers,,, 8, 1 and 1 (let to riht and top to bottom) In this particular case the second and the third basis vectors nearly coincide in the imae projections For visual clarity a circle has been put at the end o the third basis vector Variance in Structure 1 Variance in Structure Conclusions We have presented a trackin alorithm based on a- ne structure which sinicantly improves the perormance over just Kalman lter based trackin with very little extra computational overheads The alorithm does not suer rom problems like the occlusion and disappearance o corners and never requires a chane o basis The basis computation is done rom a lare set o corners which leads to hiher accuracy The averae per-pixel error in prediction o imae positions o corners rom the computed structure and basis is also smaller Consequently, the aze demand is smoother and more accurate Moreover, such improved correspondences and structure and basis computation may be useul or other applications like object reconition The alorithm is ast and amenable to real-time implementation Acknowledments We wish to thank Vardhan Verma, Himanshu Nautiyal and Ashish Thusoo or their help in the course o this work Reerences [1] Y Bar-Shalom and T E Fortmann Trackin and Data Association Academic Press, 1988 [] A Blake, R Curwen, and A Zisserman A Framework or Spatio-temporal Control in the Trackin o Visual Contour Intl J Computer Vision, 11():1{ 1, 199 [] C Harris and M Stephens A Combined Corner and Ede Detector In Proc th Alvey Vision Con, paes 1{18, 1988 [] J J Koenderink and A J van Doorn Ane Structure rom Motion Journal o the Optical Society o America, Series A, 8:{8, 1991 [] J L Mundy and A Zisserman Geometric Invariance in Computer Vision MIT Press, 199 [] I D Reid and D W Murray Trackin Foveated Corner Clusters usin Ane Structure In Proc Intl Con Computer Vision, paes {8, 199 [] I D Reid and D W Murray Active Trackin o Foveated Feature Clusters usin Ane Structure Intl J Computer Vision, 18(1):1{, 199 [8] G Sudhir, S Banerjee, and A Zisserman Findin Point Correspondences in Motion Sequences Preservin Ane Structure In Proc British Machine Vision Con, 199 Also accepted or CVGIP: Imae Understandin [9] C Tomasi and T Kanade Shape and Motion rom Imae Streams under Orthoraphy: A Factorization Method Intl J Computer Vision, 9:1{1, 199 Fiure : Averae variance o ane structure coordinates