/99 $10.00 (c) 1999 IEEE

Transcription

1 The PDAF Based Active Contour N. Peterfreund Center for Engineering Systems Advanced Research Oak Ridge National Laboratory, P.O.Box 8 Oak Ridge, TN, vp@ornl.gov Abstract We present a new active contour model in spatiovelocity space which is based on the probability data association lter (PDAF) approach. It employs a directional-based measurement model of image potential and of optical-ow along the contour points. The proposed directional approach of measurements is the basis for the velocity-based discrimination between measurements of the object to that of image clutter. The model chooses the appropriate measurements which are most consistent with previous estimation of motion in the sense of the PDAF approach. In order to obtain reliable measurements of image motion and of gradientbased image potential, we propose a directional smoothing operator which is the basis for discriminating the objects measurements from that of image clutter. The method was applied toreal world tracking problems such as a walking leg and a waving hand. Introduction Visual tracking of nonrigid objects with active contour models has been the subject of intensive research during the past years (e.g. [8], [], [], [] and [7]). One of the major problems in tracking with these models arise in the presence of image clutter. During tracking of moving objects, the spurious objects could then "trap" parts of the active contour, and thus may result with serious tracking problems [3]. The problem is caused, mainly, by the error introduced within the system measurements [3]. Due to pre-smoothing of the image sequence, which is essential to reduce noise and to obtain reliable measurements of image velocity [] and of image potential [], the image clutter introduces large errors to the systems measurements. In order to reduce the sensitivity of tracking to image clutter, we propose in this work a new active contour model with directional-based measurements of image velocity and of gradient-based image potential. The problem of incorporating the appropriate measurements to obtain optimal tracking in the sense of the PDAF approach, is the subject of this work. Considerable work has been done on the subject of object tracking with visual sensors. In this paper we focus on methods which are based on the Kalman ltering approach, in which models of objects dynamics and of measurement noise are incorporated to obtain the optimal minimum variance estimation of state (e.g. [3]). Examples of applications of this method in visual tracking problems can be found in 3-D model-based tracking [], [8] and in contour-based tracking of ob < vl vr > vl Figure.: Illustration of velocity estimation in the vicinity of the intersection region between two moving objects. (a) two moving objects (b) the resulting estimation of motion. ject's contour [5], [8], [5]. Recently a new stochastic snake model was proposed for visual tracking in spatiovelocity space [3]. This method, which is a generalization of the velocity snake proposed by Peterfrend [], employs measurements of both gradient-based image potential and of optical-ow to obtain optimal estimation of the contour position and velocity. Though the proposed method increase the robustness of tracking to measurements noise [3], it may fail however in the present of high contrast image clutter. Due to pre-smoothing operation (which is essential to reduce image noise), this clutter may introduce bias into velocity and potential eld measurements, and thus may result with target loss. The problem is illustrated in Figure., where velocity estimation in the vicinity of the intersection region between two moving objects, results in some weighted averaging of both velocities. The edge-based potential eld measurements in the vicinity of the intersection point, suer from the same problem and could result with attracting forces towards image clutter. In order to overcome this problem, we propose in this work a new directional based approach for measurements of image motion and potential. This method is based on new directional smoothing and derivative operators. Referring to the problem introduced in Figure.,onecould then obtain two unbiased directional measurements of image velocity and of potential eld. The tracking scheme could then choose the appropriate directional measurements which are most consistent with previous estimation of motion. This tracking approach is the subject of this work. We use the following notions: The vector x (x; y) denote -D spatial coordinates. The spatial gradient of ascalar function I(x; t) is given by ri (I x ;I y ) where and denote the partial derivatives of I vr /99 $. (c) 999 IEEE

2 with respect to x and y, respectively. The matrix O MN denote an M N zero matrix, O M is an M dimensional zero vector, and I M denote an M M identity matrix. The Velocity Snake Model: Kalman Filtering Approach In this section we review the continuous-time stochastic model of the velocity snake, originally proposed in [3]. This model is based on the Extended Kalman ltering method (e.g. [3]) with system measurements of image motion and of gradient based image potential. The discrete-time version of this model is given in [3]. Consider the closed contour v(s; t) (x(s; t); y(s; t)) for some spatial parametric domain s [; ] and time t [; ). Let and v The dynamics of the Real-Time velocity snake model, originally proposed in [], is given by v tt + ri(ri v t + I @ (w v s (w v ss ),rp (v(s; t);t) : () where P is the potential eld energy of the contour, and ; ; ; w and w are some positive scalars. The discretization of the velocity snake model in space is based onequidistant sampling of v(s; ) along s, with u [u ;:::;u M ] and u i v(s i ; ) denoting the snake points, and on nite dierence approximation of partial derivatives in space [7], [5]. The stochastic contour model is based on discretization in space of the velocity snake (). Let [x ;x :::x M ] T and [y ;y :::y M ] T denote the vectors of sampling points corresponding to u, with u i [x i ;y i ], V [ T ; T ] T and _V [ _ T ; _ T ] T, where _V denote the derivative of V with respect to t. Wedenote the Gaussian probability density with mean vector h and covariance matrix Q by N(h; Q). The Markov process model corresponding to () with is given by [3] OMM V_V d dt, where V_ V OMM I M u p + K K I M,, T M q ; q N(;Q) D () D Here the matrix K consists of the deformation constraints imposed byw and w in () [5], [7], and D is the space derivative-approximation matrix [3]. Let [ x ; y ], where x and y are diagonal matrices with the diagonal elements given by I x (u) and I y (u), respectively, [Px T (u); Py T (u)] T. The measurement vector of () is given by [3] I t u p, _ V + M w (3) : Note that the second measurement vector I t results from rst order approximation of the intensity preserving equation I(V;t + dt) I(V, Vdt;t); _ which is a special case of the optical-ow constraint equation []. T The estimation problem of ^V [^V T V ^_ ] T is solved by the Extended Kalman Filter method [3]. The state estimation of () is then given by [3], OMM where, for OMM d ^V I M dt,, T ^V;t)+G( ^V;t) I t ( ^V;t)+ ^_V I M H [ ;,] and L we have the Kalman gain matrix () L L L T ; (5) L G( ^V;t)LH T W,, L L T W, : () The covariance matrix L of ^V is given by the solution of the continuous-time matrix Riccati equation (e.g. [3]). 3 The PDAF Tracking Approach in Spatio-Velocity Space In this section we present the probabilistic data association lter (PDAF) corresponding to the stochastic model in previous section. This method, which is based on the formalism proposed in [5], improves the robustness of tracking as it results with reliable measurements even in the presence of large image clutter. According to the proposed model of system measurements, a number of directional measurements of image potential and motion are sampled at each snake point. The tracking model then evaluates the probability of having each of these measurements, based on previous estimation of motion. Modeling of velocity and potential eld measurements as a weighted average of directional measurements, is the basis for the proposed PDAF snake model approach. Within this model, the weight applied to each measurement is dened as the probability of having it, given previous estimation of motion. Next we dene the directional measurements. The "right hand" and "left hand" directional derivatives of ascalar function I(x; y) along x, are dened as D + x I lim dx! + I(x+dx;y),I(x;y) dx D, x I lim dx!, I(x+dx;y),I(x;y) dx ; Here + denote small positive scalar, and, asmall negative one. The directional derivatives along y are dened in a similar manner. The proposed scheme is the basis for computation of directional measurement w N(;W) of : the gradient of image potential and of optical-ow. (7) In this work we limit the directional measurements to the four types given in Figure /99 $. (c) 999 IEEE

3 D x D y D x D+y Figure 3.: Example of four types of directional measurements, where D x + and D x, denote the "right hand" and the "left hand" directional derivatives along x, respectively, and D y + and D, y are the corresponding measurements along y axis. 3. The PDAF Tracking Model The following is based on the PDAF approach proposed in [5]. Consider the system state at time k, V(k), and let Z k fz(j)g k j denote the vector of state measurements, where z(j) is the measurement at time j. We assume that z(k) fz i (k)g m k for some i m k where z i (k) is the i'th measurement vector. In our model, z i (k) denes the vector of velocity and potential eld measurements along the contour points, where each pair of measurements of velocity and of potential eld at a contour point is chosen from one of the four types of directional measurements as dened in Figure 3.. We assume that the probability of having V(k), based on the set of system measurements Z k,,, satises k, P V(k)jZ N V(k); ^V, (k);l, (k) (8) where N(;a;Q) denote the normal probability distribution with mean a and covariance matrix Q, and ^V, (k) and L, (k) are the estimation of V(k) based onz k,, and the corresponding covariance matrix of estimation, respectively. In the proposed model we assume that only one of the m k measurements was generated by the system [5]. Let i denote the assumption that z i (k) is the true measurement. The probability i (k) that i is the true assumption, satises i (k)p ( i jz k ) D y with D+x i From probability theory, it follows that P (V(k)jZ k ) i D+y D+x i (9) P (VjZ k ; i )P ( i jz k ) () Let ^V(k) E(V(k)jZ k ) denote the optimal estimator of V(k) [5], and ^V i (k) E(V(k)jZ k ; i ) denote the one under the assumption i. It follows from (9) and () that ^V(k) i i (k) ^V i (k) () Let L(k) and L i (k) for i :::m k denote the covariance matrices of ^V(k) and ^V i (k), respectively. Similar to computation of ^V(k) in (), it can be shown that L(k) i i L i (k)+ i i ^V i ^V T i, ^V ^V T () A major problem with the proposed estimation approach is the large number of measurements (and assumptions) m k. In case of M dimensional vector z i (k) with each component chosen from one of L dierent directional measurements (in our case we assume L ), we have on the total m k M L measurements vectors. We shall show that this approach degenerates to only LM types of measurements and assumptions. 3. The PDAF-Based Active Contour Model In this section we apply the proposed PDAF approach to the tracking model (), (3). Measurement Model Consider the set of measurements at the vector of snake points u. Each measurement at a snake point is assumed tohavefourtypes of directional measurements, as proposed in Figure 3.. Let C [c ;:::;c M ] with c i :::, where c i ;:::; denote the type ofdirectional measurement at the i'th point and c i denote a spurious measurement. Using (3), the set of measurements at time k are then given C (u) u p I t (u), C _V + w z C (k) C [c ;:::;c M ] c i ;:::; (3) C and C correspond to the appropriate directional derivatives dened byc. Note that as I t denote derivative along time, it does not contain the superscript C. System Model: The Continuous Time Approach We assume the system model () with a diagonal covariance matrix of measurement noise W I M,for some >. Using ()-() and a rst order approximation of integration along time, we have ^V C ^V +t where A ^V, B@PC + G C (I t + C ^_V ) G C, () L L CT (5) and A and B are the appropriate matrices in (). Substituting ^V C into (), the optimal estimate ^V, based on the measurement set (3), satises d ^V dt f () /99 $. (c) 999 IEEE

4 where L types of directional measurements as dened in Figure 3.. Using (3), we have (, c zi c i V _ + w i ;:::;M (k)i t (u i ) c ;:::; (9) b c f A ^V, BE(@PC ), L E( CT I t )+E( CT C ) ^_ V and E() denote expectation. The result is obtained under lim t!. Analyzing the products CT I t, CT C,and@P C,itfollowsthateach component of these terms depends only on measurements on a single snake point. Hence the expectation of each component is independent of other snake points, and depends only on the four types of directional measurements on that snake point. Let c i(u i ) for c i ;:::; denote the probability of having the type ofdirectional measurement dened by c i, at the snake point u i. The i expectation of the i'th component of CT I t, h CT I t i Ic i x (u i )I t (u i ),is then given by h X E( CT I t ) ii c i c i (u i )I c i x (u i )I t (u i ) (7) The same property holds for all components in CT C C. Next we compute the dynamics of the covariance matrix of ^V. Using the continuous-time matrix Ricatti equation (e.g. [3]) and (), it can be shown that the covariance matrix of ^V satises where dl dt E(F C )L + LE(F C ) T, L E( CT C ) L T L + Q + ^Vf T + f ^V T (8) E(F ) L I M,, +E(@ P ), T From analysis of components P, it follows that the matrix consists of M nonzero components, where each component is only function of a single snake point. Hence, the expectation is done similar to (7). From the above results, it follows that the complexity of computation of each of the components in () and (8) is only in the order of M. This is in contrast to the original result in () and () which have a complexity in the order of M. Probability Evaluation and Validation of Measurements The probability c i(u i ) of having a given directional measurement at the snake point u i is dened as the probability of the corresponding directional velocity measurement. Let zi c (k) denote the directional measurement of velocity at snake point u i,where c denote a spurious measurement, and c ;:::; correspond to the four where c i is the i'th row of C corresponding to the directional measurement c, and w N(; ). The random variable b is uniformly distributed over [,a; a] for some a>. Consider the continuous-time estimation problem. Given the estimation ^V of state at time step k, the corresponding covariance matrix L, and the potential eld C,forsomeC, theprediction of velocity measurement at the snake point u i is zi c, [, c i] ^V C, and Si c, (k) c i LC, c i T () where the prediction of state ^V C, is given by () with G C, and the corresponding covariance matrix L C, is obtained from the Ricatti equation with no system measurements. It can be shown that both ^z i c, and Si c, depend only on current estimation ^V and on the measurement of potential eld at the snake point u i. No other measurements on other snake points aect this estimation. The probability c i i (u i) of having the appropriate velocity measurement at the snake point u i is thus independent of the measurements at the other snake points, and hence, can be evaluated separately. Under the gaussian assumption of measurements noise, the probability of having a directional measurement at a snake point u i is dened as c (u i ) r expf,(z c i, ^zc i, ) T S c i,, (z c i, ^zc i, )g i ;::: r a i () where r denote the normalization scalar. Note that in the case where i we assume uniform probability distribution function over [,a; a]. On Spatial Directional Smoothing and Dierentiation In order to obtain reliable measurements of optical- ow (e.g. []) and to increase the "domain of attraction" of image potential at the object's boundary, the image sequence should be smoothed using a low pass operation. In the presence of clutter boundary, however, this smoothing operation might introduce errors within the system measurements, as illustrated in Figure.. According to this result, most samples of image potential within the region between the two image boundaries (object-left, clutter-right) would point towards the clutter edge, and thus might cause serious tracking problems. The same applies to measurements of optical- ow. In order to overcome this problem, we propose in this work the right-hand and left-hand smoothing operators given in Figure.. It is shown that when applying the left-hand smoothing lter, measurements within the region between the two boundaries would be /99 $. (c) 999 IEEE

5 (a) (b) Figure.: (a) A two boundary one-dimensional image, (b) The image in (a) after low-pass ltering operation. (a) (b) (c) (d) Figure.: (a) Right-hand directional pre-smoothing lter, (b) left-hand directional pre-smoothing lter, (c) the ltering result of the image in.(a) with the righthand smoothing lter, (d) the ltering result of the image in.(a) with the left-hand smoothing lter. only aected by the left boundary, and via-versa when applying the right-hand smoothing lter. According to the directional based measurement approach proposed in this work, prior to calculation of right hand directional derivatives, a right hand directional pre-smoothing lter would be applied. The same holds for left hand measurements. This method can be directly extended to the two dimensional case by applying one-dimensional lter along each axis. 5 Experimental Results We demonstrated the performance of the proposed tracking scheme by applying it to real image sequence with nonrigid objects such as a human leg andawaving hand. We employed the discrete approximation of the model ()-(8). The parameters of the dynamical model were initialized to obtain stable tracking using stability analysis of a linearized model, in the presence of a constant velocity motion of image boundary []. These parameters were then adjusted manually to improve the performance of tracking, resulting with, 3and w w. These parameters could be obtained automatically using learning methods as the one proposed in [8]. The sampling interval was set to t. The contour lines were initialized manually using polygonal approximation of shape. The results of tracking the waving hand are given in Figure 5. (a)-(f). The positions of snake points in previous frame are marked withred dots, the current estimation and the corresponding velocities by the blue lines and arrows. Points which were detected tohave only spurious measurements are markedwithagreen star. In 5. (a), (d) we present the results at the 'th frame, in (b), (e) the ones at the 'th, and in (c), (f) the ones at the 'th frame. In 5.(d) we also show the probabilities of the directional measurements. It is shown that the directional probabilities tends to zero in the vicinity of the shelf at the top right hand side of the hand. The results of tracking the walking leg are given in Figure 5. (g)-(l), with (g), (j) denoting the ones at the 'th frame, (h), (k) at the 8'th, and (i), (l) the ones at the 55'th frame. It can be shown that measurements at the moving hand were detected to be spurious. The kalman snake model of Szeliski and Terzopoulos [5] failed totrack these objects (see [3] for these results). Conclusions We proposed in this work a new active contour model which is based on directional measurements of image velocity and of edge-based potential eld, and on the Probability Data Association lter method. The directional-based approach, separates measurements of image motion from that of image clutter at each snake point. The system then incorporates these measurements into a single measurement vector using weighted averaging, where a weight is dened tobetheproba- bility of having the measurement, given previous estimation of motion. According to the proposed approach, for M contour points with four types of direction measurements at each point, we have on the total M types of measurement vectors. We show that these measurements degenerate into a single M dimensional measurement vector, wherein each scalar is given by a weighted averaging of the four types of measurements at this point. Tracking a moving object with a dynamical tracking system, can be viewed asacontrol problem wherein the image potential and motion dene the system inputs, the smoothing constraints on shape and velocity are used to increase the stability of the system, and the motion control is used as velocity feedback to reduce tracking errors in the presence ofconstant motion of image boundary []. The parameters of the system are chosen to obtain stable tracking in the presence ofknown limits on image motion and on potential-eld characteristics. Within the proposed control frame-work, the the system parameters are chosen in a way that, during tracking, the trajectory will remain within the "domain of attraction" of the objects contour. This domain of attraction is function of the known limits on image motion, and the properties of image potential. In the case of image motion which is higher then the dened limits, the system contour is no longer guaranteed to stay within the "domain of attraction" of the object's contour and might converge to other objects which belong to image clutter. This problem of change in the limits on image motion is the subject of current research work /99 $. (c) 999 IEEE

6 References [] G. Adiv, \Determining Three-Dimensional Motion and Structure from Optical Flow Generated by Several Moving Objects, IEEE Trans. Pattern Anal. and Machine Intell., Vol PAMI-7, No., pp. 38-, 985. [] A.A. Amini, R.W. Curwen and J.C. Gore, \Snakes and Splines for Tracking Non-Rigid Heart Motion", in B. Buxton and R. Cipolla (Eds.), Computer- Vision: ECCV'9, Springer-Verlag, pp. 5-, 99. [3] K.J. Astrom and B. Wittenmark, Adaptive Control, Addison-Wesley, 995. [] J.L. Barron, D.J. Fleet and S.S. Beauchemin, \Performance of Optical Flow Techniques", Int. J. of Computer Vision, Vol., No., pp.3-77, 99. [5] Y. Bar-Shalom and T.E. Fortmann, Tracking and Data Association, Academic Press Inc., Orlando, Florida, 988. [] B. Bascle and R. Deriche, \Stereo Matching, Reconstruction and Renement of 3D Curves using Deformable Contours', in Proc. 'th Int. Conf. on Computer Vision, Berlin, Germany, pp. -3, 993. [7] B. Bascle and R. Deriche, \Energy-Based Methods for D Curve Tracking, Reconstruction and Renement of 3D Curves and Applications", in Proc. of SPIE: Geometric Methods in Computer Vision II, San Diego, CA., pp. 8-93, 993. [8] A. Blake and M. Isard, \3D Position, Altitude and Shape Input using Video Tracking of Hands and Lips", in Proc. Computer Graphics, SIGGRAPH, pp. 7-78, 99. [9] I.J. Cox and S.L. Hingorani, \An Ecient Implementation of Reid's Multiple Hypothesis Tracking Algorithm and Its Evaluation for the Purpose of Visual Tracking, IEEE Trans. Pattern Anal. and Machine Intell., Vol. 8, No., pp. 38-, 99. [] R. Curwen and A. Blake, \Dynamic Contours: Real-Time Active Splines", in A. Blake and A. Yuille (Eds), Active Vision, MIT Press, pp , 99. [] M.P. Dubuisson, S. Lakshmanan and A.K. Jain, \Vehicle Segmentation and Classication using Deformable Templates", IEEE Trans. Pattern Anal. and Machine Intel., Vol. 8, No. 3, pp , 99. [] D. B. Gennery, \Visual Tracking of Known Three-Dimensional Objects", Int. J. of Computer Vision, Vol. 7, No. 3, pp. 3-7, 99. [3] M.S. Grewal and A.P. Andrew, Kalman Filtering: Theory and Practice, Prentice Hall Inc., Upper Saddle River, New Jersey, 993. [] B.K.P. Horn and B.G. Schunk, \Determining Optical Flow", Articial Intelligence, Vol. 7, pp. 85-, 98. [5] M. Isard and A. Blake, \Visual Tracking by Stochastic Propagation of Conditional Density", in Proc. 'th European Conf. on Computer Vision, Cambridge, England, pp , 99. [] M. Kass, A. Witkin and D. Terzopoulos, \Snakes: Active Contour Models", Int. J. Computer Vision, Vol., No., pp. 3-33, 987. [7] F. Leymarie and M.D. Levine, \Tracking Deformable Objects in the Plane Using an Active Contour Models", IEEE Trans. Pattern Anal. and Machine Intell., Vol. 5, No., pp. 7-3, 993. [8] D. G. Lowe, \Robust Model-Based Motion- Tracking Through the Integration of Search and Estimation", Int. J. of Computer Vision, Vol. 8, No., pp. 3-, 99. [9] M.R. Luettgen, W.C. Karl and A.S. Willsky, \Ef- cient Multiscale Regularization with Applications to the Computation of Optical Flow", IEEE Trans. Image Processing, Vol. 3, No., pp. -, 99. [] R. Malladi, J.A. Sethian and B.C. Vemuri, \Shape Modeling with Front Propagation: A Level Set Approach", IEEE Trans. Pattern Anal. and Machine Intell., Vol. 7, No., pp , 995. [] D. Metaxas and D. Terzopoulos, \Shape and Nonrigid Motion Estimation Through Physics-Based Sythesis", IEEE Trans. Pattern Anal. and Machine Intell., Vol. 5, No., pp , 993. [] N. Peterfreund, \The Velocity Snake", in Proc. IEEE Nonrigid and Articulated Motion Workshop, Virgin Islands, 997. [3] N. Peterfreund, \Robust Tracking of Position and Velocity with Kalman Snakes, in Proc. ICCV, Jan [] S. Rowe and A. Blake, \Statistical Mosaics for Tracking", Image and Vision Computing, Vol., No. 8, pp. 59-5, 99. [5] D. Terzopoulos and R. Szeliski, \Tracking with Kalman Snakes", in A. Blake and A. Yuille (Eds.), Active Vision, MITPress, pp. 3-, /99 $. (c) 999 IEEE

7 (a) (b) (c) (d) (e) (f) (g) (h) (i) (j) (k) (l) Figure 5.: Tracking results of the waving hand (a)-(f) and the walking leg (g)-(l) /99 $. (c) 999 IEEE