A Unied Factorization Algorithm for Points, Line Segments and. Planes with Uncertainty Models. Daniel D. Morris and Takeo Kanade

Size: px

Start display at page:

Download "A Unied Factorization Algorithm for Points, Line Segments and. Planes with Uncertainty Models. Daniel D. Morris and Takeo Kanade"

Karin Underwood
5 years ago
Views:

1 Appeared in: ICCV '98, pp , Bombay, India, Jan. 998 A Unied Factorization Algorithm or oints, Line Segments and lanes with Uncertainty Models Daniel D. Morris and Takeo Kanade Robotics Institute, Carnegie Mellon University, ittsburgh, A -89 Abstract In this paper we present a unied actorization algorithm or recovering structure and motion rom image sequences by using point eatures, line segments and planes. This new ormulation is based ondirec- tional uncertainty model or eatures. oints and line segments are both described by the same probabilistic models and so can be recovered in the same way. rior inormation on the coplanarity o eatures is shown to t naturally into the new actorization ormulation and provides additional constraints or the shape recovery. This ormulation leads to a weighted least squares motion and shape recovery problem which is solved by an ecient quasi-linear algorithm. The statistical uncertainty model also enables us to recover uncertainty estimates or the reconstructed three dimensional eature locations. Introduction We address the structure rom motion problem: given an image sequence o a scene or rigid object taken by a camera undergoing unknown motion, reconstruct the three dimensional (D) geometry o the scene. While we also recover camera motion, in this report we ocus on D structure. Further, we require that the solution be robust to the inevitable imprecision and small errors involved in imaging and registration. Early structure rom motion work ocused on the two image problem [, 7]. Starting rom a set o corresponding eatures in the images, the camera motion and point locations could be recovered. Since then research has shited to the more dicult multiple-image problem where more data samples can lead to greater accuracy which is our motivation. Much current work ocusses on recovering shape using a ull perspective camera model [,,,, ]. Solving the perspective equations typically entails non-linear minimization and the usual problems o many local minima, and high computational expense, with convergence and accuracy being sensitive to special parameters and initial guesses. Noting this and that even the perspective model is only a rough approximation when eects like radial distortion are large, we ollow [,, 6] and side-step many o these problems by working with an ane camera model. Very accurate results can still be obtained provided the perspective distortion eects are small. Batch actorization methods have typically weighted eatures uniormly, or as in [9], allowed condence weightings. Registration algorithms can oten provide directional uncertainty measures or eature location and here we ormulate an algorithm to incorporate these directional uncertainties. We note that a signicant missing aspect o current methods is their lack o condence measure in their data and results. Our algorithm provides an uncertainty measure or the resulting shape eatures. Recent work with line eatures [,, 8] illustrates that, while lines provide valuable constraints or shape recovery, the reconstruction process using them is more sensitive to noise and being trapped by local minima than or point eatures. Further the line actorization algorithm [] requires at least seven directions or lines in the scene, a signicant practical problem. Our uncertainty ormulation provides a natural method or modeling line segments, thus avoiding the above diculties and, in particular, allowing us to jointly recover shape rom point and line-segment eatures. We also extend the algorithm to incorporate multiple planar constraints. We begin by summarizing the actorization algorithm in Section. Then, in Section, we describe our uncertainty model and the Bilinear algorithm or points, lines and planes. Next we present results rom synthetic and real image sequences in Section and we end with the conclusions. Factorization Algorithm Overview We provide a brie overview o the Factorization algorithm ound in more detail elsewhere [,, 6]. The algorithm is eature-based, where a eature is a distinctive part o an image, such as a corner, edge or mark on a rigid object. A set o eatures in F images

2 Figure : Feature uncertainty indicated by ellipses Figure : Feature tracked in an image sequence with coordinates w p (u p ;v p )j ;:::;F;p ;:::;g are tracked in the image sequence as illustrated in Figure. Feature coordinates are transormed to object coordinates by subtracting their center o mass: wp is replaced by w p wp, w c or all and p, where w c p w p. A measurement matrix, W, is created containing the relative eature coordinates: W 6 u u u F u F v v v F v F Assuming orthography, Tomasi and Kanade [6] showed that W is rank or less, and can be actored into the product o a motion matrix, M, and shape matrix, S, i.e., W MS () where M 6 m T m T F n T n T F 7 F 7 and S s s The rows o M encode the rotation or each rame, and the columns o S contain the D position o each eature in the reconstructed object. When there are errors or noise in the measurement matrix, we seek a least squares approximation to it. We dene an error unction: E SV D (M;S) kw, MSk T p p ; () ;p up, m where T s p p, which we seek to minimize. This can be achieved by perorming Singular vp, n T s p Value Decomposition (SVD) on the measurement matrix, W UV, and zeroing all but the three largest singular values in to get ^W U ~V.Factoring this provides least squares estimates or motion and shape: ^M U ~, and ^S ~ V T.We note that ^M and ^S are dened only up to an ane transormation, but we can obtain Euclidean coordinates by applying appropriate constraints, depending on the projection model, to the rows o ^M as in [6]. From here on we drop the \^" symbol and assume this step has been perormed. Bilinear Factorization Algorithm To improve the standard actorization algorithm, we propose using a more general Gaussian eature uncertainty model and derive our Bilinear algorithm or this model extending a similar technique in [9].. Directional Uncertainty Formulation Since eature tracking and registration algorithms do not provide exact values or eature positions, we model the actual eature position, x p, with a D Gaussian probability density unction: p (x p )k p exp(, (w p, x p ) T G p (w p, x p )) () where G p is the inverse covariance and w p the mean. The covariance determines an ellipse o equal probability density, and the major and minor axes give the directional uncertainty in eature location as indicated in Figure. Assuming independence o eature densities, the total probability density unction or eatures in F images will be the product o all the individual density unctions: T Y p p (x p ): () The maximum likelihood solution to this is obtained by minimizing the cost unction, E B, given by: E B p (w p, x p ) T G p (w p, x p ) () To recover shape and motion, we assume a ane projection model, M, or each image acting on the D object centered point coordinates s p spx s px s pz. Thus, eature locations are constrained by the ollowing equation: m T x p M s p s p n T s p (6)

3 We note that, when this constraint is substituted into () and the covariance matrix is identity, the cost E B reduces to the standard actorization cost, E SV D, showing that SVD actorization is a special case o our algorithm.. Bilinear Minimization The maximum likelihood solution or shape and motion is obtained by minimizing E B with respect to the shape and motion parameters. We note that the partial derivatives o E B are bilinear in these parameters: m, n and s p B B p p p Here G p h,c p (u p, m T sp)sp, d p(v p, n T sp)sp,d p (u p, m T sp)sp, e p(v p, n T sp)sp (7),c p (u p, m T sp)m, d p (u p, m T sp)n, d p (v p, n T sp)m, e p (v p, n T sp)n i c p d p d p e p is positive denite and is split into its components. By setting the partials to zero and rearranging the resulting equations we obtain a set o bilinear equations or motion and shape: p c ps ps T p p d ps ps T p p d ps ps T p p e ps ps T p h m n i p (c p u p + d p v p )s p p (d p u p + e p v p )s p " # c p m m T + d p(m n T + n m T )+e pn n T [s p] i h cp upm + dp (up n + vp m )+ep vp n (9) or p ;:::; and ;:::;F.We start with an initial guess or motion M g, and use equation (9) to obtain the least squares solution or S, and then this new S and equation (8) to solve or the least squares motion M. This step is repeated iteratively until convergence is achieved. We call this our Bilinear Factorization Algorithm. The complexity o each iteration is O(F) operations, compared to SVD which, rom [], has complexity O(F ) or F > or O(F ) otherwise. The Bilinear algorithm monotonically reduces the cost unction but it is not guaranteed to converge to the global minimum, hence we usually start with reasonable estimates or motion or shape such as obtained rom the SVD solution which is a global minimum o E SV D. With this estimate our experiments typically converged within a ew iterations. (8) L Figure : Line segment eature uncertainty. Line Segments Recent work has extended the actorization algorithm to innite lines [], but this involves more stages and is more sensitive to noise than the point eature algorithm. The primary reason or this sensitivity is that each line provides a single constraint, whereas point eatures provide two constraints. ractical line registration algorithms actually work with line segments which are t to edges the images. The positions o the recovered line endpoints may correspond to dierent parts o the edge in any given image, but they are restricted to all within the physical limits o the object. Thus, by using line segments which we know correspond to a particular region o the innite line, we can extract more constraint inormation than is possible or innite lines. We dene line segments by their end-points and model each end-point as a Gaussian density unction with large uncertainty along the length o the line and small uncertainty perpendicular to the line, as illustrated in Figure. Line segments dened by their end-point density unctions have the same orm as point eatures and so can be used directly by the Bilinear algorithm on their own or in conjunction with point eatures.. Shape Uncertainty We seek an estimate or the uncertainty in the recovered D positions o the eatures by the actorization algorithm. We model second order error terms with D Gaussian distributions around the maximum likelihood solution to T rom (). Our inverse covariance or each eature will be the Hessian o E B in the shape parameters which we obtain as: Hp p σ 7 σ M T G p M () For this to be an exact and ull description o the error probability density, our equations or shape would need to be linear in measurement values, and individual eatures must be independent o each other, i.e. the equations could be expressed in the orm: s p A w p + b p 8p with some constants A and b p. However, our equations, (8) and (9), are bilinear

4 in shape and motion, resulting in coupling between motion and shape, and hence coupling also between eature positions. To second order this coupling is described by the cross terms o the large covariance matrix o the shape vector, s v, ormed by stacking the columns o S vertically. With no inter-eature correlation this would be a block diagonal matrix, but with coupling there will be non-zero cross terms. By perorming multiple experiments we can estimate the value o these cross-terms as well as the accuracy o our eature covariance estimate H, p. Our experiments indicated that in some instances, depending on the motion and shape o the object, the block diagonal dominate the cross-terms, but in others there were signicant cross-correlation eects, see Figure (a) and (b). For the purpose o this report we ignore these cross-terms but note that, depending on the application o our uncertainty models, these terms may or may not be o signicance. Independent o the cross-terms, however, we ound that equation () accurately predicted the individual eature covariance terms as shown in Figure (c).. lanes A plane can be determined using three noncollinear points s; s; sg. oints on the plane can all be written as a linear combination o the dierence vectors: s p s + a p (s, s)+b p (s, s) (s, s) (s, s) s a p b p () σ x σ y σ z r xy x (b) 6 8 with some constants a p and b p dening the position on the plane. Multiple points in a plane can thus be actored into two matrices: S R r r t a a a p b b b p () The vectors r and r are parallel to the plane and t is a point on the plane. The equation or a point, p, on the plane is thus: (p, t) (r r) : () When the shape matrix contains points rom more than one plane, points rom each plane can be gathered into blocks and actored separately. For Q planes the global actorization equation becomes: W MS M R R R Q Q () r xz (a) r yz.... Actual and redicted Values (c) Figure : Error distribution o reconstructed shape data o the synthetic cube rom runs with equal, uncorrelated Gaussian noise added to all image eatures. (a) art o the covariance matrix or s v illustrating the block diagonal terms dominating cross-correlation elements. Here there is 6 degrees pitch and yaw rotation. (b) art o the covariance matrix or the same object undergoing dierent motion ( degree rotation around each axis) illustrating large cross terms. (c) A histogram o the actual values o the elements in the covariance matrices or each eature, with the values predicted by equation () marked with a \*" symbol, showing close agreement. (The variances,, are scaled by ij, and the r ij terms are the standard correlation coecients)

5 Equation () is trilinear in motion, plane and position parameters. While it could be solved with a trilinear set o equations analogous to equations (8) and (9), we choose to break it into two bilinear steps. In the rst step we directly apply the Bilinear algorithm to solve or motion and shape. In the next step we dene an error unction, E i, or each plane, i : E i p (s p, R i q p ) T H p (s p, R i q p ) () where q p are the columns o i, and H p is the inverse covariance rom (). This error is minimized with a similar set o bilinear equations to (8) and (9). The parameter t i can be either solved or in these equations, or calculated directly as a weighted center o mass o the points in a plane. This algorithm diers rom simply tting planes to the resulting shape in that the camera motion as well as uncertainty estimates are used to guide the tting using the inverse covariance matrices H p. The two bilinear steps can be iterated until convergence is achieved..6 Object Centered Coordinates ast work generally used the center o mass as the origin or the object coordinate rame which, or equal density eatures, is the minimum variance point. Here we have covariance estimates or each eature in each rame so we seek an origin, w c p a pw p, which isaweighted sum o individual eatures in an image, that has minimum total variance. Since the relative orientation o eatures between images is unknown, we use the non-directional radial variance,, or pr each eature which is equal to the sum o the principal components o its covariance matrix, G,, i.e.: p pr p + p. Solving or the minimum variance point we obtain the ollowing coecient values (see [8]): a F i ( ip + ip), F A j i ( ij + ij ) : (6) Essentially this amounts to weighting each eature by the inverse o its total uncertainty in all rames. We note that this uncertainty involves the sum o the directional uncertainties: p and p, and hence eatures with a large dominant uncertainty, such asa line segment, will contribute ar less than point eatures having small variances in both directions. This indicates that line segments are generally inerior to point eatures or determining object centered coordinates and so provides added motivation or combining line-segment and point-eature shape recovery. Experiments We report results or a synthetic shape, and then or a real image sequence o a cube. A synthetic cube was used to create image sequences to which Gaussian noise was added with standard deviation chosen randomly rom the range: ; : which isup to percent o the size o the object. The sequence consisted o rames and rotations o about degrees around each axis. The SVD and the Bilinear algorithms were compared based on their respective reconstructions rom this data. Figure shows sample reconstructions by both algorithms and compares planes t to the SVD reconstruction, with planar actorization. arts (e) and () plot the total error between the recovered and true shape or point and planar reconstruction over a sequence o runs. The Bilinear algorithm, which uses knowledge o the covariance in its reconstruction, gives better results or both o these cases than the SVD algorithm. Next, we perormed our shape recovery algorithm on a sequence o images o a cube, the rst and last being shown in Figure 6. Features are represented as square templates that are tracked in the image sequence with a warpable Sum o Squared Dierence technique [6, 8, ]. Here we permitted skewing o the templates in the minimization rather than ull ane warping. We took the positional Hessian o each template error unction to be our estimate or the inverse covariance o each eature obtaining the ellipses shown in the gure. The large ellipses indicate poorly located eatures. Figure 7 shows the Bilinear shape reconstruction with ellipsoids representing the relative uncertainty o eatures. By tting planes to the SVD and Bilinear cube sides, we estimate that the SVD results have percent larger error than the Bilinear. Figure 8 shows the end-points o the automatically tracked line segments with ellipses representing positional uncertainty or each end-point. Here our uncertainty estimates were crude; we simply choose the variance along the line to be proportional to the line length and the variance perpendicular to the line to be inversely proportional to the length, since we expect to localize longer lines more accurately. We plan to do better in the uture by characterizing the line tracking errors, but, despite this rough estimate, the results were good. The recovered shape is shown in Figure 9 with each end-point being represented by an ellipsoid corresponding to its D covariance, and corresponding eatures are joined to orm the recovered line segments. To test the robustness o the algorithm to changes in line segment lengths, we added noise along the line direction to the end-points and compared the

..... (a).... (b)....... Figure 7: Reconstructed point eatures with exaggerated uncertainty ellipsoids. (a) (let) is a view o the let ace, (b) (right) is a top view.. (c).... (d).

(e) Shows Factorization, and () lanar actorization D errors or a sequence o runs comparing SVD (dashed) and Bilinear (solid) reconstruction.

art (c) shows the improvement gained in this case when point eatures are used to help determine the object centered coordinates.

() Figure 8: Ellipses modeling line segments Conclusion Our new Bilinear actorization algorithm uses a D Gaussian model or points and line segments, and, along with planar constraints, unies

6 ..... (a).... (b) Figure 7: Reconstructed point eatures with exaggerated uncertainty ellipsoids. (a) (let) is a view o the let ace, (b) (right) is a top view.. (c).... (d)... Error Error 6 8 Runs (e) 6 8 Runs Figure : Recovered shapes with (a) SVD, (b) Bilinear, (c) tting planes to SVD, and (d) planar actorization. (e) Shows Factorization, and () lanar actorization D errors or a sequence o runs comparing SVD (dashed) and Bilinear (solid) reconstruction. Figure 6: First and last images in sequence with eature uncertainty estimates reconstruction results using SVD and the Bilinear algorithm as shown in Figure. art (c) shows the improvement gained in this case when point eatures are used to help determine the object centered coordinates. oint and line segment reconstruction are easily combined and the result is illustrated in Figure. () Figure 8: Ellipses modeling line segments Conclusion Our new Bilinear actorization algorithm uses a D Gaussian model or points and line segments, and, along with planar constraints, unies reconstruction rom points, line segments and planes. The computation or each iteration is very ast, O(F), and convergence rapid. Shape recovery accuracy is improved by incorporating directional uncertainty measures that weight each eature's constraints by our condence in them. Thus the algorithm can utilize a wider spectrum o eatures than previous algorithms to raise accuracy. The D uncertainty estimates that are generated can be used in subsequent surace-modeling steps or in active vision applications. Further work we would like to do includes investigating the determining actors or correlation between eatures and the extension o uncertainty modeling rom the ane to perspective case. We also hope to investigate better methods to estimate eature uncertainty in the tracking process. Acknowledgements The automatic line tracker provided by Naoki Chiba is greatly appreciated. Reerences [] A. Azarbayejani B. Horowitz, and A. entland, \Recursive estimation o structure rom motion using relative orientation constraints," roc. IEEE Con. Computer Vision attern Recognition (CVR'9), pp. 9-99, June 99.

7 Figure 9: Reconstructed line segments with ellipsoids giving uncertainty o endpoints (b) (a) Figure : Cube reconstruction using line segments with percent line length variation at each endpoint. (a) SVD result, (b) Bilinear line segment result, and (c) Bilinear line segment result but with point eatures aiding in determining object coordinates (c) []. Comon and G. H. Golub, \Tracking a Few Extreme Singular Values and Vectors in Signal rocessing," roc. o IEEE, vol. 78, No. 8, pp. 7- Aug. 99. [] N. Cui, J. Weng and. Cohen, \Extended Structure and Motion Analysis rom Monocular Image Sequences," IEEE roc. Third Int. Con. Computer Vision (ICCV'9), pp. -9, Osaka, Japan, Dec. 99. [] O. Faugeras, F. Lustman and G. Toscani, \Motion and Structure rom Motion rom oint and Line Matches," ICCV '87, pp. -, London, UK, June 987. [] H. Longuet-Higgins, \A computer algorithm or reconstructing a scene rom two projections," Nature, 9:-, 98 [6] B. Lucas and T. Kanade, \An Iterative Image Registration Technique with an Application to Stereo Vision," roc. DARA Image Understanding Workshop, pp. -, April 98. [7] L. Matthies and S. Shaer, \Error Modelling in Stereo Navigation," CMU-CS-86-, 986. [8] D. D. Morris and T. Kanade, \A Unied Factorization Algorithm or oints, Line Segments and lanes with Uncertainty Models," CMU-RI-97-, 997. [9] C. oelman, \The araperspective and rojective Factorization Methods or Recovering Shape and Motion," CMU- CS-9-7, July 99. [] C. oelman and T. Kanade, \A araperspective Factorization Method or Shape and Motion Recovery," IEEE Trans. attern Analysis Machine Intelligence (AMI-9), vol. 9, no., pp. 6-8, March 997 [] W. ress, S. Teukolsky,W. Vettering and B. Flannery, Numerical Recipes in C, Cambridge University ress, 99. [] L. Quan and T. Kanade, \A Factorization Method or Ane Structure rom Line Correspondences," CVR'96, pp. 8-8, San Fran., CA, USA, June 996. [] J. Shi and C. Tomasi, \Good Features to Track," CVR'9, pp. 9-6, Seattle, June 99. [] M. Spetsakis and J. Aloimonos, \A Multi-rame Approach to Visual Motion erception," Int. J. Computer Vision, Vol. 6, No., Aug. 99, pp. -. [] R. Szeliski and S. B. Kang, \Recovering D shape and motion rom image streams using non-linear least squares." TR CRL 9/, DEC, Cambridge Research Lab, Mar. 99. [6] C. Tomasi and T. Kanade, \Shape and motion rom image streams under orthography: a actorization method," Int. J. Computer Vision, vol. 9, no., pp. 7-, Nov. 99 [7] R. Y. Tsai and T. S. Huang, \Uniqueness and Estimation o Three-Dimensional Motion arameters o Rigid Objects with Curved Suraces," AMI-6, pp. -7, Jan. 98. [8] C. J. Taylor, \Structure and Motion rom line segments in Multiple Images," AMI-7, no., pp. -, Nov. 99. [9] C. Therrien, Discrete Random Signals and Statistical Signal rocessing, rentice-hall Inc., 99 [] B. Triggs, \Factorization methods or projective structure and motion," CVR'9, pp. 8-8, 99. Figure : Joint point and line segment reconstruction

A Factorization Method for 3D Multi-body Motion Estimation and Segmentation

1 A Factorization Method for 3D Multi-body Motion Estimation and Segmentation René Vidal Department of EECS University of California Berkeley CA 94710 rvidal@eecs.berkeley.edu Stefano Soatto Dept. of Computer