Mirrors in motion: Epipolar geometry and motion estimation

Transcription

1 Mirrors in motion: Epipolar geometry and motion estimation Christopher Geyer Kostas Daniilidis University of California, Berkeley University of Pennsylvania Berkeley, CA 970 Philadelphia, PA Abstract In this paper we consider the images taken from pairs of parabolic catadioptric cameras separated by discrete motions. Despite the nonlinearity of the projection model, the epipolar geometry arising from such a system, like the perspective case, can be encoded in a bilinear form, the catadioptric fundamental matrix. We show that all such matrices have equal Lorentzian singular values, and they define a nine-dimensional manifold in the space of matrices. Furthermore, this manifold can be identified with a quotient of two Lie groups. We present a method to estimate a matrix in this space, so as to obtain an estimate of the motion. We show that the estimation procedures are robust to modest deviations from the ideal assumptions. 1. Introduction The widespread use of omnidirectional cameras in panoramic visualization and robot navigation motivated many of us to explore how such systems could be used beyond simple display or localization from vertical edges. For example it has been shown that the class of catadioptric sensors with a single viewpoint [3], i.e., equivalent to a perspective camera up to an unknown distortion, obey a simple projection model [8]. In this model we can describe the projections induced by these cameras as a composition of two central projections. The first is a central projection of the point in space to the sphere; the second is a central projection from a point lying between the sphere center and its north pole to a plane. This model puts traditional off-the-shelf and catadioptric cameras in the same context: the former is just a special case in which the two projection centers coincide. It has long been suspected, and in places proven, that increasing the field of view improves the conditioning of motion estimation. What else can we expect from mirrors in motion regarding motion estimation? This is the question we study in this paper. A resulting algorithm would have significant impact, both on visualization and navigation: By taking few panoramic pictures with a high resolution catadioptric device we would be able not only to unwarp to any direction from the recording viewpoints but also from any other viewpoint up to occlusion. Regarding navigation, such an algorithm would be of great help in map building: several scene scans, even obtained by other sensors, could be registered correctly in the same coordinate system if we had accurate motion estimates. One of the first issues we address is the appropriate representation of image features. Models of catadioptric projection are still non-linear, so what are efficient ways of decoupling motion and structure estimation? This has been achieved for perspective cameras using homogeneous coordinates, but what can we say in other cases? For parabolic cameras, does a space exist where the epipolar constraint can be written bilinearly? If so, what are the properties of the bilinear forms and what kind of space do all such matrices form? Then, how do we estimate optimally and robustly? Recently we have been able to give answers which lead to simple and effective algorithms in all these questions. In [9, 10] we gave a representation of image features with which one can express the epipolar constraint and general multiview relations for parabolic systems as linear constraints. We showed that the set of transformations of this feature space is the group O(3, 1) of Lorentz transformations. Furthermore, the left and right nullspaces of the new parabolic catadioptric fundamental matrix encode the intrinsic parameters of the two cameras so that self-calibration reduces to kernel estimation. In this paper our contribution is in the following points essential to estimation in the spaces of bilinear constraints: A complete characterization of the parabolic catadioptric fundamental matrices is given. It is shown that, similar to essential matrices for perspective cameras, all such bilinear forms have two equal non-zero Lorentzian singular values, even though they are constraints for uncalibrated cameras. The set of all of these fundamental matrices, and in fact the bilinear constraints for perspective cameras as well, can be identified with a quotient of Lie groups, which is known as a homogeneous space. In particular, the catadioptric fundamental matrices form a nine-dimensional manifold in the space of matrices. The exponential map yields a nowhere-singular and surjective parameterization of these matrices. We present a correct estimation process of the epipolar geometry which at each step is restricted to the manifold. Demonstration with real and simulated experiments, the latter of which demonstrates that the proposed estimation algorithm is robust to modest deviations of the assumptions, such as misalignment of the camera and mirror.

2 Some of these results necessitate elementary concepts from differential geometry, algebra, and their intersection, namely the theory of Lie groups. We attempt to explain all of the concepts, albeit very briefly, and the reader is referred to two highly readable sources [, 5]. With regards to related work, Baker and Nayar [13] described all possible configurations of central catadioptric cameras. Svoboda and Pajdla were the first to study the epipolar geometry of central catadioptric devices []. Kang devised a non-linear self-calibration algorithm for two views [15]. We introduced in [9] a bilinear epipolar constraint for parabolic cameras. Based on this work Sturm [1] derived linear multiview relationships mixing perspective and parabolic cameras. Simultaneously in [10] we introduced multiview linear constraints and derived conditions that an arbitrary matrix be a catadioptric fundamental matrix. A non-linear motion estimation algorithm has been proposed for both hyperbolic and parabolic systems in [7]. For the calibrated case, Gluckman and Nayar [1] estimate motion from the optical flow estimated in catadioptric images. Other calibrated algorithms making assumptions about the environment and produced for visualization can be found in [3, 1]. Calibrated motion estimation algorithms have been also introduced for non-central cameras [17, 18]. A valuable resource for omnidirectional vision literature is the book by Benosman and Kang [] and the proceedings of the the four Workshops on Omnidirectional Vision. A useful reference on structure-from-motion can be found in [1] and [1].. Parabolic catadioptric projections Suppose that a parabolic mirror is placed above an orthographically projecting camera so that the optical axes of the two are parallel. Such an optical system has a single effective viewpoint at the focus of the paraboloid. If the calibration parameters are known and previously taken into account then the projection induced by the combination of the mirror and lens can be expressed by the following function which we call the canonical parabolic catadioptric projection of the point X = (x, y, z, w): q(x) = (x,y) z+sign w x +y +z p if x = y = 0 and z/w > 0 undefined if w = 0 or X = (0, 0, 0, 1) In contrast, the canonical perspective projection is the map p(x) = (x, y, z) undefined for X = (0, 0, 0, 1), where the range is P, the projective plane. This can be expressed as a linear map from P 3 to P using the 3 canonical perspective camera matrix P = (I, 0); for then p(x) = P X. Both canonical projections p and q have the same viewpoint, namely the origin, expressed as O = (0, 0, 0, 1). (1) Two exceptions are present in the definition of q but not that of p, let us try to justify them. First, notice that the denominator of the first case in (1) is zero only when the numerator is also zero. In particular, for all points X = (0, 0, z, 1) with z > 0. Hence, the projective plane is not a natural representation for parabolic catadioptric images. Instead we add a single point, p, to R, and call this R or the extended plane. Second, observe that points on opposite sides of the viewpoint have different images, in particular q(x, y, z, 1) q( x, y, z, 1). As a consequence, projections of points on the plane at infinity in P 3 are ambiguous since their sided-ness with respect to the viewpoint is undefined, i.e., in P 3 is defined such that (x, y, z, 0) = ( x, y, z, 0). In general, the parabolic camera will be uncalibrated and the actual induced projection may differ from the canonical projection by some transformation k which we assume consists of a scale and translation. We will assume throughout this paper that the skew is zero and the aspect ratio is one. If (c x, c y ) is the image center and f is the focal length, then k is of the form: k(u, v) = (c x + fu, c y + fv) and k(p ) = p. () The latter holds because the point at infinity is unaffected by either scale or translation. A general parabolic catadioptric projection can be modeled by a function, g = k q e, where e SE(3) accounts for the world coordinate system. 3. Linear projection model Recalling that p is the canonical perspective projection, consider the following question: If g = k q, does there exist a function h, independent of k, and a matrix A k such that for all X, p(x) = A k h( g(x) )? (3) In other words, can we embed the uncalibrated, twodimensional image plane in a higher dimensional space via h, such that the resulting image points are linearly related to the equivalent perspective projection? Though A k may be unknown and dependent on k, the function h ought to be independent of k and X. If this were true then in any expression where we would use p(x), e.g., the epipolar constraint, we could substitute the right hand side of (3) and A k would be absorbed into any linear constraint. We will demonstrate that for parabolic cameras there does exist a family of A k s and an h. First, let h be defined by: h(u, v) = (u, v, u + v 1, u + v + 1) T, () with the exception that h(p ) = (0, 0, 1, 1) = N. The range of h is the unit sphere S in P 3, equal to the set of points p P 3 satisfying the quadratic form: p T Qp = 0 where

3 Q = diag(1, 1, 1, 1). In addition, h 1 S, the inverse restricted to h s range, is stereographic projection, i.e., central projection from N, the north pole. It is not hard to show that (h k)(u, v) = ( (c x + f u), (c y + f v),... ) T, is linearly dependent on h(u, v). Thus, there is a matrix K, defined in equation (1) of the appendix, such that (h k)(u, v) = Kh(u, v). In other words, the similarity transformation k in R induces a projective transformation K which preserves S. This commutative relationship is shown in Figure 1.. The epipolar constraint It is well known that two perspective projections f 1 = p e 1 and f = p e satisfy the following equation known as the epipolar constraint: f (X) T E f 1 (X) = 0, (7) for some essential matrix E depending ( ) on the inter-frame motion e e 1 1. If e e 1 R t 1 =, then E = tr. 0 1 According to the last section, if g 1 = k 1 q e 1 and g = k q e, then there exist matrices K 1 and K, respectively depending on k 1 and k such that g replacements (projective linear) K f 1 (X) = P K1 1 g 1 (X) and f (X) = P K 1 g (X) where the operator is defined in equation (). Now substitute the right hand sides into equation (7): (nonlinear) h h (nonlinear) k (similarity) Figure 1: A similarity or change of intrinsics commutes linearly with a projective linear transformation which preserves the sphere. To denote the application of K we write k (X) = K X. Now, suppose a catadioptric projection g = k q, so e = id; we can rewrite the composition: h g = h k q = k h q, (5) Observe the following algebraic derivation: (p h q)(x, y, z, 1) = (p h) [(x, y)/( z + r)] = p [(x, y, z, r)/(r z)] = p(x, y, z, 1), where r = x + y + z. This proves that p h q = p. Putting this equation and equation (5) together yields: p(x) = (p h q)(x) = (p k 1 k h q)(x) = (p k 1 h g)(x) = } P K {{ 1 } h(g(x)). A k Thus we have verified that equation (3) can be satisfied when A k = P K 1 and h is defined in (). In subsequent sections we will use the following shorthand notation: x = h(x). () g (X) T K T P T E P K1 1 g 1 (X) }{{} = 0. (8) F The matrix F is called the parabolic catadioptric fundamental matrix, first introduced in [9]. Thus, in spite of the nonlinearity of q, there is a bilinear constraint on the liftings of corresponding image points in two parabolic catadioptric views. This matrix defines the epipolar geometry for the pair of parabolic projections. For example, the locus of all points y satisfying ỹ T F x = 0 for any constant x is an epipolar circle, and vice versa for all x and any constant y. The equations F x = 0 and F T ỹ = 0 each have two solutions, and each solution pair defines an epipolar point pair (e i,1, e i, ), equivalent to one epipole in a perspective image. 5. Self-calibration In general, the transformations k 1 and k are unknown, therefore we ask: What information about k 1 and k can be inferred from F? We shall show that if k 1 = k = k then under most conditions we can infer k from F. We begin by examining properties of the projective linear transformation specified by the matrix K. Up until now we have only considered its effect on points x lying on the sphere. Clearly, though, it acts on all of P 3. Let us consider its effect on the origin O = (0, 0, 0, 1). Using the definition of K found in equation (1), we can directly calculate its transformation, O = KO: (c x, c y, c x + c y + f 1, c x + c y + f + 1) T. (9)

4 If c x, c y and f are unknown we can calculate them from the point O. Thus, if we are somehow able to determine O, we can find K. We now show that O can be obtained directly from the fundamental matrix. To begin with, since O is the viewpoint of the canonical perspective projection, it satisfies P O = 0. Therefore, if O 1 = K 1 O, then F O 1 = K T P T E ( P K 1 1 K 1 O ) = 0. Similarly, F T O = 0 where O = K O. Consequently O 1 N (F ) and O N (F T ), where N (F ) denotes the nullspace of F. The fundamental matrix, however, is rank two and therefore its nullspaces are two-dimensional. Thus neither O 1 nor O are uniquely defined by these conditions. Yet, if it is known a priori that K 1 = K, then O 1 = O N (F ) N (F T ). Provided N (E) N (E T ), the intersection is a single point in P 3 and we can uniquely recover c x, c y and f. It can be shown that the nullspaces coincide only when R t = t, i.e., the axis of rotation is about the direction of translation.. The Lorentz group The matrix K, defined in equation (1) of the appendix, can be shown to satisfy K T QK = Q. If p = Kp for any point such that p T Qp = 0, then apparently p T Qp = 0. In other words K preserves point-wise the unit sphere. There are, however, many more matrices A which are not in the form given in equation (1) but which do satisfy A T QA = Q. In fact like the set of matrices satisfying A T A = I, i.e., the orthogonal group, the set of all matrices A satisfying A T QA = Q is a Lie group, i.e., a group whose elements form a manifold. This group is called the Lorentz group and is denoted by O(3, 1). The (3, 1) denotes the fact that the diagonal of Q consists of three +1 s and one 1, and there is a study of more general groups O(p, q). The structure of a matrix Lie group is determined by the possible derivatives of curves through the identity matrix. Suppose that A(t) is a curve restricted to O(3, 1) such that A(0) = I. Then its derivative at t = 0 satisfies: A (0) T Q + QA (0) = 0. The derivative is therefore of the form ( ) A ˆx y (0) = y T = L(x, y), (10) 0 for some x, y R 3, and therefore the Lorentz group is sixdimensional. We let so(3, 1) be the set of all such matrices. It can be shown that if X so(3, 1), then e X O(3, 1), where e X = I + X + X / +. A closed-form formula, analogous to Rodrigues formula for skew-symmetric matrices, is given in equation of the appendix. It is proven in [] that e X is surjective on the connected component of the identity in O(3, 1). Looking back at K, notice that the set of all K as c x, c y and f vary over all real numbers, with f > 0, is a threedimensional Lie subgroup of O(3, 1). We shall denote by K the set of all such matrices and call it the calibration subgroup (for parabolic projections). It can be shown that when K O(3, 1), K K if and only if Kp = p. This demonstrates that a subgroup of O(3, 1) is relevant to calibration, what, though, are the effects of the remaining Lorentz transformations? It can be shown that any A O(3, 1) has the following unique decomposition: ( ) R 0 A = K where R O(3) and K K. 0 1 The first part of the decomposition has the effect of rotating the sphere about the origin, and is therefore equivalent to a pure rotation of the scene. Consequently any element of the Lorentz group is a uniquely factorable composition of an element of K, which accounts for the intrinsic parameters of the camera, and an element of O(3), which accounts for the rotation of the scene. 7. Properties of the fundamental matrix We are interested in the following question not resolved in [10]: Under what condition(s) is an arbitrary matrix a parabolic fundamental matrix? Recall that if we let E be the set of perspective fundamental matrices, then E E if and only if E has equal singular values, i.e., for some U, V O(3), E = U T diag(1, 1, 0)V. Let us denote by F the set of all parabolic catadioptric fundamental matrices. The following theorem offers an analogous characterization of any F F. Theorem 1. A matrix F can be decomposed into the form defined in equation (8) if and only if there exist matrices U, V O(3, 1) such that F = U T diag(1, 1, 0, 0) V. (11) Moreover, a decomposition can be found where K 1 = K only if det(f + F T ) = 0. In other words the Lorentzian singular value decomposition of F must have equal Lorentzian singular values. This holds even though F is a bilinear constraint for uncalibrated cameras. One mistake that should not be made is that an orthogonal projection to F will not be obtained by averaging the Lorentzian singular values, for the Frobenius distance between the original matrix and the projected one will not

5 be minimized. Unfortunately this does not answer the question of optimal projection. Because of space constraints, we give only an outline of a proof. The forward is trivial because the K i s absorb the rotations of the singular value decomposition of E, leaving elements of O(3, 1). The reverse relies on the decomposition of the element U O(3, 1) into U = RK where K K and R O(3), stated in the previous section. Note that this theorem can be made more specific and restrict U and V to lie in O(3, 1) 0, the connected component of the identity. 8. Global characteristics of F Finally we consider the following questions: What is the structure of the set of all parabolic fundamental matrices? If it is a manifold what is its dimension? To begin with we make the following observation which is a corollary of Theorem 1: F F if and only if A T F F for any A O(3, 1) 0. In other words, F is closed under left multiplication by elements of O(3, 1) 0. Similarly F F if and only if F A F. Consequently, elements of the group G = O(3, 1) 0 O(3, 1) 0 act upon elements of F via a map ϕ in the following way: ϕ ( (A 1, A ), F ) = A T 1 F A. Theorem 1 guarantees that the range ϕ(g, F 0 ) F. Now suppose that g = (G 1, G ), h = (H 1, H ) and e = (I, I). The map ϕ satisfies ϕ(e, F ) = F and ϕ(g h, F ) = ϕ(g, ϕ(h, F )), where g h = (G 1 H 1, G H ). Any such function is known as a group action. What might this have to do with the structure of F? For example, it is clearly not a group, but the group G acts on it. We start by noting that ϕ, and therefore G, can be used to parameterize F. If ϕ(g, F 0 ) = F, for any constant F 0, in other words is surjective in F, then ϕ(, F 0 ) is a parameterization of F. This is also a corollary of Theorem 1, for if F i=0,1 = U T i ΣV i then ϕ ( (U 1 0 U 1, V 1 0 V 1 ), F 0 ) = F1, where Σ = diag(1, 1, 0, 0). So, ϕ(, F 0 ) is surjective in F. If it were the case that ϕ F0 = ϕ(, F 0 ) provided a oneto-one mapping between G and F, then it would show that F is a group since ϕ F0 would be an isomorphism. However, ϕ F0 is not one-to-one in this case. What, though, is the redundancy in its parameterization of F? Consider the set of elements for which ϕ F0 = F 0, i.e., H F0 = {g G : ϕ(g, F 0 ) = F 0 } G. Since it is closed under composition and inversion, and preserves F 0, it is called the isotropy subgroup of F 0. Furthermore, it can be shown that it is a 3-dimensional Lie subgroup. Now we make the following claim. The left cosets of H F0, i.e., gh F0 = {gh : h H F0 }, are in one-to-one correspondence with the elements of F. Thus F is equivalent to the quotient space G/H F0, the quotient space being the partition of G into the pairwise disjoint left cosets. To prove this claim one must show that ϕ F0 (gh F0 ) = ϕ F0 (gh) is constant for all h H F0, and therefore ϕ F0 (gh F0 ) is well-defined; then we must show that ϕ F0 is injective. Both properties follow from the properties of the group action ϕ or cosets. A theorem of differential geometry, see [5] for example, states that the quotient of a Lie group G and some Lie subgroup H is a manifold of dimension dim G dim H. This and the arguments above justify the following theorem. Since all of the preceding arguments apply equally well to essential matrices, we can also obtain an analogous result. Theorem. The set of parabolic catadioptric fundamental matrices is equal to a quotient of Lie groups: F = O(3, 1) O(3, 1) / H F0, where H F0 is the isotropy subgroup of any F 0 F. Consequently F is a nine-dimensional manifold. Similarly, the set of essential matrices E is given by: E = O(3) O(3) / HE0, where H E0 is the isotropy subgroup of any E 0 E in the context of the natural action ϕ : O(3) O(3) E E. 9. Epipolar geometry estimation Now that the epipolar constraint is shown to be bilinear in the liftings of the corresponding points, most of the perspective algorithms for motion estimation apply. The most important difference is, so far, the absence of a linear orthogonal projection to F. We are not yet aware of a linear 9-point algorithm. However, supposing {(p i, q i )} 1 i N is a set of point correspondences, we can minimize, in the usual way, the Sampson approximation to the geometric error [19]: E(F ) = ) n T ( p i F q i (p i,q i) pt i F q i = i=1 where b i = q i p i and n i=1 f T b i b T i f f T A i A T i f A = ( p i u q i p i v q i p i q i u p i q i v ). The notations p u and p v respectively denote partial differential of h with respect to u and v, evaluated at p = (u 0, v 0 ). The map ϕ F0 conveniently provides us with a global and nowhere-singular, though redundant, parameterization of F. We define φ F0 : R 1 F given by: φ F0 (x 1, y 1, x, y ) = e L(x1,y1)T F 0 e L(x,y),

6 Figure : Estimation of the epipolar geometry found by minimizing the Sampson error. Each green line is the epipolar circle determined from the point in the other frame, hence the points do not lie exactly on the epipolar circles. The epipolar circle corresponding to a given point is highlighted in the region of the point. Note that in parabolic catadioptric images there are always two epipoles present in the image. The light purple circle encodes the intrinsic parameters; its center is the image center and its radius is twice the focal length, it has been found by choosing the least singular value of (F, F T ) T. where L is defined in (10). Equation (13) gives an explicit formula for the exponential map. If p = (x 1, y 1, x, y ), then we can directly calculate the Jacobian J = / pe(φ F0 (p)) and the 1 1 Hessian H = / p E(φ F0 (p)) using the approximation e X = I + X. Algorithm 1. (Motion estimation) 1. Obtain an initial estimate F 0 by guessing (poor) calibration matrices K 1 and K, and estimating an essential matrix E 0. Then F 0 = K T P T E 0 P K1 1.. With the initial estimate, minimize the function E(F ) using Levenberg-Marquardt by explicitly calculating J and H. Let ˆF be the final local minimum. 3. If it is assumed that K 1 = K, then the least singular vector of ( ˆF, ˆF T ) is an estimate of the vector O in equation (9), whereby we can estimate c x, c y and f and consequently K.. Ê = P K T ˆF KP T is an estimate of the essential matrix which can be factored to obtain four possible motions. Only one leads to a consistent reconstruction. We have implemented this algorithm and tested it using features obtained from an automatic point tracker [0]. The estimated epipolar geometry is shown in Figure. In Figure 3 we show three images, the left and right of which are rectifications of the images of Figure. We lack the space here to describe the rectification in this paper, except to say that it is conformal and so locally there is no distortion; it is described in [11]. Using the disparity estimated with a stereo algorithm we show in the middle image a warping of the right image to match the left image. 10. Model deviations We have, until now, assumed that the projection model is ideal, that the camera is exactly orthographic and that the mirror is aligned and has the shape of a paraboloid. In this section we examine through simulation the effects that deviations from these assumptions have on the estimation of the motion. When these deviations occur, there is no closed form solution for the projection of a point in space. Hence, in the simulations, to calculate the projection of a point we use the principle of least action in which the image of a point in space is determined from the path with the shortest distance from that point to the focus of the camera. We now describe four simulations in which a single parameter is varied and errors in rotation and translation are shown. In Figure are graphs of the error in translation and rotation as a function of the independent variable. The error 1 in rotation equals log ˆRR 0 where ˆR and R 0 are the estimated and true rotations respectively. The error in translation is similarly cos 1 ˆt T t 0, where ˆt and t 0 are the estimated and true translations respectively. From left to right in Figure : A perspective camera with focal length 0 times the focal length of the mirror. The optical axes remain parallel but are displaced a distance x lying between 0.5 to 3 times the mirror s focal length. An orthographic camera and a mirror which is rotated at angle of θ about its focus, where 0 θ 5. A perspective camera, with optical axis aligned with mirror, and where the ratio ρ of the distance between the mirror focus and camera focus to the mirror focal length varies between (orthographic) and 13.

7 Figure 3: Left: Rectification of left image in Figure. Middle: Warping of right rectification to left using estimated disparity. Right: Rectification of right image in Figure. An orthographic camera and a mirror whose eccentricity ε varies from 0.9 to 1.1 (1 is a paraboloid). We find that the motion estimation algorithm is robust to modest deviations from the ideal assumptions. It appears that the usage of a perspective camera has little effect on the results, though there is a greater effect if the optical axes are not aligned. 11. Conclusion In conclusion we have presented a framework for structure from motion for parabolic catadioptric systems. The methods presented formulate the structure from motion in a way which linearly decouples the structure and the motion despite a non-linear projection model. We demonstrated local and global properties of the manifold of fundamental matrices and the formulation of the set as a quotient of groups, and which has given us insight into the perspective case. We propose an algorithm for estimation of the epipolar geometry which minimizes on the nine-dimensional manifold of fundamental matrices. Finally, we have shown that these algorithms are robust to small deviations from ideal assumptions. We hope to address some of the following problems arising out of this work: Estimation algorithms. Ideally we would like to find linear algorithms for estimation of fundamental matrices. Another issue is that the decreased resolution per viewing angle effectively decreases the magnitude of the baseline, therefore algorithms robust to small baselines should be addressed. Harmonic analysis. We believe one of the most important contributions of this paper is the characterization of the set of bilinear forms as a quotient space of Lie groups. Because of their symmetry, it is possible to define the Fourier transform of functions on such spaces. Is it possible to combine elements of structure-from-motion with methods in signal processing? Control on F. Control has been investigated on quotient spaces, e.g. [], can these ideas be extended to visual servoing directly on the quotient space F? Acknowledgements. The authors are grateful for support through the following grants: NSF-IIS , NSF-EIA , NSF-IIS and DARPA DAAH The authors would also like to thank Rana Molana, Rene Vidal, Jana Košecká and Ameesh Makadia for their helpful comments on the paper. K = Appendix First we give the matrix K depending on the intrinsic parameters c x, c y and f: f 0 f c x f c x 0 f f c y f c y c x c y 1 c x c y + f 1 + c x + c y f c x c y 1 c x c y f 1 + c x + c y + f λ 3 sinh λ 1+λ 3 1 sinh λ λ 1 λ (λ 1 λ ). (1) Second we give the analog to Rodrigues formula for matrices A so(3, 1): ( ) ( ) e A λ = sinh λ 1 λ 1 sinh λ A λ 1 λ (λ 1 λ ) 3 cosh λ + 1 cosh λ A + λ 1 λ ( ) ( ) λ A cosh λ 1 +λ 1 cosh λ I. (13) λ 1 λ where {λ 1, λ 1, λ, λ } are the eigenvalues of A. If A has two zero eigenvalues, then there is a simpler form resembling Rodrigues formula which we leave to the reader. References [1] D. Aliaga. Accurate catadioptric calibration for real-time pose estimation in room-size environments. In Proc. Int. Conf. on Computer Vision, pages 17 13, Vancouver, Canada, Jul. 9-1, 001. [] A. Baker. Matrix Groups: An Introduction to Lie Group Theory. Springer Verlag, 00. [3] S. Baker and S. Nayar. A theory of single-viewpoint catadioptric image formation. International Journal of Computer Vision, 35:175 19, [] R. Benosman and S. Kang. Panoramic Vision. Springer- Verlag, 000. [5] W. Boothby. An Introduction to Differentiable Manifolds and Riemannian Geometry. Academic Press, London, 198.

8 1 translation error.5 translation error 3.5 translation error 3.5 translation error translation direction RMS error in degrees rotation error rotation error rotation error rotation error replacements rotation RMS error in degrees displacement from axis in multiples of focal length rotation of the parabola in degrees ratio of camera focal length to mirror focal length eccentricity of mirror Figure : Experiments demonstrating robustness to deviations from the ideal assumptions. See text for details. [] R. W. Brockett. System theory on group manifolds and coset spaces. SIAM J. Control, 10():5 8, 197. [7] P. Doubek and T. Svoboda. Reliable 3d construction from a few catadioptric images. In Workshop on Omnidirectional Vision, Copenhagen, June, pages 71 78, 00. [8] C. Geyer and K. Daniilidis. Catadioptric projective geometry. International Journal of Computer Vision, 3:3 3, 001. [9] C. Geyer and K. Daniilidis. Structure and motion from uncalibrated catadioptric views. In IEEE Conf. Computer Vision and Pattern Recognition, Hawaii, Dec , 001. [10] C. Geyer and K. Daniilidis. Properties of the catadioptric fundamental matrix. In Proc. Seventh European Conference on Computer Vision, pages 10 15, Copenhagen, Denmark, 00. [11] C. Geyer and K. Daniilidis. Conformal rectification of omnidirectional stereo pairs. In Workshop on Omnidirectional Vision and Camera Networks, Madison, WI, June, 003. [1] J. Gluckman and S. Nayar. Ego-motion and omnidirectional cameras. In Proc. Int. Conf. on Computer Vision, pages , Bombay, India, Jan. 3-5, [13] J. Gluckman and S. Nayar. Rectifying transformations that minimize resampling effects. In IEEE Conf. Computer Vision and Pattern Recognition, Hawaii, Dec , 001. [1] R. Hartley and A. Zisserman. Multiple View Geometry. Cambridge Univ. Press, 000. [15] S. Kang. Catadioptric self-calibration. In IEEE Conf. Computer Vision and Pattern Recognition, pages I 01 07, Hilton Head Island, SC, June 13-15, 000. [1] Y. Ma, S. Soatto, J. Košecká, and S. Sastry. An Invitiation to 3D Vision: From Images to Geometric Models. Springer Verlag, New York, 003. [17] J. Neumann, C. Fermüller, and Y. Aloimonos. Eyes from eyes: New cameras for structure from motion. In Workshop on Omnidirectional Vision, Copenhagen, June, pages 19, 00. [18] R. Pless. Discrete and differential two-view constraints for general imaging. In Workshop on Omnidirectional Vision, Copenhagen, June, pages 53 59, 00. [19] P. Sampson. Fitting conic sections to very scattered data: an iterative refinement of the Bookstein algorithm. Computer Graphics and Image Processing, 18:97 108, 198. [0] J. Shi and C. Tomasi. Good features to track. In IEEE Conf. Computer Vision and Pattern Recognition, pages , Seattle, WA, June 1-3, 199. [1] P. Sturm. Mixing catadioptric and perspective cameras. In Workshop on Omnidirectional Vision, Copenhagen, June, pages 37, 00. [] T. Svoboda, T. Pajdla, and V. Hlavac. Epipolar geometry for panoramic cameras. In Proc. 5th European Conference on Computer Vision, pages 18 31, [3] C. J. Taylor. Video plus. In IEEE Workshop on Omnidirectional Vision, Hilton Head, SC, June 1, pages 3 10, 000.