Camera Models and Affine Multiple Views Geometry

Transcription

1 Camera Models and Affine Multiple Views Geometry Subhashis Banerjee Dept. Computer Science and Engineering IIT Delhi May 29,

2 1 Camera Models A Camera transforms a 3D scene point = (, Y, Z) T into an image point x = (x, y) T. 1.1 The Projective Camera The most general mapping from P 3 to P 2 is x 1 x 2 x 3 = T 11 T 12 T 13 T 14 T 21 T 22 T 23 T 24 T 31 T 32 T 33 T 34 where (x 1, x 2, x 3 ) T and ( 1, 2, 3, 4 ) T are homogeneous coordinates related to x and by (x, y) = (x 1 /x 3, x 1 /x 3 ) (, Y, Z) = ( 1 / 4, 2 / 4, 3 / 4 ) The transformation matrix T = T ij has 11 degrees of freedom since only the ratios of elements T ij are important. (see Zisserman and Mundy). 1.2 The Perspective Camera A special case of the projective camera is the perspective (or central) projection, reducing to the familiar pin-hole camera when the leftmost 3 3 sub-matrix of T is a rotation matrix with its third row scaled by the inverse focal length 1/f. The simplest form is: T p = which gives the familiar equations x y /f 0 = f Z Each point is scaled by its individual depth, and all projection rays converge to the optic center. Y

3 1.3 The Affine Camera The affine camera is a special case of the projective camera and is obtained by constraining the matrix T such that T 31 = T 32 = T 33 = 0, thereby reducing the degrees of freedom from 11 to 8: x 1 x 2 x 3 = T 11 T 12 T 13 T 14 T 21 T 22 T 23 T T In terms of image and scene coordinates, the mapping takes the form x = M + t where M is a general 2 3 matrix with elements M ij = T ij /T 34 while t is a general 2-vector representing the image center. The affine camera preserves parallelism. 1.4 The Weak-Perspective Camera The affine camera becomes a weak-perspective camera when the rows of M form a uniformly scaled rotation matrix. The simplest form is yielding, M wp = f Z ave T wp = Z ave /f and x y = f Z ave Y This is simply the perspective equation with individual point depths Z i replaced by an average constant depth Z ave The weak-perspective model is valid when the average variation of the depth of the object ( Z) along the line of sight is small compared to the Z ave and the field of view is small. We see this as follows. Expanding the perspective projection equation using a Taylor series, we obtain x = f Z ave + Z Y = f ( 1 Z ( ) Z 2 ) +... Z ave Z ave Z ave Y

4 When Z << Z ave only the zero-order term remains giving the weak-perspective projection. The error in image position is then x err = x p x wp : x err = f ( ) Z Z ave Z ave + Z Y showing that a small focal length (f), small field of view (/Z ave and (Y/Z ave ) and small depth variation ( Z) contribute to the validity of the model. 1.5 The orthographic camera The affine camera reduces to the case of orthographic (parallel) projection when M represents the first two rows of a rotation matrix. The simplest form is T orth = yielding, M orth = and x y = Y 2 Affine Multiple Views Geometry Epipolar Geometry Structure determination Affine structure Euclidean structure 2.1 Affine Epipolar Geometry When the perspective effects are small, the problem of locating perspective epipolar lines becomes ill-conditioned. In such cases it is convenient to assume the parallel projection model of the affine camera which explicitly models the ambiguities. The affine epipolar constraint can be described in terms of the affine fundamental matrix F as p T Qp = 0, i.e., x i y i a 0 0 b c d e x i y i 1 = 0

5 Z Scene point Z ave Average depth plane f Image plane Optic Center p wp orth Figure 1: 1D image formation with image plane at Z = f. p, wp and orth are the perspective, weak-perspective and orthographic projections respectively. where p = (x, y, 1) T and p = (x, y, 1) are homogeneous 3-vectors representing corresponding image points in two views. (See Shapiro, Zisserman and Brady). To derive the above, we write M as (B b) where B is a general (non-singular) 2 2 matrix and b is a 2 vector. The projection equation then gives x i = B i Y i + Z i b + t Similarly, for M A, we have x i = B i Y i + Z i b + M D + t Eliminating scene coordinates ( i, Y i ) gives x i = Γx i + Z i d + ɛ where Γ = B B 1, d = b B B 1 b and ɛ = t Γt + M D.

6 x1 x2 x3 u 1 u 3 u 2 x1 O x2 x3 xe x e u 3 u 1 u 2 O (a) (b) Figure 2: Affine and Perspective Epipolar Geometries Γ and d are functions only of camera parameters {M, M } and the motion transformation A, while ɛ explains the motion of the reference point (centroid) and depend on the translation of the object D and the camera origins t and t. This equation shows that x i associated with x i lies on a line (epipolar) on the second image with offset Γx i + ɛ and direction d. The unknown depth Z i determines how far along this line does x i lie. Inverting the equation we obtain x i = Γ 1 x i Z i Γ 1 d Γ 1 ɛ The translation invariant versions of these equations are x i = Γ x i + Z i d x i = Γ 1 x i Z i Γ 1 d We can eliminate Z i from the above equations and obtain a single equation in terms of image measurables: (x i Γx i ɛ).d = 0 where, d = (d x, d y ) and its perpendicular d = (d y, d x ). This equation can be written as ax i + by i + cx i + dy i + e = 0 where (a, b) T = d, (c, d) T = Γ T d and e = ɛ T d. This gives us x i y i a 0 0 b c d e x i y i 1 = 0

7 2.1.1 Computation of Affine epipolar Geometry Given correspondences in two views the affine fundamental matrix can be computed using orthogonal regression by minimizing n 1 1 n 2 (r i n + e) 2 i=0 Here r i = (x i, y i, x i, y i ) T and n = (a, b, c, d) T. The minimization finds a hyper-plane that globally minimizes the sum of the squared perpendicular distances between r i and the hyper-plane. Defining v i = r i r and W = n 1 i=0 v i v i T it can be shown that the solution satisfies the eigenvector equation Wn = λ i n, n 2 = Affine Structure Consider a set of n 3D world points i (i = 0,..., n 1) in affine (non-rigid) motion described by i = A i + D where i is the new 3D position of the i th point, A is an arbitrary 3 x 3 matrix and D is a 3-vector representing translation. Removing the effects of translation by registering the points with respect to a reference point 0 to obtain = 0 and = 0 = A Affine projections If the affine camera models for the two views are given by the parameters {M, t} and {M, t } respectively, then x = M and x = M = M A

8 Basis and Affine structure Now, consider four non-coplanar scene points 0,..., 3 with 0 as the origin. We define three axis vectors E j = j 0 for j = 1,..., 3. {E 1, E 2, E 3 } form a basis for the 3D affine space and any of the n vectors can be represented in this basis as i 0 = α i E 1 + β i E 2 + γ i E 3 for i = 1,..., n where (α i, β i, γ i ) are the 3D affine coordinates of i. We call the 3D affine coordinates the affine structure of the point i. It can be shown that the affine structure remains invariant to affine motion with respect to the transformed basis, that is, i = α i E 1 + β i E 2 + γ i E 3 i = α i E 1 + β i E 2 + γ i E (1) 3 where E j = AE j. Computation of Affine structure From the above we obtain x i = x i x 0 = α i e 1 + β i e 2 + γ i e 3 x i = x i x 0 = α i e 1 + β i e 2 + γ i e 3 (2) where, e i = ME i and e i = M E i. Thus, to compute the affine structure, we require two images with at least four points in correspondence, i.e., {x 0, x 1, x 2, x 3 } and {x 0, x 1, x 2, x 3} These correspondences establish the bases {e 1, e 2, e 3 } and {e 1, e 2, e 3} provided no two axes, in either images, are collinear. Each additional point gives four equations in 3 unknowns xi x i = e1 e 2 e 3 e 1 e 2 e 3 α i β i γ i and the affine structure can be computed. The redundancy in the system enables us to verify whether the affine projection model is valid.

9 2.2.1 Tomasi and Kanade factorization In case n point correspondences (n 4) over k views (k 2) are available, we use the factorization procedure of Tomasi and Kanade to obtain the bases and structure. Their formulation can be written as an extension of the above equation as x 1 x 2... x n 1 x 1 x 2... x n 1 x 1 x 2... x n 1. = e 1 e 2 e 3 e 1 e 2 e 3 e 1 e 2 e 3. α 1 α 2... α n 1 β 1 β 2... β n 1 γ 1 γ 2... γ n 1 where the left measurement matrix W represents the n point correspondences in k views and has dimensions 2k x (n 1). The matrices on the right, M (2k x 3) and S (3 x (n 1)), are called motion and structure matrices respectively. The matrix S gives the invariant affine structure of the n points in motion, and the i t h row of M, M(i), along with the corresponding image center x 0 (i), gives the projection parameters for the i t h view { M(i), x 0 (i)}. Clearly, in the absence of noise, W must have a rank at-most 3. Tomasi and Kanade perform a singular value decomposition of W and use the 3 largest eigenvalues to construct M and S. If the SVD returns a rank greater than 3, then the affine projection model is invalid and we use this as a check. The rank 2 case signifies either a planar object (which is not possible for facial images!) or degenerate motion. In such a case, the 3D affine structure cannot be determined and the views are related by 2D affine transformations. The 2D affine structure can then be recovered in only two axes using the same formalism Image transfer and linear combination of views Once the affine structure has been computed, it can be used to generate a new view of the object ( transfer ) by simply selecting a new spanning set {e 1, e 2, e 3}. No camera calibration is needed. Note that this is same as choosing a new projection matrix M. x i = x 0 + α i e 1 + β i e 2 + γ i e k If the affine structure is not of interest (graphics), it is possible to bypass the affine coordinates and express the new image coordinates x directly in terms of the first two sets of image coordinates x and x. One can write the projection equations in the first two views as x = G x = G

10 where G and G are 2 3 matrices with rows {G 1, G 2 } and {G 1, G 2} respectively. The new view can be similarly written as x = G where G has rows {G 1, G 2}. Now, any three rows of {G 1, G 2, G 1, G 2} define a linearly independent spanning set for A 3, say {G 1, G 2, G 1}. So, there exists scalars such that G = a1 a 2 b 1 b 2 G + a3 0 b 3 0 G Then, x = G gives x = a1 a 2 b 1 b 2 x + a3 0 b 3 0 x = a1 a 2 a 3 b 1 b 2 b 3 x y x Thus, if images of an object are obtained using affine cameras, then a novel view can be expressed as a linear combination of views (this is useful for object recognition) Change of basis Given the current spanning set {e 1, e 2, e 3 } and {e 1, e 2, e 3} in the two images, we have that α xi e1 e x = 2 e i 3 i e 1 e 2 e β i 3 γ i Suppose that we now wish to express the same set of points using alternative spanning sets {h 1, h 2, h 3 } and {h 1, h 2, h 3}, the new affine coordinates must obey xi x i Koenderink and Van Doorn = h1 h 2 h 3 h 1 h 2 h 3 ˆα i ˆβ i ˆγ i Instead of choosing E 3 = 3 0, KVD choose E k = k (i.e. the direction of viewing in the first frame). Since e k = ME k = 0, the projection of E k in the first image is degenerate reducing it to a single point. Thus, only two basis vectors are chosen in the first image x i = α i e 1 + β i e 2

11 P Q ~ P ~ Q reference plane p q ~ p p q V 1 V 2 ~ q Figure 3: Affine Structure from Motion In the second image, the third axis vector is no longer degenerate, given by e k = ME k = MAE k. e k is actually an epipolar line. If we use e 1 and e 2 to predict the position where each point would appear in image 2, as if they lay on plane {E 1, E 2 }, we get ˆx i = x 0 + α i e 1 + β i e 2 the disparity between the predicted position and the observed position is solely due to the γ i component 2.3 Rigid reconstruction Assumptions Rigid transformation (isometry) x i = x 0 + α i e 1 + β i e 2 + γ i e k x i ˆx i = γ i e k

12 Affine projection Metric constructions Procedure Image Plane Fronto-parallel plane Figure 4: Euclidean reconstruction 1. Translation in fronto-parallel plane merely produces a shift in projections. This can be factored out by putting two projections of O in to coincidence. 2. Rotation can be decomposed into i) a rotation in the image plane (cyclorotation) and rotation about an axis in the fronto-parallel plane. Projection of the third affine frame vector is the projection of a plane perpendicular to the axis of rotation in the fronto-parallel plane. One can reconstruct the projection in the first view (only affine construction) and factor out the relative rotation in the two images. This yields the cyclo-rotation. 3. Since the axis of rotation is known in both views, one can find the overall scale difference due to translation in depth. Points on the axis of rotation do not

13 rotate. Consider the projection of all image points on to this axis. If they differ in the two views, they must differ by only a constant scale factor. Otherwise, the rigidity assumption is falsified. 4. Now the two views differ only by a rotation about an axis in the fronto-parallel plane. Define a Euclidean frame (e 1, e 2, e 3 ), such that e 1,2,3 are unit vectors with e 1 along the axis of rotation and e 3 along the line of sight. Let G 1 e 1 + G 2 e 2 denote the depth gradient of a plane in the object. That is, the depth of a point αe 1 + βe 2 in the image with respect to the fronto-parallel plane is αg 1 + βg 2. Note that G 1 = tan σ cos τ G 2 = tan σ sin τ where σ is the slant and τ is the tilt of the plane. Consider any triangle OY in the plane. Let the coordinates of and Y be ( 1, 2 ) and (Y 1, Y 2 ) respectively. Then the third coordinates must be 3 = G G 2 2 Y 3 = G 1 Y 1 + G 2 Y 2 For a given turn ρ the rotation can be represented by cos ρ sin ρ 0 sin ρ cos ρ Of the three transformed coordinates, the first one is trivially unchanged and the third one is not observable. The second coordinate is observable, and the equations are: 2 1 = 2 0 cos ρ sin ρ(1g G 0 2 ) Y2 1 = Y2 0 cos ρ sin ρ(y1 0 G 1 + Y2 0 G 2 ) here the upper indices label the views and the lower indices label the components. Because the turn ρ is unknown, we eliminate it from these equations to obtain a single equation in (G 1, G 2 ). This equation represents a one-parameter solution for the two view case. The parameter is the unknown turn ρ. The equation is quadratic in (G 1, G 2 ) with the linear term absent; and represents a hyperbola in the (G 1, G 2 ) space (please derive it).

14 5. Repeating the steps above between the second and a third view, we obtain a pair of two view solutions. Each two view solution represents a one-parameter family of solutions. The one-parameter families for the 0-1 transition and the 1-2 transition are represented by the hyperbolic loci in the gradient space. The pair of hyperbola has either two or four intersections. The case of no intersection occurs only in the non-rigid case. If the motion is rigid, then there has to be one solution and hence a pair of them. The intersections represent either one or two pairs of solutions that are related through a reflection in the fronto-parallel plane.