A Linear Algorithm for Computing the Homography from Conics in Correspondence

Transcription

1 Journal of Mathematical Imaging and Vision 13, , 2000 c 2000 Kluwer Academic Publishers Manufactured in The Netherlands A Linear Algorithm for Computing the Homography from Conics in Correspondence AKIHIRO SUGIMOTO Advanced Research Laboratory, Hitachi, Ltd, Japan Abstract This paper presents a study, based on conic correspondences, on the relationship between two perspective images acquired by an uncalibrated camera We show that for a pair of corresponding conics, the parameters representing the conics satisfy a linear constraint To be more specific, the parameters that represent a conic in one image are transformed by a five-dimensional projective transformation to the parameters that represent the corresponding conic in another image We also show that this transformation is expressed as the symmetric component of the tensor product of the transformation based on point/line correspondences and itself In addition, we present a linear algorithm for uniquely determining the corresponding point-based transformation from a given conic-based transformation up to a scale factor Accordingly, conic correspondences enable us to easily handle both points and lines in uncalibrated images of a planar object Keywords: conic correspondences, homography, uncalibrated images, tensor product, linear computation, 2-D objects 1 Introduction One of the fundamental difficulties in recognizing objects from images is how to deal with a number of images obtained from the same object Clarifying the relationship between images of the same object is thus of fundamental importance Knowledge of the relationship between images provides us with many advantages in handling images of the same object For example, it allows us to know the number of stored images required to establish a geometric model of an object We can also synthesize a new image of an object from its stored images with the help of the relationship between images Moreover, we have only to transmit the stored images of an object to a recipient, who can then synthesize an image of the object from any viewpoint Thus, the relationship between images of the same object plays a key role in important problems in computer vision and multimedia including three-dimensional Present address: Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University, Sakyo, Kyoto , Japan; sugimoto@ikyoto-uacjp reconstruction from multiple images, object recognition, image synthesis and image coding A conic is one of the most important image features This is because many man-made objects have circular parts, and circles are perspectively projected onto conics Furthermore, the conic is a more compact primitive than points or lines, and can be more robustly and more exactly extracted from images In addition, finding correspondences between conics is much easier than that between points Unlike points, conics have features that distinguish one from another and can be used to narrow down the possible matches Hence, developing algorithms dealing with conics is significant Nevertheless, there are fewer articles (for instance, [12 16, 23, 26, 27]) dealing with conics to clarify the relationship between images while the multilinear constraints on multiple images of points or lines were derived See, for example, [3, 4, 8 11, 19 21, 24] for points and [3, 6 8] for lines Ma [12] and Ma, Si, and Chen [13] developed an analytical method, based on conic correspondences, for motion and structure reconstruction and pose determination from stereo images This method, however, is developed under the

2 116 Sugimoto assumption that the camera is calibrated In addition, the procedure requires complex nonlinear computation in the general case to obtain closed-form solutions and it does not ensure the uniqueness of the solution Quan [14, 15] showed that two polynomial constraints are satisfied by corresponding conics in two uncalibrated perspective images Based on this result, Quan [14, 15] addressed a geometric invariant and conic reconstruction from two images However, it has not yet established the uniqueness of the solution This is due to the nonlinearity of the derived polynomial constraints Weiss [26, 27] clarified that two conics are sufficient for calibration under the affine projection and derived a nonlinear algorithm for calibration Here perspective images are not concerned and complex nonlinear computation is also required As seen above, methods dealing with conics are developed based on nonlinear constraints on conics The procedures therefore require complex nonlinear computation and the uniqueness of the solution is not ensured This paper presents a study, based on conic correspondences, on the relationship between two perspective images acquired by an uncalibrated camera We show that for a pair of corresponding conics, the parameters representing the conics satisfy a linear constraint To be more specific, the parameters representing a conic in one image are transformed by a fivedimensional projective transformation (called a conicbased transformation) to the parameters representing the corresponding conic in another image We also establish the link between this transformation and the point-based transformation 1, ie, the transformation based on point correspondences in the two images We show that a conic-based transformation is expressed as the symmetric component of the tensor product of the corresponding point-based transformation and itself Furthermore, we exploit the problem of determining the corresponding point-based transformation from a given conic-based transformation This problem is reduced to solving an overdetermined system of nonlinear equations Full use of redundant information is made so that the problem is solved with only linear computation Our method is simple and ensures the uniqueness of the solution Accordingly, conic correspondences enable us to easily handle both points and lines in uncalibrated images of a planar object In particular, when we observe a planar object, we can use conic correspondences to transfer one image to another image and to establish point correspondences and line correspondences in two images (see Fig 1) Note that a part of this work was presented in [22] Figure 1 Two views of a planar object This paper is organized as follows In Section 2, we discuss the point-based transformation between two images of coplanar points in 3-D within the framework of projective geometry In Section 3, we first clarify that when we observe a conic in 3-D from two different viewpoints, the parameters characterizing its images are related by a five-dimensional projective transformation We then investigate the relationship between a conic-based transformation and its corresponding point-based transformation and establish the link between them In Section 4, we present a linear algorithm for uniquely determining the corresponding point-based transformation from a conic-based transformation up to a scale factor Description of the proposed algorithm is presented in Section 5 Experimental results for evaluating the performance of the proposed algorithm is presented in Section 6 2 Plane Transformation In this section, we discuss the relationship between two different images of coplanar points on the basis of the framework of projective geometry Note that if not explicitly stated, the coordinates of a point/line are understood to be homogeneous throughout this paper An introduction to elementary projective geometry can be found in [2] or [18] Let P n be the n-dimensional projective space over the real number field R We embed the plane in 3-D into P 2 so that the Euclidean coordinates 2 (X, Y ) T are expressed by the homogeneous coordinates (X, Y, 1) T We also embed the image planes I (V is the viewpoint) and I (V is the viewpoint) in P 2 Embedding, I and I, respectively, in P 2 allows us to express the transformation from I to I as a plane projective transformation (see Fig 2) That is, letting the homogeneous coordinates of the corresponding points in I and I be x and

3 Linear Algorithm Conic-Based Relationship Between Two Images We exploit a relationship between two images based on conic correspondences, and then establish the link between the conic-based relationship and a point-based transformation 31 Conic-Based Transformation Figure 2 Relationship between two different images of points in plane (V and V are the view-points, and I and I are the images) x,wehave x M 1 x, where M 1 is a 3 3 nonsingular matrix and implies the equality sign up to a scale factor In this formulation, all the information of camera parameters is included in M; we need not assume that the camera is calibrated M is known as the homography relating I and I In this paper, however, we call M a point-based transformation Every time we observe a point in and are given the correspondence between its images in I and I, we have two independent linear constraints on the entries of M We can thereby uniquely determine the point-based transformation up to a scale factor if we observe four points (any three of them are not aligned) in A line and a point are dual each other in P 2 That is, lines have a one-to-one correspondence to points When we observe a line in, therefore, the transformation between the coordinates of corresponding lines in I and I is expressed in terms of the point-based transformation In fact, denoting by l and l the coordinates of corresponding lines in I and I, we see that l and l are related by M in the form of A conic in the image plane is uniquely expressed in the form of a quadratic homogeneous equation: x T C x = 0, (31) where C is the 3 3 symmetric matrix given by a f e C f b d e d c Since in (31) one conic corresponds to one C up to scale and another conic corresponds to another C up to scale, we see that conics have a one-to-one correspondence with Cs ie, 3 3 symmetric matrices up to scale C has six independent entries and only the ratios between them are significant In other words, C bijectively corresponds to a point in P 5 Hereafter, for conic (31), we refer to C as a conic matrix and to q = (a, b, c, d, e, f ) T ( P 5 ) as a conic vector Conics are transformed into conics under projective transformations Denote by q and q the conic vectors for the corresponding conics in I and I, both of which are obtained by observing a conic in plane (see Fig 3) Since each conic in the image plane bijectively corresponds to a point in P 5, q is transformed to q by l M T l The relationship between two images with respect to points or lines is aggregated into a point-based transformation When a point-based transformation between two images is given, for example, we can establish both the point correspondences and the line correspondences between the two images We can also predict the coordinates of points or lines in the second image from those in the first image Figure 3 Two different images of conics in plane (V and V are the viewpoints)

4 118 Sugimoto a five-dimensional projective transformation That is, when we let S bea6 6matrix, we have q S q (32) Here S is nonsingular due to Remark 32 below (32) gives the linear constraint on the corresponding conics in two images when we observe a conic in space In this paper, we call S a conic-based transformation When we observe coplanar conics from two different viewpoints, a conic-based transformation relates their two images Every time we observe a conic in and are given the correspondence between its images in I and I, we have five independent linear constraints on the entries of S We can thereby uniquely determine the conic-based transformation up to a scale factor if we observe seven conics in a general position in Here seven conics are said to be in a general position when the seven points in P 5 that correspond to the seven conics can be a projective coordinate frame for P 5 We thus obtain the following proposition that is known in projective geometry [18] Proposition 31 [18] Suppose that we observe a conic in plane and that we are given the correspondence of its images in planes I and I Then we can uniquely determine the conic-based transformation between I and I up to a scale factor if we observe seven conics in a general position in In particular, if nothing is known about the entries of S, at least seven conics are needed Remark 31 We see that a conic-based transformation S belongs to an eight-dimensional submanifold of all 6 6 matrices defined up to scale This is because two images are essentially determined by a point-based transformation M and because M has only eight independent parameters In other words, the entries of S are not independent; algebraic constraints exist on the entries of S and only eight entries are independent (See Appendix A for the properties of this submanifold) This implies that using these constraints allows us to reduce the number of conics to determine S In principle, it should be possible to estimate S from two conics since only eight entries of S are independent and each conic gives five independent constraints on them 32 Link Between Conic-Based Transformation and Point-Based Transformation When points are subject to a point-based transformation M, the conic matrix C is transformed to C M T CM (33) This equation indicates that each entry of C is expressed as the linear combination of the entries of C and that the coefficients are quadratic homogeneous in the entries of M Each entry of S is thus quadratic homogeneous in the entries of M In fact, if we define 3 Q ijkl := M ki M lj (i, j,k,l {1,2,3}) and then rewrite (33) in terms of the entries of the conic vectors, it follows that C (ij) σ(k,l) Q (ij)(kl) C (kl), (34) (kl) where C (kl) denotes the entry of the conic vector that is deduced from C kl (C(ij) is defined in the same manner) The notation ( ) implies the symmetrization of the indices inside parentheses; we define that the symmetrized indices are aligned in such a way as (11), (22), (33), (23), (31), (12) Moreover, function σ is defined by { 1 (k = l) σ(k,l):= 2 (k l) To be concrete, C (kl) and Q (ij)(kl) are given as follows: C (kl) := C kl + C lk 2 = C kl (since C T = C), Q (ij)(kl) := Q ijkl + Q jikl + Q ijlk + Q jilk 4 = M ki M lj + M kj M li 2 Here we used tensor analysis See [17] for details of tensor analysis and the notations From (32) and (34), we establish the link between a conic-based transformation S and its corresponding point-based transformation M: S (ij)(kl) σ(k,l)m (k(i M l) j) (35) Note that, from the definition, M (k(i M l) = M ki M lj + M kj M li j) 2 We see that we obtain this equation by taking the tensor product of M and itself, and then symmetrizing the rows and the columns respectively Proposition 32 Let S be a conic-based transformation and M be its corresponding point-based transformation We denote by S(M M) the matrix that is

5 Linear Algorithm 119 deduced from the tensor product of M and itself by symmetrizing the rows and the columns respectively Then, we have the following link between S and M: S S(M M) D, where D is the 6 6 diagonal matrix whose entries are 1, 1, 1, 2, 2 and 2, ie, D = diag(1, 1, 1, 2, 2, 2) Remark 32 It follows from (35) that S is nonsingular iff M is nonsingular We see that a conic-based transformation is essentially expressed as the symmetric component of the tensor product of the corresponding point-based transformation and itself Therefore, a conic-based transformation is determined from its corresponding point-based transformation Then, is the converse true? Namely, can we determine the corresponding point-based transformation from a conic-based transformation? In the next section, we answer this question affirmatively Remark 33 In general, two coplanar conics algebraically have four intersection points though they may be complex points However, we have no way, in particular in the case of complex points, to establish the correspondences between intersection points in two images (we face a combinatorial problem even when the conics actually have real intersection points) In this sense, determining the corresponding point-based transformation from a conic-based transformation is not trivial Moreover, deriving a linear algorithm for the determination is significant from the practical point of view 4 Decomposing Conic-Based Transformation to Point-Based Transformation Our aim here is, for a given Ŝ SD 1, to uniquely determine M up to a scale factor We have eight unknown entries of M and 36 quadratic constraints on them up to a scale factor The problem is thus reduced to solving an overdetermined system of nonlinear equations Here we develop a method that makes full use of the redundancy involved in Ŝ and uniquely determines M up to a scale factor with only linear computation We remark that for an unknown nonzero real number ρ, is satisfied Ŝ (ij)(kl) = ρ M (k(i M l) j) (41) 41 Column-Based Decomposition We derive the differences of the column vectors of M by handling the row vectors of Ŝ Denote by u (ij) the (ij)th row vector of Ŝ ((ij) = (11), (22),,(12)) and also denote by c i the ith column vector of M (i = 1, 2, 3) Defining a mapping from a vector in R 6 to a symmetric 3 3 matrix, x 1 x 6 x 5 Mtx : (x 1, x 2, x 3, x 4, x 5, x 6 ) T x 6 x 2 x 4, x 5 x 4 x 3 we have Mtx ( ) ρ ( u (ij) = ci c T j +c j c T ) i, 2 which yields 4 Mtx ( ) u (ii) + u ( jj) 2u (ij) =ρ(c i c j )(c i c j ) T = ρ c i c j 2 (c i c j ) c i c j (c i c j ) T c i c j This equation gives the spectrum resolution of Mtx (u (ii) + u ( jj) 2u (ij) ) Proposition 41 Let i jmtx(u (ii) +u ( jj) 2u (ij) ) is a matrix of rank 1, and its eigenvalue λij c that has the maximum absolute value and eigenvector c ij ( c ij =1)associated with λij c are given by where λ c ij = sgn(ρ)[ ρ c i c j ] 2, (42) c ij c i c j, (43) { 1 (ρ > 0) sgn(ρ) = 1 (ρ < 0) Remark 41 In the presence of low levels of noise, the rank of Mtx(u (ii) + u ( jj) 2u (ij) )is not less than 2 In this case, two eigenvalues other than λij c are near each other but are far from λij c (λij c is separated from the other eigenvalues) That is, from the viewpoint of magnitude, the two eigenvalues are grouped together, whereas λij c belongs to another group λij c and c ij are therefore computed robustly with respect to noise [1, 5] Expressing c i in terms of λ c ij and c ij, we obtain the following column-based decomposition of M (c ij and

6 120 Sugimoto c i c j do not necessarily have the same sign): c 2 c 3 = µ c 23 γ λ c 23 c23, (44) c 3 c 1 = µ c 31 γ λ c 31 c31, (45) c 1 c 2 = µ c 12 γ λ c 12 c12, (46) where µ ij c {1, 1}and γ = 1/ ρ (44), (45), and (46) together give a system of linear homogeneous equations in the entries of c i and γ Only six of the nine equations are independent This is because the sum of (44), (45), and (46) results in the zero vector Paying attention to the sum of (44), (45), and (46) allows us to determine µ ij c Namely, we first fix a certain component on the right-hand side, and then determine µ ij c so that the sum of the three values of the component becomes zero The following procedure determines µ ij c and it requires only judging which is the greater of given two values Thus, it is robust with respect to noise We denote by cij k the kth component of λij c c ij (i, j,k {1,2,3}) [Determining µ c ij ] Step 1 Fix k {1,2,3}so that cij k 0 for all i and j, and align cij k in order of their magnitude Let c k i 1 j 1 c k i 2 j 2 c k i 3 j 3 Step 2 Evaluate the sign of c k ij : 1 When c k i 3 j 3 > 0, set µ c i 1 j 1 = 1, µ c i 2 j 2 = µ c i 3 j 3 = 1 2 When c k i 1 j 1 < 0, set µ c i 1 j 1 = µ c i 2 j 2 = 1, µ c i 3 j 3 = 1 3 When c k i 2 j 2 > 0 > c k i 3 j 3, if c k i 1 j 1 = max (ij) ck ij, then set µ i c 1 j 1 = µ i c 3 j 3 = 1, µ i c 2 j 2 = 1; if c i k 3 j 3 = max c ij k, then set (ij) µ c i 1 j 1 = µ c i 2 j 2 = µ c i 3 j 3 = 1 4 When c k i 1 j 1 > 0 > c k i 2 j 2, if c k i 1 j 1 = max c k, ij then set (ij) µ i c 1 j 1 = µ i c 2 j 2 = µ i c 3 j 3 = 1; if c k i3 j 3 = max c k, ij then set (ij) µ i c 1 j 1 = µ i c 3 j 3 = 1, µ i c 2 j 2 = 1 42 Row-Based Decomposition In the same manner with the previous subsection, we derive the differences of the row vectors of M by handling the column vectors of Ŝ The procedure here is similar to the one in the previous subsection, but the procedures are independent of each other We denote by v (kl) the (kl)th row vector of Ŝ ((kl) = (11), (22),,(12)) and by r k the kth row vector of M (k = 1, 2, 3) Then, we have Mtx ( v (kk) +v (l l) 2v (kl) ) = ρ r k r l 2 (r k r l ) r k r l (r k r l ) T r k r l Proposition 42 Let k l Mtx(v (kk) +v (l l) 2v (kl) ) is a matrix of rank 1, and its eigenvalue λ r kl that has the maximum absolute value and eigenvector r kl ( r kl = 1) associated with λ c kl are given by λ r kl = sgn(ρ) [ ρ r k r l ] 2, (47) r kl r k r l (48) From Proposition 42 we obtain the row-based decomposition of M: r 2 r 3 = µ r 23 γ λ r 23 r 23, (49) r 3 r 1 = µ r 31 γ λ r 31 r 31, (410) r 1 r 2 = µ r 12 γ λ r 12 r 12, (411) where µ r kl {1, 1} Note that we can easily determine µ r kl (cf the procedure determining µc ij ) (49), (410), and (411) together therefore give a system of linear homogeneous equations in the entries of r k and γ, only six of which are independent Although (44), (45), (46) and (49), (410), (411) respectively give a system of linear homogeneous equations in M and γ, we cannot regard them all together as

7 Linear Algorithm 121 one system of linear homogeneous equations This is because the column-based decomposition and the rowbased decomposition are obtained independently and because in deriving them we did not pay any attention to the fact that c i and r k express the same M In the next subsection, we take this fact into account and modify (44), (45), (46) and (49), (410), (411) so that they all can be regarded as one system We then apply the method of least squares to the system to express M with two parameters 43 Parametrizing Point-Based Transformation The column-based decomposition of M and the rowbased decomposition of M are obtained independently In other words, µ ij c ensures the consistency of signs in (44), (45), and (46), whereas µ ij r ensures the consistency of signs in (49) (410), and (411); µ ij c and µ r kl together, however, do not necessarily ensure the consistency of signs when we regard all the equations together as one system Originally, µ ij c and µ r kl are closely related to each other since c i (i = 1, 2, 3) and r k (k = 1, 2, 3) express the same M in different ways In fact, the kth component of c i is equal to the ith component of r k This indicates that, for example, the order in magnitude of four values, ie, the first and second components of c 2 c 3 and the second and third components of r 1 r 2, has to be considered in determining µ c 23 and µr 12 The consistency of the order in their magnitude is ensured by a pair of µ c 23 and µr 12 or by a pair of µ c 23 and µr 12 The consistency of signs between the column-based decomposition and the row-based decomposition is ensured by a pair of {µ ij c }and {µr kl } or by a pair of {µc ij } and { µ r kl } Proposition 43 allows us to easily determine which pair ensures this consistency That is, we put κ {1, 1}and choose a κ that nullifies ( [ the π(i, j)th component of µ c ij c ij (ij), i j +κµ r ij r ij]) Here, for i j (i, j {1,2,3}), we define π(i, j) :={1,2,3} {i,j} (412) If κ = 1, the consistency is ensured by the pair of {µ ij c } and {µr kl }; otherwise, it is by the pair of {µc ij } and { µ r kl } When noise exists, we choose a κ that minimizes the absolute value of the above equation Since κ is determined by judging which is the greater of given two values, this computation is also robust Proposition 43 The column vectors c i and the row vectors r i of M satisfy (the π(i, j)th component of [(c i c j ) (ij), i j (ij), i j + (r i r j )]) = 0, (the π(i, j)th component of [(c i c j ) (r i r j )]) 0, where π(i, j) is defined in (412) and the summation is taken over for (ij)=(23), (31), (12) We replace each µ r kl with κµr kl on the right-hand side of (49), (410), and (411) Consequently, we can regard all the equations together as one system of linear homogeneous equations in 10 unknowns (M and γ ) We have 18 constraints, only eight of which are independent We see that the pair of γ = 0 and any point-based transformation whose entries are all equal to each other is always the solution of the system This solution, however, is spurious since γ 0 from its definition To put aside this spurious solution, we express M in a special form Proposition 44 Let m 11 m 12 m 13 U := 1 1 1, V := m 21 m 22 m m 31 m 32 0 (where m ij R), and put M = su + V (s R) (413) The system of linear homogeneous equations derived from (44), (45), (46) and the modified (49), (410), (411) (ie, with µ r kl replaced by κµr kl ) is expressed in the form of A f = 0 (rank A = 8; f 0), (414) where A is an 18 9 coefficient matrix 5 and f = (m 11, m 12,,m 32,γ) T R 9 Expressing M in the form of (413) decreases the number of unknowns in our system from 10 to 9 (indeed, s disappears) Since we know the rank of A, linear computation such as the QR factorization or the singular value decomposition (SVD) of A enables us to uniquely

8 122 Sugimoto determine the least-squares solution, ie, the optimal solution in the sense that it minimizes the residual sum of squares, up to a scale factor Denote by f ˆ = ( ˆm 11, ˆm 12,, ˆm 32, ˆγ) T (ˆγ = 1) the solution obtained by setting γ to be 1 (Any other criterion to eliminate the scale factor can be used; it reduces to the same results) Putting ˆV to be derived from ˆm ij,wenow express M with two parameters s and t in the following form (When t = 0, all the entries of M are equal to each other and we have γ = 0 This contradicts γ 0; therefore, t 0) M = su + t ˆV (s, t R ; t 0) (415) 44 Determining s/t From (415) it is easy to see that the scale factor eliminated by setting ˆγ = 1 is equal to t We thus have γ =ˆγt=t Noting that sgn(ρ) = sgn(λij c )(cf (42)), we express (41) in terms of s and t, from which we obtain ( s 2 ˆmki ˆm lj +ˆm kj ˆm li + sgn ( ) λij c )Ŝ(ij)(kl) t ˆm ki +ˆm lj +ˆm kj +ˆm li 2 st = 0 (416) This is a system of quadratic homogeneous equations in s and t (the number of equations is 36) We regard (416) as a system of linear homogeneous equations in independent parameters s 2, t 2 and st Solving (416) with respect to s and t is then equivalent to solving (416) with respect to s 2, t 2 and st under the decomposition constraint ( s 2 ) st det st t 2 = 0 (417) We perform two steps to obtain a solution: solve a given system ignoring the decomposition constraint and then modify the solution so that it satisfies the decomposition constraint In general, the solution we finally obtain in this way is not necessarily the least-squares solution of the original system In our case, however, we can expect that our solution is almost the same as the least-squares solution of the original system for two reasons One is that the system is highly redundant (36 constraints for only two parameters) and the other is that the decomposition constraint is unique and simple (cf (417)) Putting g = (s 2, t 2, st) T, we can then express (416) in the form of B g = 0 (rank B = 2; g 0), (418) where B isa36 3 coefficient matrix Linear computation such as the QR factorization or the SVD of B enables us to uniquely determine the solution of (418) up to a scale factor Denote by ĝ = (ŝ2, t 2, ŝt) T (ŝt = 1) the solution obtained by setting st to be 1, and define (ŝ2 ) ŝt W := ŝt t2 Letting W = ψ 1 w 1 w T 1 + ψ 2 w 2 w T 2 (ψ 1 ψ 2 0) be the spectrum resolution of W and putting W := ψ 1 w 1 w T 1, we see that W has the following property The proof is similar to that of Tsai and Huang [25] Proposition 45 Of the 2 2 symmetric matrices that satisfy the decomposition constraint (417), W is nearest from W under the Frobenius norm Hence, putting w 1 = (w 1,w 2 ) T, we see that w 1/ w 2 gives s/t By returning to (415), we can uniquely determine M up to a scale factor (The scale factor is determined by setting the criteria that the determinant of M is positive and that M is normalized under the Frobenius norm, ie, detm > 0 and M F = 1) Remark 42 Whereas we have ψ 2 = 0 in the case where no noise exists, we have ψ 2 > 0 in the presence of noise Even in the presence of noise, however, we have ψ 1 ψ 2 This indicates that the computation of w 1 is robust 5 Description of Algorithm The following algorithm determines the corresponding point-based transformation M from a given conicbased transformation S Step 2 and Step 3 are executed independently and in parallel It should be noted that the spectrum resolution of a symmetric matrix is easily and robustly computed by the singular value decomposition

9 Linear Algorithm 123 Step 0 [Computing a conic-based transformation] By using (32), compute S in the least-squares sense from given conic correspondences Step 1 Step 2 Ŝ := SD 1 [Column-based decomposition] (1): For (ij)=(23), (31), (12), compute the spectrum resolution of Mtx(u (ii) +u (jj) 2u (ij) ) Let λij c be the eigenvalue that has the maximum absolute value and let c ij be the eigenvector associated with λij c (2): Determine µ ij c according to the procedure in Subsection 41 Step 3 [Row-based decomposition] (1): For (kl) = (23), (31), (12), compute the spectrum resolution of Mtx(v (kk) +v (l l) 2v (kl) ) Let λ r kl be the eigenvalue that has the maximum absolute value and let r kl be the eigenvector associated with λ r kl (2): Determine µ r kl (cf the procedure of determining µ ij c ) [Parametrizing the point-based transforma- Step 4 tion] (1): Put κ {1, 1}, and choose a κ that minimizes the absolute value of ( [ the π(i, j)th component of µ c ij c ij (ij), i j +κµ r ij r ij]) Let κ be the solution (2): From (44), (45), (46) and the modified (49), (410), (411) (ie, with µ r kl replaced by κ µ r kl ), construct an 18 9 matrix A as in (414) (3): Compute the least-squares solution of (414) Let f ˆ= ( ˆm 11, ˆm 12,, ˆm 32, ˆγ) T (ˆγ = 1) be the solution (4): Put U : = 1 1 1, ˆm 11 ˆm 12 ˆm 13 ˆV : = ˆm 21 ˆm 22 ˆm 23 ˆm 31 ˆm 32 0 Step 5 [Determining the ratio s/t] (1): From (416), construct a 36 3 matrix B as in (418) (2): Compute the least-squares solution of (418) Let ĝ = (ŝ2, t 2, ŝt) T (ŝt = 1) be the solution (3): (ŝ2 ) ŝt W := ŝt t2 (4): Compute the spectrum resolution of W Let w 1 = (w 1,w 2 ) T be the eigenvector associated with the eigenvalue that has the maximum absolute value (5): Put ( ) w1 M := ξ U + ˆV (51) w 2 Step 6 Determine ξ so that detm > 0, M F = 1 Substitute the computed ξ into (51) to obtain M 6 Experimental Results On the basis of the algorithm above, we present two kinds of experiments: one is on noise sensitivity using numerical data The other is on the conic-based image transfer using real images In any case, our experimental results show that the outputs of the algorithm are stable and robust with respect to noise 61 Robustness Experiments Using Numerical Data We have conducted two kinds of evaluations for noise sensitivity of the proposed algorithm First, we perturbed each datum based on its magnitude Secondly, we perturbed each datum based on the variance of the distribution characterized by all the data In the first experiment, we created a 3 3 matrix M, each entry of which was randomly and independently generated within the interval of [ 5000, 5000] We used this M to compute a 6 6 matrix S(M M)We then computed S by multiplying an unknown scale to S(M M), where the scale factor was randomly generated within the interval of [ 1000, 1000] Next, we added Gaussian noise to each entry s ij of S (i, j {1, 2,,6}), independently Namely, s ij s were perturbed by independently adding Gaussian noise; we

10 124 Sugimoto Figure 4 Mean under several noise levels (Experiment 1) obtained S The mean of Gaussian distribution was set to be 00 and its standard deviation was set to be r s ij, where r represents a noise level and r was changed by 001 from 00 (noiseless) to 01 (10% noise) Here we evaluate the accuracy of each entry of a computed conic-based transformation in terms of noise levels based on the magnitude of the concerned entry By applying our algorithm to S, we estimated M For each noise level, we iterated the procedure compute S by adding noise to S and apply our algorithm to S to estimate M 50 times The mean values of estimated M over the 50 times are shown in Fig 4 We also calculated coefficients of variations, ie, the ratio of the standard deviation to the mean, which are shown in Fig 5 In the second experiment, only the way of adding noise to S was changed Perturbation of each datum was conducted based on the variance of the distribution characterized by all the data That is, each s ij was perturbed by independently adding Gaussian noise where the mean of Gaussian distribution was 00 and its standard deviation was r σ S Here, σ S is the standard deviation calculated from all the entries of S Asaresult, noise levels here indicate the ratios of noise to the size of the distribution characterized by the entries of S The mean values of estimated M over 50 times and coefficients of variations in this case are, respectively, shown in Figs 6 and 7 Figures 4 and 6 show that each entry of M is accurately estimated up to the second digit of precision under all noise levels in the both experiments Depending on the noise level, the third digit of precision changes, but the change is limited to at most 2% of each value We can thus conclude that M is accurately estimated even in the presence of noise; our algorithm is robust Figures 5 and 7, on the other hand, show that coefficients of variations (almost) monotonically increase as the noise level increases This increase is, however, compatible with the noise level This indicates that our algorithm stably estimates M We therefore numerically found that our algorithm stably performs in the presence of noise and that its outputs are reliable 62 Experiments Using Real Images We applied our algorithm to real images to see its effectiveness for the conic-based image transfer We randomly put 10 different conics on a planar board 6 and then acquired two different images of the board by an uncalibrated camera (see Fig 8) We

11 Linear Algorithm 125 Figure 5 Coefficeints of variations under several noise levels (Experiment 1) Figure 6 Means under several noise levels (Experiment 2)

12 126 Sugimoto Figure 7 Coefficients of variations under several noise levels (Experiment 2) Figure 8 Two uncalibrated images of 10 conics on a planar board: (a) image 1; (b) image 2 extracted the 10 conics from image 1 and seven conics, ie, conics 0, 1, 2, 3, 4, 5 and 7, from image 2 Here each conic was named by the number in its inside We note that seven conics in image 2 were randomly selected These results are shown in Fig 9 For each extracted conic, we randomly selected by hand about 35 points on the conic and then applied the method of least squares to the points to determine its conic vector Next, we gave the correspondences between the seven conics 0, 1, 2, 3, 4, 5 and 7 in the two images We used their conic vectors to estimate the conic-based transformation S between images 1 and 2 We then input the estimated S to our algorithm to calculate the pointbased transformation M between the two images We used this M to transfer all the conics in image 1 to those in image 2 To be more specific, for one conic in

13 Linear Algorithm 127 Figure 9 Extracted conics and lines of the two uncalibrated images: (a) conics and lines of image 1; (b) conics of image 2 (In image 2, only seven conics (0, 1, 2, 3, 4, 5, and 7) were randomly selected to determine the conic-based transformation between the two images) Figure 10 Transfered conics and lines onto image 2 (a) and the extracted conics and lines of image 2 (b) image 1, we transfered the points on the conic that were selected in determining its conic vector to the points in image 2 This transformation was conducted based on the calculated M We then applied the method of least squares to the transfered points to determine the conic vector in image 2, called the transfered conic vector The conic having the transfered conic vector was superimposed on image 2 This procedure was applied to all the conics in image 1 The results are shown in Fig 10(a) (When we directly use the estimated S to transfer the conics in image 1 to those in image 2, we obtained (almost) the same results) As the counterparts, we extracted all the conics in image 2 (Fig 10(b)) and calculated the conic vectors Comparison between the transfered conic vectors and the extracted conic vectors are shown in Table 1, where conic vectors are normalized to have the unit length In the same way, we extracted three edges of the board in image 1 (Fig 9(a)) and determined the slant and y-intercept of each extracted edge We also transfered, based on the calculated M, these edges to those in image 2 The transfered edges and extracted edges are also shown in Fig 10 Since we applied the line equation to each edge to determine its slant and y- intercept, the transfered edges have no terminal points Comparison of the slants and y-intercepts between the transfered edges and the extracted edges are shown in Table 2 Here, three lines in Fig 10 were numbered according to their locations related to the conics Namely, line 1 is the line nearest to conic 1; line 0 is the one

14 128 Sugimoto Table 1 Comparison between the transfered conic vectors and the extracted conic vectors Conic no Transfered conic vector Extracted conic vector 0 (0394, 0751, 0253, 0421, 0153, 0122) T (0394, 0753, 0251, 0420, 0154, 0123) T 1 (0595, 0668, 0243, 0274, 0238, 00994) T (0597, 0667, 0241, 0273, 0239, 00970) T 2 (0595, 0530, 0323, 0288, 0390, 0160) T (0581, 0530, 0332, 0293, 0395, 0171) T 3 (0859, 0181, 0219, 0151, 0377, 0129) T (0864, 0184, 0215, 0149, 0371, 0118) T 4 (0248, 0799, 0299, 0442, 00494, 0116) T (0257, 0796, 0298, 0440, 00487, 0121) T 5 (0601, 0366, 0488, 0248, 0452, 00186) T (0600, 0369, 0488, 0249, 0451, 00171) T 6 (0581, 0694, 0196, 0234, 0286, 00675) T (0607, 0677, 0190, 0229, 0284, 00579) T 7 (0687, 0629, 0165, 0250, 0205, 00272) T (0659, 0652, 0171, 0260, 0207, 00156) T 8 (0506, 0632, 0369, 0294, 0348, ) T (0508, 0636, 0365, 0292, 0345, ) T 9 (0636, 0449, 0428, 0311, 0333, 00522) T (0636, 0443, 0432, 0312, 0336, 00454) T Table 2 Comparison between the parameters of transfered lines and those of extracted lines Transfered Extracted Line no Slant y-intercept Slant y-intercept nearest to both conic 0 and conic 8; line 2 is the one nearest to both conic 2 and conic 9 From Fig 10 we see that all the conics and edges in image 1 are almost correctly transfered to those in image 2 In particular, three conics (ie, conics 6, 8 and 9), which were not actually used to determine the conicbased transformation, have fairly correct fittings This observation is numerically supported by Tables 1 and 2 Slightly incorrect fitting of conics and lines in Fig 10(a) is due to the numerical errors We thus may conclude that conic correspondences enable us to transfer conics and lines in one image to another, and that our algorithm performs stably for real images 7 Conclusions We clarified the relationship, based on conic correspondences, between two perspective images acquired by an uncalibrated camera Namely, we showed that the parameters representing a conic in one image are transformed by a five-dimensional projective transformation to the parameters representing the corresponding conic in another image We then established the link between a conic-based transformation and its corresponding point-based transformation, ie, the corresponding homography A conic-based transformation is essentially equivalent to the symmetric component of the tensor product of its corresponding point-based transformation and itself We developed an algorithm that uses only linear computation in uniquely determining, from a conicbased transformation, the corresponding point-based transformation up to a scale factor Our algorithm makes full use of the redundancy involved in a conicbased transformation to determine the point-based transformation with only linear computation The algorithm is simple and ensures the uniqueness of the solution Furthermore, it requires only two kinds of robust computation One is the singular value decomposition and the other is judging which is the greater of given two values In fact, our experimental results showed that the algorithm performs stably in the presence of noise and that its outputs are reliable Conic correspondences thus enable us to easily handle both points and lines in uncalibrated images of a planar object: when we observe a planar object, we can transfer one image to another image and we can establish point correspondences and line correspondences between two images Extracting points from images is sensitive with respect to noise, while conics are robustly and exactly extracted from images (see [28] for a survey of conic fitting) and, furthermore, finding correspondences between conics is much easier than that between points Therefore, we can expect that computation based on a conic-based transformation results eventually in a more stable point-based transformation than one computed from just extracted points Evaluating their differences in stability is left for future research Also left for future investigations are (1) theoretical analysis of the rounding errors in the actual implementation of the

15 Linear Algorithm 129 proposed algorithm and (2) the evaluation of the practical efficiency of the algorithm in more real situations Appendix A Algebraic Properties of Conic-Based Transformation As we see, S belongs to an eight-dimensional submanifold of all 6 6 matrices up to scale We here investigate this submanifold in detail and give the necessary and sufficient conditions on the entries of S for belonging to this submanifold A6 6 matrix up to scale has 36 1 = 35 degree of freedom and an eight-dimensional manifold has 8 degree of freedom To give the necessary and sufficient conditions on S for belonging to the submanifold constructed essentially by M, it is thus sufficient to derive 35 8 = 27 algebraically independent constraints on entries of S In Section 32 we clarified the relationship between S and M as follows: S (ij)(kl) σ(k,l)m (k(i M l) j), (35) where (ij)=(11), (22), (33), (23), (31), (12) and (kl) = (11), (22), (33), (23), (31), (12) Without loss of generality, we may handle Ŝ instead of S since Ŝ = SD 1 Ŝ (ij)(kl) = ρ M (k(i M l) j) (ρ 0) (41) = ρ M ki M lj + M kj M li 2 We pay our attention to particular cases where i = j or k = l, from which we have Ŝ (ii)(kk) = ρm 2 ki, Ŝ (ii)(kl) = ρ M ki M li, Ŝ (ij)(kk) = ρm ki M kj It is now easy to see that the entries of Ŝ satisfy the following relationship: 2Ŝ (ij)(kl) = Ŝ(ii)(kl) Ŝ (ij)(kk) Ŝ (ii)(kk) 2Ŝ (ij)(kl) = Ŝ( jj)(kl) Ŝ (ij)(kk) Ŝ (jj)(kk) + Ŝ(ii)(kl) Ŝ (ij)(ll) Ŝ (ii)(ll) (A1) + Ŝ(jj)(kl) Ŝ (ij)(ll) Ŝ ( jj)(ll) (A2) 2Ŝ (ij)(kl) = Ŝ(ii)(kl) Ŝ (ij)(kk) Ŝ (ii)(kk) 2Ŝ (ij)(kl) = Ŝ(ii)(kl) Ŝ (ij)(ll) Ŝ (ii)(ll) + Ŝ(jj)(kl) Ŝ (ij)(kk) Ŝ (jj)(kk) (A3) + Ŝ( jj)(kl) Ŝ (ij)(ll) Ŝ ( jj)(ll) (A4) When i j and k l, these equations make sense and indicate that each Ŝ (ij)(kl) is expressed in four different ways in terms of other entries of Ŝ (The equations above may be rewritten as cubic constraints on the entries of Ŝ) It is not difficult to see that any three of the four expressions are algebraically independent (From three expressions, the other one is derived) We thus have 9 3 = 27 algebraically independent (cubic) constraints on the entries of Ŝ, which are the necessary and sufficient constraints on the entries of Ŝ for belonging to the eight-dimensional submanifold that is essentially constructed by M In fact, 6 6 matrices up to scale satisfying these 27 constraints have a oneto-one correspondence to homographies The concrete description of the one-to-one mapping is given by (35) Acknowledgments The author is thankful to Kiyotake Yachi and Shohei Nobuhara for helping to perform the experiments with real images He is also grateful to anonymous referees for their helpful comments in improving this paper Notes 1 This transformation is widely known as a homography in the computer vision literature In this paper, however, we use the terminology a point-based transformation to stress the distinction from a transformation based on conic correspondences 2 We use column vectors and denote by x T the transposition of a vector x 3 M ki or M ki denotes the (k, i) entry of a matrix M 4 x denotes the Euclidean norm of a vector x 5 It is easy to see that A is a sparse matrix 6 The color of the board was coincidently similar to that of the background This unfortunately causes difficulty in observing the board References 1 F Chatelin, Valeurs propres de matrices, Masson: Paris, 1988 (W Ledermann (trans), Eigenvalues of Matrices, John-Wiley and Sons, Chichester, 1993) 2 HSM Coxeter, Projective Geometry, 2nd edn, Springer- Verlag: New York, USA, 1987

16 130 Sugimoto 3 O Faugeras and B Mourrain, On the geometry and algebra of the points and lines correspondences between N images, in Proc of the 5th ICCV, 1995, pp OD Faugeras and L Robert, What can two images tell us about a third one?, Int J of Computer Vision, Vol 18, No 1, pp 5 19, GH Golub and CF van Loan, Matrix Computation, 2nd edn, Johns Hopkins Univ Press, R Hartley, Lines and points in three views An integrated approach, in Proc of ARPA Image Understanding Workshop, 1994, pp R Hartley, A linear method for reconstruction from lines and points, in Proc of the 5th ICCV, 1995, pp R Hartley, Lines and points in three views and the trifocal tensor, Int J of Computer Vison, Vol 22, No 2, pp , R Hartley, Computation of the quadrifocal tensor, in Proc of the 5th ECCV, Vol 1, 1998, pp A Heyden, A common framework for multiple view tensors, in Proc of the 5th ECCV, Vol 1, 1998, pp A Heyden, Reduced multilinear constraints: Theory and experiments, Int J of Computer Vision, Vol 30, No 1, pp 5 26, SD Ma, Conics-based stereo, motion estimation, and pose determination, Int J of Computer Vision, Vol 10 No 1, pp 7 25, SD Ma, SH Si, and ZY Chen, Quadric curve based stereo, in Proc of the 11th ICPR, Vol 1, 1992, pp L Quan, Invariant of a pair of non-coplanar conics in space: Definition, geometric interpretation and computation, in Proc of the 5th ICCV, 1995, pp L Quan, Conic reconstruction and correspondence from two views, IEEE Trans on PAMI, Vol 18, No 2, pp , C Schmid and A Zisserman, The geometry and matching of curves in multiple views, in Proc of the 5th ECCV, Vol 1, 1998, pp JA Schouten, Tensor Analysis for Physicists, 2nd edn, Dover: New York, USA, JG Semple and GT Kneebone, Algebraic Projective Geometry, Clarendon Press: Oxford, UK, 1952 (reprinted in 1979) 19 A Shashua, Trilinearity in visual recognition by alignment, in Proc of the 3rd ECCV, Vol 1, 1994, pp A Shashua, Algebraic functions for recognition, IEEE Trans on PAMI, Vol 17, No 8, pp , A Shashua and M Werman, Trilinearity of three perspective views and its associated tensor, in Proc of the 5th ICCV, 1995, pp A Sugimoto, Conic based image transfer for 2-D objects: A linear algorithm, in Proc of the 3rd ACCV, Vol II, 1998, pp A Sugimoto and T Matsuyama, Multilinear relationships between the coordinates of corresponding image conics, in Proc of the 15th ICPR, B Triggs, Matching constraints and the joint image, in Proc of the 5th ICCV, 1995, pp RY Tsai and TS Huang, Uniqueness and estimation of three dimensional motions parameters of rigid objects with curved surfaces, IEEE Trans on PAMI, Vol 6, No 1, pp 13 27, I Weiss, 3D curve reconstruction from uncalibrated cameras, Technical Report, CSTR-TR-3605, Computer Vision Laboratory, Center for Automation Research, University of Maryland, I Weiss, 3-D curve reconstruction from uncalibrated cameras, in Proc of ICPR, Vol 1, 1996, pp Z Zhang, Parameter estimation techniques: A tutorial with application to conic fitting, Image and Vision Computing, Vol 15, No 1, pp 59 76, 1997 Akihiro Sugimoto received his BS degree, MS degree and Dr Eng in mathematical engineering from the University of Tokyo, Japan in 1987, 1989 and 1996, respectively He joined Hitachi Advanced Research Laboratory in 1989 and moved to ATR (Advanced Telecommunications Research Institute), Japan in 1991, where he worked for Auditory and Visual Perception Labs and for Human Information Processing Labs, and he got into the research field of computer vision In 1995, he returned to Hitachi Advanced Research Laboratory where he lead a project on content-based image retrieval supported by Ministry of International Trade and Industry in Japan Since 1999, he has worked for Kyoto University, where he is currently a lecturer at the graduate school of Informatics Dr Sugimoto s research interest is in mathematical methods in/for engineering In particular, his current main research interests are discrete mathematics, approximation algorithm, vision geometry and modeling of human vision