Trinocular Geometry Revisited

Transcription

1 Trinocular Geometry Revisited Jean Pounce and Martin Hebert 报告人 : 王浩人

2 Contents 1. Introduction 2. Converging Triplets of Lines 3. Converging Triplets of Visual Rays 4. Discussion

3 1. Introduction 1.1 What is the problem here? 1.3 What is the goal of this paper? 1.2 What is wrong with the classical answers? 1.4 Proposed Approach

4 1.1 What is the problem here? When do the visual rays associated with triplets of point correspondences converge, that is, intersect in a common point? (theoretically)

5 1. Introduction 1.1 What is the problem here? 1.3 What is the goal of this paper? 1.2 What is wrong with the classical answers? 1.4 Proposed Approach

6 1.2 What is wrong with the classical answers? Classical answers: Two views: fundamental matrix Bilinear epipolar constraint

7 1.2 What is wrong with the classical answers? Classical answers: Three views: trifocal tensor Epipolar constraint only? Almost always converge. ONLY one exception Left: Visual rays associated with three (correct) correspondences. Middle: Degenerate epipolar constraints associated with three coplanar, but non-intersecting rays lying in the trifocal plane τ. Right: Collinear case of middle.

8 1.2 What is wrong with the classical answers? Classical answers: Three views: trifocal tensor Interestingly, Hartley and Zisserman state that the fundamental matrices associated with three cameras with non-collinear pinholes determine the corresponding trifocal tensor [6, Result 15.4]. While Faugeras and Mourrain [3] and Ponce et al. [12], for example, note that the rays associated with three points only satisfying certain (and different) subsets of the trilinearities alone must intersect. All 4 4 minors of some k 4 matrix are zero will guarantee that the three lines intersect. [6] R. Hartley and A. Zisserman. Multiple view geometry in computer vision. Cambridge University Press, [3] O. Faugeras and B. Mourrain. On the geometry and algebra of the point and line correspondences between n images. Technical Report 2665, INRIA, [12] J. Ponce, K. McHenry, T. Papadopoulo, M. Teillaud, and B. Triggs. The absolute quadratic complex and its application to camera self calibration. In CVPR, 2005.

9 1.2 What is wrong with the classical answers? What is wrong? Rays satisfy epipolar constraints do not always converge, but they are true under some general configuration assumptions, rarely made explicit. We are not aware of any fixed set of four trilinearities that, alone, guarantee convergence in all cases.

10 1. Introduction 1.1 What is the problem here? 1.3 What is the goal of this paper? 1.2 What is wrong with the classical answers? 1.4 Proposed Approach

11 1.3 What is the goal of this paper? To clarify the general configuration assumption. To understand exactly how much the trifocal constraints add to the epipolar ones for point correspondences. Necessary and sufficient conditions

12 1. Introduction 1.1 What is the problem here? 1.3 What is the goal of this paper? 1.2 What is wrong with the classical answers? 1.4 Proposed Approach

13 1.4 Proposed Approach Approach: transversal

14 Contents 1. Introduction 2. Converging Triplets of Lines 3. Converging Triplets of Visual Rays 4. Discussion

15 2. Converging Triplets of Lines 2.1 Geometrical Point of View 2.2 Analytical Point of View

16 2.1 Geometrical Point of View Proposition 1. A necessary and sufficient condition for three lines to converge is that they be pairwise coplanar, and that they admit a transversal not contained in the planes defined by any two of them. Lemma 1 Lemma 2 Proposition 1

17 2.1 Geometrical Point of View Lemma 1. Three distinct lines can be found in exactly six configurations : (1) the three lines are not all coplanar and intersect in exactly one point; (2) they are coplanar and intersect in exactly one point; (3) they are coplanar and intersect pairwise in three different points;

18 2.1 Geometrical Point of View Lemma 1. Three distinct lines can be found in exactly six configurations : (4) exactly two pairs of them are coplanar (or, equivalently, intersect); (5) exactly two of them are coplanar; or (6) they are pairwise skew.

19 2.1 Geometrical Point of View

20 2.1 Geometrical Point of View Regulus : a line field, formed by all lines in a plane; a line bundle, formed by all lines passing through some point; the union of all lines belonging to two flat pencils lying in different planes but sharing one line; or a non-degenerate regulus formed by one of the two sets of lines ruling a hyperboloid of one sheet or a hyperbolic paraboloid

21 2.1 Geometrical Point of View Lemma 2. Three distinct lines always admit an infinity of transversals, that can be found in exactly six configurations: (1) the transversals form a bundle of lines; (2) they form a degenerate congruence consisting of a line field and of a bundle of lines; (3) they form a line field;

22 2.1 Geometrical Point of View Lemma 2. Three distinct lines always admit an infinity of transversals, that can be found in exactly six configurations: (4) they form two pencils of lines having one of the input lines in common; (5) they form two pencils of lines having a line passing through the intersection of two of the input lines in common; or (6) they form a non-degenerate regulus, with the three input lines in the same ruling, and the transversals in the other one.

23 2.1 Geometrical Point of View Proposition 1. A necessary and sufficient condition for three lines to converge is that they be pairwise coplanar, and that they admit a transversal not contained in the planes defined by any two of them.

24 2. Converging Triplets of Lines 2.1 Geometrical Point of View 2.2 Analytical Point of View

25 2.2 Analytical Point of View Preliminaries: Plucker coordinate: any line in Ρ 3 with its Plucker coordinate vector ξ = (u; v) in R 6, where u and v are vectors of R 3, and we have u = x 4 y 1 x 1 y 4 x 4 y 2 x 2 y 4 x 2 y 3 x 3 y 2 x 3 y 1 x 1 y 3 x 4 y 3 x 3 y 4, and v = x 1 y 2 x 2 y 1. (1) Bilinear product: two elements λ =(a; b) and µ = (c; d) of R 6, then the scalar (λ µ) = a d + b c. ξ in R 6 to represent a line is that (ξ ξ) = 0; two lines λ and µ to be coplanar (or, equivalently, to intersect) is that (λ µ) = 0.

26 2.2 Analytical Point of View Preliminaries: Join operator: The unique line joining two distinct points is called the join of these points and it is denoted by x y. The unique plane defined by a line ξ = (u; v) and some point x not lying on this line is called the join of ξ and x, and it is denoted by ξ x = ξ x, where ξ is the join matrix defined by ξ = [ u v v T 0 ] (2) A necessary and sufficient condition for a point x to lie on a line ξ is thatξ x = 0.

27 2.2 Analytical Point of View Preliminaries: Fundamental points (basis points) of some arbitrary projective coordinate system: x 0 = (0, 0, 0, 1) T ; x 1 = (1, 0, 0, 0) T ; x 2 = (0, 1, 0, 0) T ; x 3 = (0, 0, 1, 0) T ; x 4 = (1, 1, 1, 1) T. Fundamental planes : p j = x j, (j = 0, 1, 2, 3) (unit point)

28 2.2 Analytical Point of View Let us consider three distinct lines ξ j = (ξ 1j,, ξ 6j ) T, (j = 1, 2, 3) and define to be the 3 3 minor of the 6 3 matrix ξ 1, ξ 2, ξ 3 corresponding to its rows i, j, and k. A necessary and sufficient condition for this matrix to have rank 2, and thus for the three lines to form a flat pencil, is that all the minors T 0 = D 456, T 1 = D 234, T 2 = D 135, and T 3 = D 126 be equal to zero. [12] Appendix

29 2.2 Analytical Point of View Lemma 3. Given some integer j in {0, 1, 2, 3}, a necessary and sufficient condition for ξ 1, ξ 2, and ξ 3 to admit a transversal passing through x j is that T j = 0. To prove j = 0

30 2.2 Analytical Point of View A necessary and sufficient condition for a line δ = (u; v) to pass through x 0 is that v = 0 (this follows from the form of the join matrix). Thus a necessary and sufficient condition for the existence of a line δ passing through x 0 and intersecting the lines ξ j = (u j ; v j ) is that here exists a vector u 0 such that ξ j δ = v j u = 0 for j = 1, 2, 3, or, equivalently, that the determinant T 0 = D 456 = v 1, v 2, v 3 = 0.

31 2.2 Analytical Point of View Proposition 1 Lemma 3 Proposition 2

32 2.2 Analytical Point of View Proposition 2. A necessary and sufficient condition for three lines ξ 1, ξ 2, and ξ 3 to converge is that ξ i ξ j = 0 for all i j in {1, 2, 3}, and that T j = 0 for all j in {0, 1, 2, 3}.

33 Contents 1. Introduction 2. Converging Triplets of Lines 3. Converging Triplets of Visual Rays 4. Discussion

34 3. Converging Triplets of Visual Rays Exception: when the corresponding rays lie in the trifocal plane Non-collinear collinear

35 3. Converging Triplets of Visual Rays 3.1 Bilinearities and Trilinearities Non-Collinear Pinholes Collinear Pinholes 3.2 Minimal Parameterizations Non-Collinear Pinholes Collinear Pinholes 3.3 Preliminary Implementation

36 3.1 Bilinearities and Trilinearities Proposition 3. Given three cameras with non-collinear pinholes c 1, c 2 and c 3, and any projective coordinate system such that x 0 does not belong to the trifocal plane, a necessary and sufficient condition for the three rays ξ j = c j y j, j = 1, 2, 3 to converge is that is that ξ i ξ j = 0 for all i j in {1, 2, 3}, and T 0 = 0. Non-collinear

37 3. Converging Triplets of Visual Rays 3.1 Bilinearities and Trilinearities Non-Collinear Pinholes Collinear Pinholes 3.2 Minimal Parameterizations Non-Collinear Pinholes Collinear Pinholes 3.3 Preliminary Implementation

38 3.1 Bilinearities and Trilinearities Proposition 4. Given three cameras with collinear pinholes c 1, c 2 and c 3, and any projective coordinate system such that x 0 and x j and the baseline β joining the pinholes are not coplanar, a necessary and sufficient condition for the three rays ξ j = c j y j, j = 1, 2, 3 to converge is that is that ξ i ξ j = 0 for all i j in {1, 2, 3}, and T 0 = T j = 0 for some j 0. For collinear pinholes, there exists a single scene plane π 0 in the pencil passing through the baseline β that contains x 0 and for which the condition T 0 = 0 is ambiguous.

39 3. Converging Triplets of Visual Rays 3.1 Bilinearities and Trilinearities Non-Collinear Pinholes Collinear Pinholes 3.2 Minimal Parameterizations Non-Collinear Pinholes Collinear Pinholes 3.3 Preliminary Implementation

40 3.2 Minimal Parameterizations In this case, we can always choose a projective coordinate system such that the three fundamental points distinct from x 0 are the three camera centers that is, c j = x j for j = 1, 2, 3, and x 0 does not lie in the trifocal plane. y j = (y 1j, y 2j, y 3j, y 4j ) T p 0 is the trifocal plane (p j x j = 0, for j = 1, 2, 3) Non-collinear

41 3.2 Minimal Parameterizations Epipolar constrains can be written as: x 1 y 1 x 2 y 2 = 0 x 1 y 1 x 3 y 3 = 0 x 2 y 2 x 3 y 3 = 0 y 41 y 32 = y 31 y 42 y 41 y 23 = y 21 y 43 y 42 y 13 = y 12 y 43. (4) From proposition 2, T 0 = 0 (the other minors are trivially zero with our choice of coordinate system.) is easily written as y 21 y 32 y 13 = y 31 y 12 y 23. (5)

42 3.2 Minimal Parameterizations Π j (j = 1, 2, 3): the 4 3 matrix formed by the coordinate vectors of the basis points for the retinal plane of camera number j. The position of an image point with coordinate vector u j in that basis is thus π ij T : the i-th row of matrix Π j ; π ij k : the k-th coordinate of π ij ; y j = Π j u j.

43 Proposition 5. Given three cameras with non-collinear pinholes and hypothetical point correspondences u 1, u 2 and u 3, a necessary and sufficient condition for the three corresponding rays to converge is that and u 1 T F 12 u 2 = 0 u 1 T F 13 u 3 = 0 u 2 T F 23 u 3 = 0 where F 12 = π 41 π T T 32 π 31 π 42 F 13 = π 41 π T T 23 π 21 π 43 F 23 = π 42 π T T 13 π 12 π 43, (6) (π 21 u 1 )(π 32 u 2 )(π 13 u 3 ) = (π 31 u 1 )(π 12 u 1 )(π 23 u 3 ), (7) Where the vectors π 1 = (π 21 ; π 31 ; π 41 ), π 2 = (π 12 ; π 32 ; π 42 ), and π 3 = (π 13 ; π 23 ; π 43 ), satisfy the 6 homogeneous constraints π 1 21 = 0, π 2 32 = 0, π 3 13 = 0, 2 = π 3 41, π 3 12 = π 1 42, π 1 23 = π 43 π 31 and are thus defined by three groups of 7 coefficients, each one uniquely determined up to a separate scale. This is a minimal, 18 dof parameterization of trinocular geometry. 2, (8)

44 3.2 Minimal Parameterizations Eq. (4) == Eq. (6), Eq. (5) == Eq. (7) Minimal degree of freedom π j = π 1j ; π 2j ; π 3j (j = 1, 2, 3) provide 24dof together, up to scale; Locating the camera pinholes at the fundamental points x j (j = 1,2, 3) freezes 9 of the 15 degrees of freedom of the projective transformation; 24 (15 9) = 18 The general form of a projective transform Q mapping x j onto themselves has 7 coefficients defined up to scale. And writing that the matrices QΠ j must satisfy the constraints of Eq. (8) yields a system of 6 homogeneous equations in the 7 nonzero entries of Q. (different sets of entries of π j )

45 3.2 Minimal Parameterizations Eq. (7) has an interesting geometric interpretation: (?) Any point with coordinate vector u 1 in the first image that matches points with coordinate vectors u 2 and u 3 in the other two, must satisfy Eq. (7) and thus belong to the trinocular line (our terminology):

46 3. Converging Triplets of Visual Rays 3.1 Bilinearities and Trilinearities Non-Collinear Pinholes Collinear Pinholes 3.2 Minimal Parameterizations Non-Collinear Pinholes Collinear Pinholes 3.3 Preliminary Implementation

47 3.2 Minimal Parameterizations Assume the three pinholes are collinear. Let us position c 1 = x 1, c 2 = x 2, and c 3 = x 1 + x 2. From Eq.(4): y 41 y 32 = y 31 y 42, y 41 y 33 = y 31 y 43, y 42 y 33 = y 32 y 43, (10) and write T 0 = 0 and T 3 = 0 respectively as y 31 y 32 y 23 y 13 + y 33 y 31 y 12 y 21 y 32 = 0, y 41 y 42 y 23 y 13 + y 43 y 41 y 12 y 21 y 42 = 0. the other two minors T 1 and T 2 are zero with our choice of coordinate system. (11) ω 3 = π 23 π 13

48 Proposition 6. Given three cameras with collinear pinholes and hypothetical point correspondences u 1, u 2 and u 3, a necessary and sufficient condition for the three corresponding rays to converge is that u 1 T F 12 u 2 = 0 u 1 T F 13 u 3 = 0 u 2 T F 23 u 3 = 0 where F 12 = π 41 π T T 32 π 31 π 42 F 13 = π 41 π T T 33 π 31 π 43 F 23 = π 42 π T T 33 π 32 π 43, (12) And 0 0 ( ( u 1)( u 1)( u 2)( 3 u 2)( 3 u 3) ( u 3) ( u 3)[( u 3)[( u 1)( u 1)( u 2) ( u 2) ( u 1)( u 1)( u 2)], u 2)] (13) Where the vectors π 1 = (π 21 ; π 31 ; π 41 ), π 2 = (π 12 ; π 32 ; π 42 ), and π 3 = (ω 13 ; π 23 ; π 43 ), satisfy the 8 homogeneous constraints π 1 21 = 0, π 2 31 = 0, π 1 12 = 0, π 2 42 = 0 3 = π 3 21, π 3 32 = π 3 42, ω 1 3 = ω 2 3 = ω 3 3, π 31 (14) and are thus defined by three groups of, respectively, 6, 6, and 7 independent coefficients, each uniquely determined up to a separate scale, for a total of 16 independent parameters. This is a minimal, 16 dof trinocular parameterization.

49 3.2 Minimal Parameterizations Eq. (12) == Eq. (10), Eq. (13) == Eq. (11) Locating the camera pinholes in position c 1 = x 1, c 2 = x 2, and c 3 = x 1 + x 2 freezes 7 of the 15 degrees of freedom of the projective ambiguity of projective structure from motion. 24 (15 7) = 16

50 3. Converging Triplets of Visual Rays 3.1 Bilinearities and Trilinearities Non-Collinear Pinholes Collinear Pinholes 3.2 Minimal Parameterizations Non-Collinear Pinholes Collinear Pinholes 3.3 Preliminary Implementation

51 3.3 Preliminary Implementation Proposition 5 can be used to estimate the vectors π j from at least six correspondences between three images: Initial values for these vectors are easily obtained using an affine or projective model; The vectors π j are then refined by minimizing the mean-squared distance between all data points and the corresponding epipolar and trinocular lines.

52 An example with 38 correspondences between three images, Top: trinocular lines recovered from correspondences in three images; Bottom: Estimated epipolar lines (two sets per image). Note that the two families of epipolar lines associated with an image typically contain (near) degenerate pairs that can be disambiguated using trilinearities.

53 Quantitative results for the dataset and affine [18] and projective [2] initializations. Here, E ij refers to the distance between points in image i and the corresponding epipolar lines associated with image j, and T j refers to the distance between points in image j and the corresponding trinocular line associated with the other two images. [2] S. Carlsson. Duality of reconstruction and positioning in multiple views. In Proc. Worshop on Representation of Visual Scenes, [18] C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: a factorization method. IJCV, 9(2): , 1992.

54 Contents 1. Introduction 2. Converging Triplets of Lines 3. Converging Triplets of Visual Rays 4. Discussion

55 4. Discussion Our method is by construction robust to degeneracies with points lying near the trifocal plane. Parameterization is symmetric, none of the cameras playing a priviledged role. Although the nature of our presentation has been mainly theoretical, there are some contributions.

56 Contributions A new geometric characterization of triplets of converging lines in terms of transversals to these lines (Proposition 1). A novel and simple analytical characterization of triplets of converging lines (Lemma 3 and Proposition 2), that does not rely on the assumptions of general configuration implicit in [12]. Applying these results to camera geometry, the three epipolar constraints and one of the trifocal ones (two if the pinholes are collinear) are necessary and sufficient for the corresponding optical rays to converge (Propositions 3 and 4). A new analytical parameterization of epipolar and trifocal constraints, leading to a minimal parameterization of trinocular geometry (Propositions 5 and 6).

57 Thank you Q & A