Multiple View Geometry in Computer Vision Prasanna Sahoo Department of Mathematics University of Louisville 1
Trifocal Tensor Lecture 21 March 31, 2005 2
Lord Shiva is depicted as having three eyes. The third eye in the center of the forehead symbolizes spiritual knowledge and power. 3
Trifocal Tensor Trifocal tensor plays a similar role in three views just like the fundamental matrix for the two views. Trifocal tensor captures all the geometric relations between three views that are independent of scene structure. 4
The camera matrices can be recovered from the trifocal tensor up to a common projective transformation of 3-space. The fundamental matrices for the view-pairs can also be retrieved uniquely from the trifocal tensor. 5
Incidence relations for lines Suppose a line L in 3-space is imaged in three views. We would like to know what are the constraints on the corresponding image lines. 6
If lines in each views are back-projected then we have three planes. These three planes intersect at the line L in 3-space. 7
This provides a constraint on the set of three image lines since in general three arbitrary planes do not intersect in a single line. Next we translate this geometric constraint into an algebraic constraint on the three lines. 8
l l l will denote the set of corresponding lines. Let P = [ I 0 ], P = [ A a 4 ] and P = [ B b 4 ] denote camera matrices for the three views, where A and B are 3 3 matrices, and a i and b i are the i th columns of the respective camera matrices. 9
Suppose the camera matrix of the first view is located at the world origin, that is C = (0, 0, 0, 1) T. Then e = P C = [ A a 4 ](0, 0, 0, 1) T = a 4. This means that a 4 is the projection of the center of the first camera as seen from the second it is the epipole of the first and second view. Similarly, e = P C = [ B b 4 ](0, 0, 0, 1) T = b 4, and b 4 is the epipole of the first and third views. 10
In the camera matrices P = [ I 0 ], P = [ A a 4 ] and P = [ B b 4 ], the 3 3 matrices A and B are the infinite homographies from the first to second and first to third cameras, respectively. That is A = K R K 1 and B = K R K 1, where K, K, K are the calibration matrices and R, R are the rotation matrices. 11
Each image line back-projects to a plane. If P = [ I 0 ], P = [ A a 4 ] and P = [ B b 4 ] are the camera matrices, then the three planes are given by 12
π = P T l = [ I 0 ] T l = l 0, π = P T l = [ A a 4 ] T l = A T l a T 4 l, π = P T l = [ B b 4 ] T l = B T l b T 4 l. 13
Since the three image lines are derived from a single line in 3-space, it follows that these three planes are not independent and must meet in this common line in 3-space. Algebraically this can be expressed by the requirement that the matrix M given by M = [ π, π, π ] has rank 2. 14
Since M is of rank 2, there is a linear dependence between its columns m i. Denoting M = [ m 1, m 2, m 3 ] = l A T l B T l 0 a T 4 l b T 4 l the linear relation may be written as m 1 = α m 2 + β m 3. 15
Since the bottom left hand element of M = l A T l B T l 0 a T 4 l b T 4 l is 0, it follows that α = k b T 4 l and β = k a T 4 l where k is some scalar. 16
Applying α = k b T 4 l and β = k a T 4 l to the top 3-vectors of M = l A T l B T l 0 a T 4 l b T 4 l of each column, we get l = ( b T 4 l ) A T l ( a T 4 l ) B T l (up to scale). 17
Since inner product is commutative, we can write b T 4 l = l T b 4 and a T 4 l = l T a 4. Hence l = ( b T 4 l ) A T l ( a T 4 l ) B T l = ( l T b 4 ) A T l ( l T a 4 ) B T l = l T ( b 4 A T ) l l T ( a 4 B T ) l If we denote by l i the i th coordinate of l, then from the above we have l i = l T ( b 4 a T i ) l l T ( a 4 b T i ) l. ( ) 18
Rewriting ( ), we obtain l i = l T ( b 4 a T i ) l l T ( a 4 b T i ) l = l T ( b 4 a T i a 4 b T i = l T T i l ) l where T i = b 4 a T i a 4 b T i is a 3 3 matrix for each i = 1, 2, 3. 19
Definition: The set of three matrices { T 1, T 2, T 3, } constitutes the trifocal tensor in matrix notation. 20
Let us denote the ensemble of the three matrices T i by [ T 1, T 2, T 3 ] or more briefly as [ T i ]. Then the relation l i = l T T i l can be written as l T = (l 1, l 2, l 3 ) = (l T T 1 l, l T T 2 l, l T T 3 l ) =: l T [ T 1, T 2, T 3 ] l =: l T [ T i ] l. 21
Similarly, we have l T = l T [ T i ] l and l T = l T [ T i ] l. 22
Remark: The three tensors [ T i ], [ T i are distinct. Note that ], [ T i ] exist and they l T = l T [ T i ] l l T = l T [ T i ] l l T = l T [ T i ] l are relationships between image coordinates only. 23
They do not involve coordinates of the 3D scene points X i. The above formulas are valid for the camera matrices in canonical form only, that is P = [ I 0 ], P = [ A a 4 ] and P = [ B b 4 ]. 24
Degrees of Freedom of Trifocal Tensor The trifocal tensor consists of three 3 3 matrices. So it has 27 elements. There are 26 elements apart from the common scaling of the matrices. However, the trifocal tensor has only 18 independent degrees of freedom. 25
This can be seen as follows: 3 camera matrices has 33 in total; 15 degrees of freedom must be subtracted to account for the projective world frame. Hence 33 15 = 18 is the degrees of freedom for the trifocal tensor. 26
Homographies induced by a plane A fundamental geometric property of the trifocal tensor is the homography between the first view and the third induced by a line in the second view. 27
A line is the second view back-projects to a plane in 3- space, and this plane induces a homography between the first and the third view. 28
The homography induced by the plane π between the first and third views is given by x = H x 29
and l = H T l, respectively. l l l is the set of corresponding lines. 30
Suppose we are given the following line correspondence l l l. These lines satisfy l i = l T T i l ( ) where l i is the i th component of the vector l. Because of the homography induced by the plane π between the first and third views we have l = H T l. ( ) 31
Comparing ( ) and ( ), we see that H = [ h 1, h 2, h 3 ] with h i = T T i l. Thus H defined by the above formula represents the (point) homography H 13 between views one and three specified by the line l on view two. 32
Result 1. The homography from the first to the third image induced by a line l in the second image is given by x = H 13 (l ) x, where H 13 (l ) = [ T T 1, TT 2, TT 3 ] l. Similarly, a line l in the third image defines a homography x = H 12 (l ) x from the first to the second views, given by H 12 (l ) = [ T 1, T 3, T 3 ] l. 33
Point and line incidence relations line-line-line correspondence point-line-line correspondence Incidence Relations point-line-point correspondence 3 point-point-line correspondence 7 point-point-point correspondence 34
Next we deduce various linear relationships between lines and points in three images involving the trifocal tensor. Recall that the incidence relation for l l l correspondence is l T = l T [ T 1, T 2, T 3 ] l where l T [ T 1, T 2, T 3 ] l means the 3-vector ( l T T 1 l, l T T 2 l, l T T 3 l ). 35
Let x be a point on the line l. Then x T l = 0. Let us denote x = (x 1, x 2, x 3 ) and l = (l 1, l 2, l 3 ). Hence 0 = x T l = i x i l i = i x i ( l T T i l ) = l T i x i T i l. 36
l T i x i T i l = 0 is the incidence relation in the first image and it holds for point-line-line correspondence. 37
(a) point-line-line 38
From Result 1, we obtain a homography H 13 (l ) between first and third images. Hence x = H 13 (l ) x = [ T T 1 l, T T 2 l, T T 3 l ] x = ( i x i T T i ) l. This is valid for any line l passing through x (see figure (b)). 39
(b) point-line-point 40
We take the transpose of the both sides of x = ( i x i T T i ) l and multiply the resulting expression by [ x ] to get x T [ x ] = l T ( i x i T i ) [ x ] = 0 T. This is the incidence relation and it holds for pointline-point correspondence. 41
Similarly from Result 1, we obtain a homography H 12 (l ) between first and second images. Hence x = H 12 (l ) x = [ T 1 l, T 2 l, T 3 l ] x = ( i x i T i ) l. This is valid for any line l passing through x. 42
(b) point-line-point 43
Multiplying the both sides of x = ( i x i T i ) l by [ x ] to get [ x ] x = [ x ] ( i x i T i ) l = 0. This is the incidence relation and it holds for pointpoint-line correspondence. 44
(c) point-point-point Next we will show that for a three-point correspondence the following relation holds: [ x ] ( i x i T i ) [ x ] = 0 3 3. 45
The line l in l T ( i x i T i ) [ x ] = 0 T passes through x. So we have l = x y = [ x ] y for some point y on l. Therefore y T [ x ] ( i x i T i ) [ x ] = 0 T 46
The relation l T ( i x i T i ) [ x ] = 0 T is true for all lines l through x. Therefore y T [ x ] ( i x i T i ) [ x ] = 0 T is independent of y. Hence we get [ x ] ( i x i T i ) [ x ] = 0 3 3. 47
END 48