Basics of Epipolar Geometry

Transcription

1 Basics of Epipolar Geometry Hellmuth Stachel Seminar at the Institute of Mathematics and Physics Faculty of Mechanical Engineering, May 26,, STU Bratislava

2 Table of contents 1. Linear images of the 3-space 2. Geometry of two images (epipolar geometry) 3. Numerical reconstruction from two images May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 1/55

3 1. Linear images of the 3-space The central projection produces a linear image = central perspective... according to A. Dürer s woodcarving (-25) (in Underweysung der Messung mit dem Zirckel und Richtscheyt ) May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 3/55

4 1. Linear images of the 3-space A panorama is a nonlinear image but can be generated from a perspective by a planar transformation according to the unfolding of a right cylinder. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 4/55

5 1. Linear images of the 3-space H h H h (x,y) (x p,y p ) with x p = rarctan x d, yp = ry d2 +x 2 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 5/55

6 Central projection The central projection (according to A. Dürer) can be generalized by a central axonometry. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 6/55

7 Central axonometric principle in space E 3 : U c 3 in the image plane E 2 : U 3 E 3 O E 1 E 2 U 1 U 2 cartesian basis O;E 1,E 2,E 3 and points at infinity U 1,U 2,U 3 U c 1 E c 1 E c 3 O c E c 2 U c 2 Given: central axonometric reference system O c ;E c 1,E c 2,E c 3;U c 1,U c 2,U c 3 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 7/55

8 Central axonometric principle U c 3 Theorem (Szabó, H.S., Vogel 94): A central axonometric reference system defines a central projection α 3 ( e1 f 1 ) 2 : ( e2 f 2 ) 2 : ( e3 f 3 ) 2 = = tanα 1 : tanα 2 : tanα 3. Theorem: Each central axonometry is affine to a central projection. E c 3 α U1 c 1 α 2 U2 c f1 e1 E c 1 E c 2 O c e 2 f 2 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 8/55

9 Photo versus linear image Photo (=central perspective) or photo of a photo (=linear image)? May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 9/55

10 Definition of a linear image There is a unique collinear transformation κ: E 3 E 2 mit O O c, E i E c i, U i U c i, i = 1,2,3. Any two-dimensional image of E 3 under a collinear transformation is called linear. = { collinear points have collinear or coincident images cross-ratios of any four collinear points are preserved. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

11 Definition of a linear image There is a unique collinear transformation κ: E 3 E 2 mit O O c, E i E c i, U i U c i, i = 1,2,3. Any two-dimensional image of E 3 under a collinear transformation is called linear. = { collinear points have collinear or coincident images cross-ratios of any four collinear points are preserved. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

12 Central projection in coordinates Notation: Z... center H... principal point vanishing plane Π v x 2 Π image plane x 2 d... focal length x 1,x 2,x 3... camera frame Z x 3 H X c X x 1,x 2... image coordinate frame x 1 d x 1 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

13 Central projection in coordinates ( x 1 x 2 ) = d ( ) x1, or homogeneous x 3 x 2 ξ 0 ξ 1 ξ 2 = d d 0 ξ 0. ξ 3. Transformation from the camera frame (x 1,x 2,x 3 ) into arbitrary world coordinates (x 1, x 2, x 3 ) and translation from the particular image frame (x 1,x 2) into arbitrary (x 1,x 2) gives in homogeneous form ξ 0 ξ 1 ξ 2 = h 1 df 1 0 h 2 0 df }{{} matrix A o 1. o 3 R ξ 0. ξ 3. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

14 Central projection in coordinates ( x 1 x 2 ) = d ( ) x1, or homogeneous x 3 x 2 ξ 0 ξ 1 ξ 2 = d d 0 ξ 0. ξ 3. Transformation from the camera frame (x 1,x 2,x 3 ) into arbitrary world coordinates (x 1, x 2, x 3 ) and translation from the particular image frame (x 1,x 2) into arbitrary (x 1,x 2) gives in homogeneous form ξ 0 ξ 1 ξ 2 = h 1 df 1 0 h 2 0 df }{{} matrix A o 1. o 3 R ξ 0. ξ 3. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

15 Central projection in coordinates Left hand matrix: (h 1,h 2) are image coordinates of the principal point H, (f 1,f 2 ) are possible scaling factors, and d is the focal length. These parameters are called the intrinsic calibration parameters. Right hand matrix: R is an orthogonal matrix. The position of the camera frame with respect to the world coordinates defines the extrinsic calibration parameters. Photos with known interior calibration parameters are called calibrated images, others (like central axonometries) are uncalibrated. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

16 Central projection in coordinates Left hand matrix: (h 1,h 2) are image coordinates of the principal point H, (f 1,f 2 ) are possible scaling factors, and d is the focal length. These parameters are called the intrinsic calibration parameters. Right hand matrix: R is an orthogonal matrix. The position of the camera frame with respect to the world coordinates defines the extrinsic calibration parameters. Photos with known interior calibration parameters are called calibrated images, others (like central axonometries) are uncalibrated. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

17 How to check the interior calibration parameters of any camera? The center Z lies at the intersection of the three spheres drawn over the segments U c 1U c 2,... U 3 c H=Z Note: Zooming and (automatic) focussing changes the focal distance d! U 1 c U 2 c May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

18 DG negative plane Π vanishing plane Πv x2 Π image plane x1 H Z DG DG Positive and negative central pespective x3 H X d d May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55 x 2 X c x 2 X c x 1 x 1

19 unknown interior calibration parameters collinear bundle transformation Z Z the bundles Z and Z of the rays of sight are collinear (neither affine nor similar!) May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

20 2. Geometry of two images Given: Two linear images or two photographs showing the same object. Wanted: Dimensions of the depicted 3D-object. Historical Stadtbahn station Karlsplatz in Vienna (Otto Wagner, 97) May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

21 2. Geometry of two images The geometry of two images is a classical subject of Descriptive Geometry. Its results have become standard (Finsterwalder, Kruppa, Krames, Wunderlich, Hohenberg, Tschupik, Brauner, Havlicek, H.S.,...). Photogrammetry (Remote sensing) deals with the practical usage of these results. Why now? Advantages of digital images: less distorsion, because no paper prints are needed, exact boundary is available, and precise coordinate measurements are possible using standard software. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

22 2. Geometry of two images The geometry of two images is a classical subject of Descriptive Geometry. Its results have become standard (Finsterwalder, Kruppa, Krames, Wunderlich, Hohenberg, Tschupik, Brauner, Havlicek, H.S.,...). Photogrammetry (Remote sensing) deals with the practical usage of these results. Why now? Advantages of digital images: less distorsion, because no paper prints are needed, exact boundary is available, and precise coordinate measurements are possible using standard software. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

23 Computer Vision Why now? The geometry of two images is important for Computer Vision, a topic with the main goal to endow a computer with a sense of vision. Basic problems: Which information can be extracted from digital images? How to preprocess and represent this information? Sensor-guided robots, automatic vehicle control, Big Brother,... May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

24 Computer Vision Why now? The geometry of two images is important for Computer Vision, a topic with the main goal to endow a computer with a sense of vision. Basic problems: Which information can be extracted from digital images? How to preprocess and represent this information? Sensor-guided robots, automatic vehicle control, Big Brother,... May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

25 Computer Vision Recent textbooks: Yi Ma, St. Soatto, J. Košecká, S.S. Sastry: An Invitation to 3-D Vision. Springer-Verlag, New York 04 R. Hartley, A. Zisserman: Multiple View Geometry in Computer Vision. Cambridge University Press 00 Fortunately the authors in the cited book refer to some of these standard results (Krames, Kruppa, Wunderlich) May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava /55

26 Computer Vision Recent textbooks: Yi Ma, St. Soatto, J. Košecká, S.S. Sastry: An Invitation to 3-D Vision. Springer-Verlag, New York 04 R. Hartley, A. Zisserman: Multiple View Geometry in Computer Vision. Cambridge University Press 00 Fortunately the authors in the cited book refer to some of these standard results (Krames, Kruppa, Wunderlich) May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 21/55

27 Geometry of two images (epipolar geometry) viewing situation (two different centers Z 1 Z 2 ) two central projections together with two collinear transformations Z 1 2 Z 2 π 1 X 1 Z 2 Z 1 Z 2 δ X Z 2 1 π 1 X l 1 l 2 π X 1 2 X 2 γ 1 X 2 γ 1 γ 2 γ 2 z result in two images Z 1 = 2nd image of the 1st center Z 2 = 1st image of the 2nd center Z 2 X X π l l 1 π π 1 X π 2 Z 1 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 22/55

28 Geometry of two images (epipolar geometry) Notations: line z = Z 1 Z 2... baseline, Z 2,Z 1... epipoles (German: Kernpunkte), δ X... epipolar plane (it is twice projecting), Z 1 2 Z 2 π 1 X 1 Z 2 Z 1 Z 2 δ X Z 2 1 π 1 X l 1 l 2 π X 1 2 X 2 γ 1 X 2 γ 1 γ 2 γ 2 z l,l... pair of epipolar lines (German: Kernstrahlen, Ordner) (X,X )... correspondingviews. Z 2 X X π l l 1 π π 1 X π 2 Z 1 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 23/55

29 Epipolar constraint Theorem (synthetic version): For any two linear images of a scene, there is a projectivity between two line pencils Z 2(δ X ) Z 1(δ X ) such that the points X,X are corresponding they are located on (corresponding =) epipolar lines. Theorem (analytic version): Using homogeneous coordinates for both images, there is a bilinear form β of rank 2 such that two points X = x R = (ξ 0 : ξ 1 : ξ 2) and X = x R = (ξ 0 : ξ 1 : ξ 2) are corresponding 2 ξ β(x,x ) = b ij ξ iξ j = (ξ 0 ξ 1 ξ 2) (b ij ) 0 = x T B x = 0. i,j=0 ξ 1 ξ 2 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 24/55

30 Epipolar constraint Theorem (synthetic version): For any two linear images of a scene, there is a projectivity between two line pencils Z 2(δ X ) Z 1(δ X ) such that the points X,X are corresponding they are located on (corresponding =) epipolar lines. Theorem (analytic version): Using homogeneous coordinates for both images, there is a bilinear form β of rank 2 such that two points X = x R = (ξ 0 : ξ 1 : ξ 2) and X = x R = (ξ 0 : ξ 1 : ξ 2) are corresponding 2 ξ β(x,x ) = b ij ξ iξ j = (ξ 0 ξ 1 ξ 2) (b ij ) 0 = x T B x = 0. i,j=0 ξ 1 ξ 2 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 24/55

31 Epipolar constraint Proof (analytic version): Using homogeneous line coordinates, the projectivity between the line pencils can be expressed as β: (u 1λ 1 +u 2λ 2 )R (u 1λ 1 +u 2λ 2 )R for all (λ 1,λ 2 ) R 2 \{(0,0)}. x and x are corresponding there is a nontrivial pair (λ 1,λ 2 ) such that (u 1λ 1 +u 2λ 2 ) x = 0 (u 1λ 1 +u 2λ 2 ) x = 0. These two linear homogeneous equations in the unknowns (λ 1,λ 2 ) have a nontrivial solution the determinant vanishes, i.e., β(x,x ) := (u 1 x )(u 2 x ) (u 2 x )(u 1 x ) = 2 i,j=0 b ijξ i ξ j = 0. There are singular points of this correspondance: Z 2 corresponds to all X, and vice versa all points X correspond to Z 1 = rk(b ij ) = 2. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 25/55

32 Epipolar constraint β(x,x ) = 2 i,j=0 b ij ξ iξ j = (ξ 0 ξ 1 ξ 2) (b ij ) ξ 0 ξ 1 ξ 2 = x T B x = 0. The epipoles solve systems of homogeneous linear equations with the coefficient matrices B and B : Z 2 = (ξ 0 : ξ 1 : ξ 2) corresponds to all X 2 i=0 b ijξ i Z 1 = (ξ 0 : ξ 1 : ξ 2) corresponds to all X 2 j=0 b ijξ j = 0 for j = 0,1,2. = 0 for i = 0,1,2. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 26/55

33 Epipolar constraint in the calibrated case Theorem: In the calibrated case the essential matrix B = (b ij ) is the product of a skew symmetric matrix and an orthogonal one, i.e., B = S R. Z 2 1 Z 2 π 1 X 1 Z 2 z Z 1 Z 2 δ X Z 1 2 π 1 X l 1 x x l 2 X 1 π 2 X 2 X 2 Proof: We use both camera frames and the homogeneous coordinates x = Z 1 X, x = Z 2 X. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 27/55

34 Epipolar constraint in the calibrated case For transforming the coordinates from the second camera frame into the first one, there is an orthogonal matrix R such that x 1 = z +R x with R = R 1 and z = (z 1, z 2, z 3) = Z 1 Z 2. The points X 1, X 2, Z 1, Z 2 are coplanar the triple product of the vectors x, z and x 1 = Z 1 X 2 vanishes, i.e., det(x,z,x 1) = x (z x 1) = 0. Z 2 1 Z 2 π 1 X 1 Z 2 z Z 1 Z 2 δ X Z 1 2 π 1 X l 1 x x l 2 X 1 π 2 X 2 X 2 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 28/55

35 Epipolar constraint in the calibrated case For transforming the coordinates from the second camera frame into the first one, there is an orthogonal matrix R such that x 1 = z +R x with R = R 1 and z = (z 1, z 2, z 3) = Z 1 Z 2. The points X 1, X 2, Z 1, Z 2 are coplanar the triple product of the vectors x, z and x 1 = Z 1 X 2 vanishes, i.e., det(x,z,x 1) = x (z x 1) = 0. Z 2 1 Z 2 π 1 X 1 Z 2 z Z 1 Z 2 δ X Z 1 2 π 1 X l 1 x x l 2 X 1 π 2 X 2 X 2 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 28/55

36 Epipolar constraint in the calibrated case We replace the vector product (z x 1) by z (z +R x ) = z (R x ) = S R x with S = Matrix S is skew symmetric and R is orthogonal. 0 z 3 z 2 z 3 0 z 1 z 2 z 1 0. Hence, the coplanarity of x, x and z is equivalent to 0 = x (z x 1) = x T S R }{{} B x, hence B = S R. The decomposition of the fundamental matrix B into these two factors defines the relative position of the second camera frame against the first one! May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 29/55

37 Epipolar constraint in the calibrated case We replace the vector product (z x 1) by z (z +R x ) = z (R x ) = S R x with S = Matrix S is skew symmetric and R is orthogonal. 0 z 3 z 2 z 3 0 z 1 z 2 z 1 0. Hence, the coplanarity of x, x and z is equivalent to 0 = x (z x 1) = x T S R }{{} B x, hence B = S R. The decomposition of the fundamental matrix B into these two factors defines the relative position of the second camera frame against the first one! May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 29/55

38 Singular value decomposition (SVD) A a 2 x A a 0 a 1 LinAlg U D V A α α(a 2 ) α(a 0 ) α(x) LinAlg α(a 1 ) May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 30/55

39 Singular value decomposition (SVD) A a 2 x A a 0 a 1 LinAlg U D V A α α(a 2 ) α(a 0 ) α(x) LinAlg α(a 1 ) rotation V rotation U D scaling LinAlg Lin May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 30/55

40 Singular value decomposition (SVD) Theorem: [Singular value decomposition] Any matrix A M(m,n;R) can be decomposed into a product A = U D V with orthogonal U,V and D = diag(σ 1,...,σ p ) with D M(m,n;R), σ i 0, and p = min{m,n}. The positive entries in the main diagonal of D are called singular values of A. The squares of the singular values are the non-zero eigenvalues of A A. n m n n m = m n A = U D V May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 31/55

41 Singular value decomposition (SVD) Theorem: [Singular value decomposition] Any matrix A M(m,n;R) can be decomposed into a product A = U D V with orthogonal U,V and D = diag(σ 1,...,σ p ) with D M(m,n;R), σ i 0, and p = min{m,n}. The positive entries in the main diagonal of D are called singular values of A. The singular values of A can be seen as non-vanishing principal distortion factors of the affine transformation represented by A, i.e., the semiaxes of the affine image of the unit sphere. E.g., the singular values of an orthogonal projection are (1,1) as the unit sphere is mapped onto a unit disk. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 32/55

42 Singular values of the essential matrix Theorem: In the calibrated case the essential matrix B has two equal singular values σ := σ 1 = σ 2. Proof: We have B = S R with orthogonal R. The vector S x = z x is orthogonal zu the orthogonal view x n, where z x = sinϕ x z = = x n z = σ x n. x ϕ z x n z x Π z May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 33/55

43 What means reconstruction Given: Two either calibrated or uncalibrated images. π 1 X 1 X 1 2 X X 2 π 1 π π 2 X 2 X 1 Wanted: viewing situation, i.e., determine the relative position of the two camera frames, and for each pair (X,X ) of images the location of the orginal space point X. Z 2 1 Z 2 π 1 X 1 Z 2 Z 1 Z 2 δ X Z 1 2 π 1 X l 1 l 2 X 1 π 2 X 2 X 2 z May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 34/55

44 First fundamental theorem Theorem 1: From two uncalibrated images with given projectivity between epipolar lines the depicted object can be reconstructed up to a collinear transformation. Sketch of the proof: The two images can be placed in space such that pairs of epipolar lines are intersecting. Then for arbitrary Z 1,Z 2 on the baseline z = Z 2 1Z 1 2 there is a reconstructed 3D object. Any other choice of the viewing situation gives a collinear transform of the 3D object. Z 2 1 Z 2 π 1 X 1 Z 2 Z 1 Z 2 δ X Z 1 2 π 1 X l 1 l 2 X 1 π 2 X 2 X 2 z May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 35/55

45 Second fundamental theorem Theorem 2 (S. Finsterwalder, 99): From two calibrated images with given projectivity between epipolar lines the depicted object can be reconstructed up to a similarity. Sketch of the proof: Now in the two bundles of rays the pencils of epipolar planes δ X are congruent, and they can be made coincident by a rigid motion. Then relative to the first bundle Z 1 for any Z 2 z there is a reconstructed 3D object. Any other choice of Z 2 gives a similar 3D object. Z 2 1 Z 2 π 1 X 1 Z 2 Z 1 Z 2 δ X Z 1 2 π 1 X l 1 l 2 X 1 π 2 X 2 X 2 z May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 36/55

46 Aerial photographs Historical remarks: High strategic importance of aerial photogrammetry, already from World War I on. Mechanical devices ( stereo comparators ) were developed for the reconstruction from a pair of photos. Now, in the time of GPS, the exterior calibration parameters are always available with rather high precision available. Hence numerical methods are preferred. Orthophoto, a geometrically corrected aerial photograph (cadastral boundaries) May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 37/55

47 3. Numerical reconstruction from two images Problem of Projectivity: Given: 7 pairs of corresponding points (X 1,X 1),...,(X 7,X 7). Wanted: A pair of points (S,S ) (=epipoles) such that there is a projectivity S ([S X 1],...,[S X 7]) S ([S X 1],...,[S X 7]). X 1 X 2 X 4 X 3 π X 7 X 4 X 6 X 3 X 5 π X 5 X 7 X 6 X 2 X 1 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 38/55

48 Determination of epipoles geometric meaning Problem of Projectivity: Given: 7 pairs of corresponding points (X 1,X 1),...,(X 7,X 7). Wanted: A pair of points (S,S ) (=epipoles) such that there is a projectivity S ([S X 1],...,[S X 7]) S ([S X 1],...,[S X 7]). S X 7 X 1 X 2 X X 3 4 X 6 X 5 π X 4 X 5 X 7 X 6 X 3 X 2 S π X 1 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 38/55

49 Determination of epipoles analytic solution Theorem: If 7 pairs of corresponding points (X 1,X 1),...,(X 7,X 7) ( control points ) are given, the determination of the epipoles is a cubic problem. Proof: 7 pairs of corresponding points give 7 linear homogeneous equations β(x i,x i ) = x i B x i = 0, i = 1,...,7, for the 9 entries in the (3 3)-matrix B = (b ij ) called essential matrix. det(b ij ) = 0 gives an additional cubic equation which fixes all b ij up to a common factor. For noisy image points one should use more than 7 control points and then methods of least square approximation for obtaining the best fitting matrix B: May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 39/55

50 Determination of epipoles analytic solution Theorem: If 7 pairs of corresponding points (X 1,X 1),...,(X 7,X 7) ( control points ) are given, the determination of the epipoles is a cubic problem. Proof: 7 pairs of corresponding points give 7 linear homogeneous equations β(x i,x i ) = x i B x i = 0, i = 1,...,7, for the 9 entries in the (3 3)-matrix B = (b ij ) called essential matrix. det(b ij ) = 0 gives an additional cubic equation which fixes all b ij up to a common factor. For noisy image points one should use more than 7 control points and then methods of least square approximation for obtaining the best fitting matrix B: May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 39/55

51 Determination of epipoles analytic solution 1) Let A denote the coefficient matrix in the linear homogeneous system for the entries of B. Then the least square fit for this overdetermined system is an eigenvector for the smallest eigenvalue of the symmetric matrix A A (=smallest singular value of A). 2) Since the essential matrix must have rank 2, we use the projection into the essential space. This means, the singular value decomposition of B gives a representation B = U diag(σ 1, σ 2, σ 3 ) V with orthogonal U,V and σ 1 σ 2 σ 3. Then in the uncalibrated case B = U diag(σ 1,σ 2,0) V is optimal (with respect to the Frobenius norm) and in the calibrated case B = U diag(σ,σ,0) V with σ = (σ 1 +σ 2 )/2. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 40/55

52 Determination of epipoles analytic solution 1) Let A denote the coefficient matrix in the linear homogeneous system for the entries of B. Then the least square fit for this overdetermined system is an eigenvector for the smallest eigenvalue of the symmetric matrix A A (=smallest singular value of A). 2) Since the essential matrix must have rank 2, we use the projection into the essential space. This means, the singular value decomposition of B gives a representation B = U diag(σ 1, σ 2, σ 3 ) V with orthogonal U,V and σ 1 σ 2 σ 3. Then in the uncalibrated case B = U diag(σ 1,σ 2,0) V is optimal (with respect to the Frobenius norm) and in the calibrated case B = U diag(σ,σ,0) V with σ = (σ 1 +σ 2 )/2. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 40/55

53 3. Numerical reconstruction of two images Step 1: Specify at least 7 reference points manually or automatically by methods of pattern recognition May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 41/55

54 Step 2: Compute the essential matrix Two images with depicted epipolar lines [see J. Košecká et al.] Remark: In the calibrated case 5 pairs of corresponding points are needed, since in the decomposition B = S R each factor depends on 3 parameters only. Because of the homogeneity only 5 unknowns are essential. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 42/55

55 Step 2: Compute the essential matrix Step 2: Compute the essential matrix B including the pairs of epipolar lines May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 43/55

56 Step 3: Factorize B = S R Theorem: There are exactly two ways of decomposing B = U D V with D = diag(σ,σ,0) into a product S R with skew-symmetric S and orthogonal R: S = ±U R + D U and R = ±U R + V with R + = Proof: a) It is sufficient to factorize U D = S R which implies B = S (R V ), i.e., R = R V. b) D represents the product of the orthogonal projection into the x 1 x 2 -plane and the scaling with factor σ. The rotation U transforms the x 1 x 2 -plane into the image plane of U D. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 44/55

57 Step 3: Factorize B = S R Theorem: There are exactly two ways of decomposing B = U D V with D = diag(σ,σ,0) into a product S R with skew-symmetric S and orthogonal R: S = ±U R + D U and R = ±U R + V with R + = Proof: a) It is sufficient to factorize U D = S R which implies B = S (R V ), i.e., R = R V. b) D represents the product of the orthogonal projection into the x 1 x 2 -plane and the scaling with factor σ. The rotation U transforms the x 1 x 2 -plane into the image plane of U D. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 44/55

58 Step 3: Factorize B = S R c) In the case S = we have 0 z 3 z 2 z 3 0 z 1 z 2 z 1 0 Sx = z x for z = z 1 z 2 z 3. Hence, the skew symmetric matrix S represents the product of an orthogonal projection parallel to z, a 90 -rotation about z and a scaling with factor z. x ϕ z x n z x Π z May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 45/55

59 Step 3: Factorize B = S R d) Matrix R + = is orthogonal, R + D = 0 σ 0 σ is skew-symmetric with z = (0,0,σ). We transform it by U to obtain the required position, i.e., S = ±U (R + D) U. R + commutes with D = U D = U D R + U U R + = = [ ±U R + D U ] }{{} S [±U R ] + }{{} R. e) B represents an orthogonal axonometry; its column vectors are images of an orthonormal frame. We know from Descriptive Geometry that apart from translations there are not more than two different frames with given images. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 46/55

60 Step 3: Factorize B = S R d) Matrix R + = is orthogonal, R + D = 0 σ 0 σ is skew-symmetric with z = (0,0,σ). We transform it by U to obtain the required position, i.e., S = ±U (R + D) U. R + commutes with D = U D = U D R + U U R + = = [ ±U R + D U ] }{{} S [±U R ] + }{{} R. e) B represents an orthogonal axonometry; its column vectors are images of an orthonormal frame. We know from Descriptive Geometry that apart from translations there are not more than two different frames with given images. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 46/55

61 Step 4: Intersecting corresponding rays In one of the frames compute the approximate point of intersection between corresponding rays. photo 2 s x photo 1 c 2 x c 1 For the center of the common perpendicular line segment the sum of squared distances is minimal. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 47/55

62 Summary of algorithm 1) Specify n > 7 pairs (X i,x i ), i = 1,...,n. 2) Set up linear system of equations for the essential matrix B and seek best fitting matrix (eigenvector of the smallest eigenvalue). 3) Compute the closest rank 2 matrix B with two equal singular values. 4) Factorize B = S R; this reveals the relative position of the two camera frames. 5) In one of the frames compute the approximate point of intersection between corresponding rays. 6) Transform the recovered coordinates into world coordinates. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 48/55

63 Remaining problems Analysis of precision (e.g., no precise linear images due to effects of lenses) Effects of automated calibration (autofocus and zooming change the focal distance d), There are critical configurations (e.g., all passpoints within one plane) where the problem of projectivity has no unique solution despite of an arbitrarily big number of control points. How to figure out the correct one? How to find the correct decomposition of B = S R among the two possible one? May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 49/55

64 The solution original image the reconstruction (M 1 : 0) May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 50/55

65 9 front view Z 1 Z 2 2 top view 9 Photo 1 Z 2 Position of centers relative to the depicted object Photo 2 May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 51/55 Z 1

66 Literatur H. Brauner: Lehrbuch der konstruktiven Geometrie. Springer, Wien 86. H. Brauner: Lineare Abbildungen aus euklidischen Räumen. Beitr. Algebra Geom. 21, 5 26 (86). A. Dür: An Algebraic Equation for the Central Projection. J. Geometry Graphics 7, 7 3 (03). O. Faugeras: Three-Dimensional Computer Vision. A Geometric Viewpoint. MIT Press, Cambridge, Mass., 06. O. Faugeras, Q.-T. Luong: The Geometry of Multiple Images. MIT Press, Cambridge, Mass., 01. May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 52/55

67 R. Harley, A. Zisserman: Multiple View Geometry in Computer Vision. Cambridge University Press 00. V. Havel: O rozkladu singulárních lineárních transformací. Časopis Pést. Mat. 85, (60). H. Havlicek: On the Matrices of Central Linear Mappings. Math. Bohem. 1, 1 6 (96). M. Hoffmann: On the Theorems of Central Axonometry. J. Geometry Graphics 2, 1 5 (97). M. Hoffmann, P. Yiu: Moving Central Axonometric Reference Systems. J. Geometry Graphics 9, (05) (in press). E. Kruppa: Zur achsonometrischen Methode der darstellenden Geometrie. Sitzungsber., Abt. II, österr. Akad. Wiss., Math.-Naturw. Kl. 9, (). May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 53/55

68 Yi Ma, St. Soatto, J. Košecká, S. Sh. Sastry: An Invitation to 3-D Vision. Springer-Verlag, New York 04. H. Stachel: Mehrdimensionale Axonometrie. Proceedings of the Congress of Geometry, Thessaloniki 87, 9 8. H. Stachel: Parallel Projections in Multidimensional Space. Proceedings of Compugraphics 91, Sesimbra (Portugal) 91: Vol. I, 9 8. H. Stachel: Zur Kennzeichnung der Zentralprojektionen nach H. Havlicek. Sitzungsber., Abt. II, österr. Akad. Wiss., Math.-Naturw. Kl. 4, 33 46(95). H. Stachel: On Arne Dür s Equation Concerning Central Axonometries. J. Geometry Graphics 8, (04). E. Stiefel: Zum Satz von Pohlke. Comment. Math. Helv., (38). May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 54/55

69 E. Stiefel: Lehrbuch der darstellenden Geometrie. 3. Aufl., Basel, Stuttgart 71. H. Stachel: Descriptive Geometry Meets Computer Vision The Geometry of Two Images. J. Geometry Graphics, 7 3 (06). J. Szabó, H. Stachel, H. Vogel: Ein Satz über die Zentralaxonometrie. Sitzungsber., Abt. II, österr. Akad. Wiss., Math.-Naturw. Kl. 3, 3 (94). J. Tschupik, F. Hohenberg: Die geometrische Grundlagen der Photogrammetrie. In Jordan, Eggert, Kneissl (eds.): Handbuch der Vermessungskunde III a/3.. Aufl., Metzlersche Verlagsbuchhandlung, Stuttart 72, K. Vala: A propos du théorèm de Pohlke. Ann. Acad. scient. Fennicae 270, 3 5 (59). May 26, : Seminar, Institute of Mathematics and Physics, STU Bratislava 55/55