Lecture 5 Epipolar Geometry Professor Silvio Savarese Computational Vision and Geometry Lab Silvio Savarese Lecture 5-21-Jan-15
Lecture 5 Epipolar Geometry Why is stereo useful? Epipolar constraints Essential and fundamental matrix Estimating F Examples Reading: [HZ] Chapter: 4 Estimation 2D perspective transformations Chapter: 9 Epipolar Geometry and the Fundamental Matrix Transformation Chapter: 11 Computation of the Fundamental Matrix F [FP] Chapter: 7 Stereopsis Chapter: 8 Structure from Motion Silvio Savarese Lecture 5-21-Jan-15
Recovering structure from a single view Pinhole perspective projection P p O w Calibration rig Scene C Camera K From calibration rig location/pose of the rig, K From points and lines at infinity + orthogonal lines and planes structure of the scene, K Knowledge about scene (point correspondences, geometry of lines & planes, etc
Recovering structure from a single view Pinhole perspective projection P p O w Calibration rig Scene C Camera K Why is it so difficult? Intrinsic ambiguity of the mapping from 3D to image (2D)
Recovering structure from a single view Intrinsic ambiguity of the mapping from 3D to image (2D) Courtesy slide S. Lazebnik
Two eyes help!
Two eyes help! P = l l" [Eq. 1] P? l l' K 1 =known p p K 2 =known R, T O 1 O 2 This is called triangulation
Triangulation Find P that minimizes d(p, M 1 P')+ d(p', M 2 P') P P [Eq. 2] M 1 P p M 2 P p image 1 image 2 O 1 O 2
Stereo- view geometry Correspondence: Given a point p in one image, how can I find the corresponding point p in another one? Camera geometry: Given corresponding points in two images, find camera matrices, position and pose. Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.
Epipolar geometry P p p e e Epipolar Plane Epipoles e, e Baseline Epipolar Lines O 1 O 2 = intersections of baseline with image planes = projections of the other camera center
Example of epipolar lines
Example: Parallel image planes P e p p e O 1 O 2 Baseline intersects the image plane at infinity Epipoles are at infinity Epipolar lines are parallel to x axis
Example: Parallel Image Planes e at infinity e at infinity
Example: Forward translation e P O 2 e O 1 The epipoles have same position in both images Epipole called FOE (focus of expansion)
Epipolar Constraint p - Two views of the same object - Suppose I know the camera positions and camera matrices - Given a point on left image, how can I find the corresponding point on right image?
Epipolar geometry P Epipolar line 2 p p O 1 O 2
Epipolar Constraint
!!! " # % & = 1 v u M P P O 1 O 2 p p P R, T Epipolar Constraint!!! " # % & ' ' = ' 1 v u P M P [Eq. 3] [Eq. 4] M = K I 0! " # M ' = K ' R T R T T " # % &'
Epipolar Constraint P! # K canonical = # # " 1 0 0 0 1 0 0 0 1 & & & % p p O 1 R, T O 2 M = K! I 0 " # K = K are known (canonical cameras) " M ' = K ' R T R T T # % &' M = [ I 0] [Eq. 5] " M ' = R T R T T # % &' [Eq. 6]
The cameras are related by R, T Camera 2 p T O 2 Camera 1 O 1 T = O 2 in the camera 1 reference system R is the rotation matrix such that a vector p in the camera 2 is equal to R p in camera 1.
Epipolar Constraint P p p O 1 R, T O 2 p in first camera reference system is = R p! T ( R p!) is perpendicular to epipolar plane p T [ T ( R p! )] = 0 [Eq. 7]
b a b a ] [ 0 0 0 = " " " # % % % & ' " " " # % % % & ' = z y x x y x z y z b b b a a a a a a Cross product as matrix multiplication
Epipolar Constraint P p p O 1 R, T O 2 [ T (R p )] 0 p T! = [ T ] R p 0 p T! = [Eq. 8] [Eq. 9] E = Essential matrix (Longuet-Higgins, 1981)
Epipolar Constraint p T E p' = 0 P [Eq. 10] p l l e e p O 1 O 2 l = E p is the epipolar line associated with p l = E T p is the epipolar line associated with p E e = 0 and E T e = 0 E is 3x3 matrix; 5 DOF E is singular (rank two)
Epipolar Constraint P p p R, T O 1 O 2 M = K[ I 0] " M ' = K ' R T R T # T % &' p = K 1 p p! = K ' 1 p! c c [Eq. 11] [Eq. 12]
Epipolar Constraint p c = K 1 p p! = K ' 1 p! c P p p [Eq.9] O 1 O 2 p T c [ T ] R # 1 T p = 0 (K p) [ T ] R K! p! = 0 c 1 p T K [ T ] R K! 1 p! = 0 p T! = T F p 0 [Eq. 13]
Epipolar Constraint P p p O 1 O 2 Eq. 13 p T F p! = 0 F [ ] 1 = K T T R K# F = Fundamental Matrix (Faugeras and Luong, 1992) Eq. 14
Epipolar Constraint p T F p' = 0 P p l e e l p O 1 O 2 l = F p is the epipolar line associated with p l = F T p is the epipolar line associated with p F e = 0 and F T e = 0 F is 3x3 matrix; 7 DOF F is singular (rank two)
Why F is useful? p l = F T p - Suppose F is known - No additional information about the scene and camera is given - Given a point on left image, how can I find the corresponding point on right image?
Why F is useful? F captures information about the epipolar geometry of 2 views + camera parameters MORE IMPORTANTLY: F gives constraints on how the scene changes under view point transformation (without reconstructing the scene!) Powerful tool in: 3D reconstruction Multi- view object/scene matching
O 1 O 2 p p P Estimating F The Eight-Point Algorithm!!! " # % & ' ' = ' 1 v u p P!!! " # % & = 1 v u p P 0 F p p T =! (Hartley, 1995) (Longuet-Higgins, 1981)
Estimating F [Eq. 13] p T F p! = 0! F11 F12 F13 "! u '" # # u, v,1 F F F v ' = 0 ( ) 21 22 23 # # F31 F32 F # 33 1 % &% & ( uu uv u vu vv v u v ) Let s take 8 corresponding points! F11 " # # F12 # F 13 # F # 21 ', ',, ', ',, ', ',1 # F = 0 22 # # F23 # # F31 # F 32 # F % 33 & [Eq. 14]
Estimating F
W Homogeneous system Estimating F! F11 "! u u ' u v ' u v u ' v v ' v u ' v ' 1"# F 1 1 1 1 1 1 1 1 1 1 1 1 # u2u ' 2 u2v ' 2 u2 v2u ' 2 v2v ' 2 v2 u ' 2 v ' 2 1 # 12 # # F 13 # u3u ' 3 u3v ' 3 u3 v3u ' 3 v3v ' 3 v3 u ' 3 v ' 3 1# # # F21 u4u ' 4 u4v ' 4 u4 v4u ' 4 v4v ' 4 v4 u ' 4 v ' 4 1 F # 22 u5u ' 5 u5v ' 5 u5 v5u ' 5 v5v ' 5 v5 u ' 5 v ' 5 1# # # F23 # u6u ' 6 u6v ' 6 u6 v6u ' 6 v6v ' 6 v6 u ' 6 v ' 6 1# # F31 u7u ' 7 u7v ' 7 u7 v7u ' 7 v7v ' 7 v7 u ' 7 v ' 7 1# # # F32 u8u ' 8 u8v ' 8 u8 v8u ' 8 v8v ' 8 v8 u ' 8 v ' 8 1 % &# # F % 33 & # # = Wf = 0 0 f [Eqs. 15] Rank 8 If N>8 A non-zero solution exists (unique) Lsq. solution by SVD! f =1 Fˆ
Fˆ satisfies: p T Fˆ p! = 0 ^ and estimated F may have full rank (det(f) 0) But remember: fundamental matrix is Rank2 ^ Find F that minimizes F Fˆ = 0 Frobenius norm (*) Subject to det(f)=0 SVD (again!) can be used to solve this problem (*) Sq. root of the sum of squares of all entries
Find F that minimizes F Fˆ = 0 Frobenius norm (*) Subject to det(f)=0! # F =U# # "# s 1 0 0 0 s 2 0 0 0 0 & & & %& V T [HZ] pag 281, chapter 11, Computation of F Where:! # U# # "# s 1 0 0 0 s 2 0 0 0 s 3 & & V T = SVD( ˆF) & %&
Data courtesy of R. Mohr and B. Boufama.
Mean errors: 10.0pixel 9.1pixel
Problems with the 8-Point Algorithm W f = 0, Lsq solution by SVD f =1 F - Recall the structure of W: - do we see any potential (numerical) issue?
Problems with the 8-Point Algorithm Wf = 0! F11 "! u u ' u v ' u v u ' v v ' v u ' v ' 1"# F 1 1 1 1 1 1 1 1 1 1 1 1 # u2u ' 2 u2v ' 2 u2 v2u ' 2 v2v ' 2 v2 u ' 2 v ' 2 1 # 12 # # F 13 # u3u ' 3 u3v ' 3 u3 v3u ' 3 v3v ' 3 v3 u ' 3 v ' 3 1# # # F21 u4u ' 4 u4v ' 4 u4 v4u ' 4 v4v ' 4 v4 u ' 4 v ' 4 1 F # 22 u5u ' 5 u5v ' 5 u5 v5u ' 5 v5v ' 5 v5 u ' 5 v ' 5 1# # # F23 # u6u ' 6 u6v ' 6 u6 v6u ' 6 v6v ' 6 v6 u ' 6 v ' 6 1# # F31 u7u ' 7 u7v ' 7 u7 v7u ' 7 v7v ' 7 v7 u ' 7 v ' 7 1# # # F32 u8u ' 8 u8v ' 8 u8 v8u ' 8 v8v ' 8 v8 u ' 8 v ' 8 1 % &# # F % 33 & # # = Highly un- balanced (not well conditioned) Values of W must have similar magnitude This creates problems during the SVD decomposition 0 HZ pag 108
Normalization IDEA: Transform image coordinates such that the matrix W becomes better conditioned (pre-conditioning) For each image, apply a following transformation T (translation and scaling) acting on image coordinates such that: Origin = centroid of image points Mean square distance of the image points from origin is ~2 pixels q = T p i i q = T! p!! (normalization) i i
Example of normalization T 2 pixels Coordinate system of the image before applying T Coordinate system of the image after applying T Origin = centroid of image points Mean square distance of the image points from origin is ~2 pixels
The Normalized Eight-Point Algorithm 0. Compute T and T for image 1 and 2, respectively 1. Normalize coordinates in images 1 and 2: 2. Use the eight-point algorithm to compute from the corresponding points q and q. i q = T p i i i q! = T! p! i i ˆF q 1. Enforce the rank-2 constraint. F q such that: q T F q q! = 0 2. De-normalize F q : F T ' = T F T det( Fq ) = 0 q
With transformation Without transformation Mean errors: 10.0pixel 9.1pixel Mean errors: 1.0pixel 0.9pixel
Lecture 5 Epipolar Geometry Why is stereo useful? Epipolar constraints Essential and fundamental matrix Estimating F Examples Reading: [AZ] Chapter: 4 Estimation 2D perspective transformations Chapter: 9 Epipolar Geometry and the Fundamental Matrix Transformation Chapter: 11 Computation of the Fundamental Matrix F [FP] Chapter: 7 Stereopsis Chapter: 8 Structure from Motion Silvio Savarese Lecture 5-21-Jan-15
Example: Parallel image planes P e p p e y z K K x O 1 O 2 Hint : E=? K 1 =K 2 = known R = I T = (T, 0, 0) x parallel to O 1 O 2
!!! " # % & =!!! " # % & = 0 0 0 0 0 0 0 0 0 0 T T T T T T T T x y x z y z I E Essential matrix for parallel images T = [ T 0 0 ] R = I [ ] R T E = [Eq. 20]
Example: Parallel image planes P e l p p l e y z K K O 1 What are the directions of epipolar lines? x " l = E p' = # 0 0 0 0 0 T 0 T 0 %" ' ' ' &# u v 1 % " ' ' = ' & # 0 T T v % ' ' ' & horizontal!
Example: Parallel image planes P e p p e y z K K O 1 x How are p and p related? p T E p' = 0
Example: Parallel image planes P e p p e y z K K O 1 x How are p and p related? " 0 0 0 # u! % 0 %! ( u v ( ) T ) & ' & ' = v! ( u v ) T Tv Tv! ( ) = = = ( 0 T 0 ) & 1 ' & Tv!', -. /. / v = v" T p E p 0 1 0 0 0 1 0
Example: Parallel image planes P e p p e y z K K O 1 x Rectification: making two images parallel Why it is useful? Epipolar constraint v = v New views can be synthesized by linear interpolation
Application: view morphing S. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30
Rectification P H 1 I 1 I s î 1 v u I 2 î S v u C 1 v u î 2 H 2 C s C 2
Morphing without rectifying
From its reflection!
The Fundamental Matrix Song http://danielwedge.com/fmatrix/