Lecture 4.3 Estimating homographies from feature correspondences. Thomas Opsahl

Transcription

1 Lecture 4.3 Estimating homographies from feature correspondences Thomas Opsahl

2 Homographies induced by central projection 1 H 2 1 H 2 u uu Homography Hu = u H = h 1 h 2 h 3 h 4 h 5 h 6 h 7 h 8 h 9 Point-correspondences can be determined automatically Erroneous correspondences are common Robust estimation is required to find H 2

3 Homographies induced by central projection u H uu H 2 Homography Hu = u H = h 1 h 2 h 3 h 4 h 5 h 6 h 7 h 8 h 9 H 1 2 H 1 3 Point-correspondences can be determined automatically Erroneous correspondences are common Robust estimation is required to find H 3

4 Estimating the homography between overlapping images Establish point correspondences u i uu i Find key points u i Img1 and uu i Img2 Represent key points by suitable descriptors Determine correspondences u i u i by matching descriptors Some wrong correspondences are to be expected Estimate the homography H such that uu i = Hu i i Robust estimation with RANSAC Improved estimation based on RANSAC inliers This homography enables us to compose the images into a larger image Image mosaicing Panorama 4

5 Adaptive RANSAC Objective To robustly fit a model y = f x; α to a data set S containing outliers Algorithm 1. Let N =, S II = and #iiiiiiiiii = 0 2. while N > #iiiiiiiiii repeat Estimate parameters α ttt from a random n-tuple from S 4. Determine inlier set S ttt, i.e. data points within a distance t of the model y = f x; α ttt 5. If S ttt > S II, set S II = S ttt, α = α ttt, ω = S II S Increase #iiiiiiiiii by 1 and N = lll 1 p lll 1 ω n with p =

6 Estimating the homography Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for n point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography 6

7 Estimating the homography Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for n point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography The homography has 8 degrees of freedom, but it is custom to treat all 9 entries of the matrix as unknowns instead of setting one of the entries to 1 which excludes all potential solutions where this entry is 0 Let us solve the equation Hu = u for the entries of the homography matrix Hu = u h 1 h 2 h 3 u u h 4 h 5 h 6 v = v h 7 h 8 h

8 Basic homography estimation h h h h h h u u uh + vh + h = u u v 1 vu vv v h h h h v = v uh + vh + h = v u v uu uv u h h h h 1 1 uh vh h = vu vv v uu uv u h h h h = 0 Ah= 0 0 8

9 Basic homography estimation h h h h h h u u uh + vh + h = u u v 1 vu vv v h h h h v = v uh + vh + h = v u v uu uv u h h h h 1 1 uh vh h = vu vv v uu uv u h h h h Observe that the third row in A is a linear combination of the first and second row rrr 3 = u rrr 1 v rrr = 0 Ah= 0 0 Hence every correspondence u i u i contribute with 2 equations in the 9 unknown entries 9

10 Basic homography estimation Since H (and thus h) is homogeneous, we only need the matrix A to have rank 8 in order to determine h up to scale It is sufficient with 4 point correspondences where no 3 points are collinear We can calculate the non-trivial solution to the equation Ah = 0 by SVD svd A = UUV T The solution is given by the right singular vector without a singular value which is the last column of V, i.e. h = v 9 h1 h u1 v1 1 vu 1 1 vv 1 1 v 1 h 3 0 u1 v uu 1 1 uv 1 1 u 1 h u2 v2 1 vu 2 2 vv 2 2 v 2 h 5 = 0 u2 v uu 2 2 uv 2 2 u 2h6 0 h 7 h8 h 9 Ah= 0 10

11 Basic homography estimation Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for n point-correspondences 2. A way to determine which of the point correspondences that are inliers for a given homography Direct Linear Transform u1 v1 1 vu 1 1 vv 1 1 v 1 A= u1 v uu 1 1 uv 1 1 u 1 1. Build the matrix A from at least 4 point-correspondences u i, v i u i, v i 2. Obtain the SVD of A: A = UUV T 3. If S is diagonal with positive values in descending order along the main diagonal, then h equals the last column of V 4. Reconstruct H from h 11

12 Basic homography estimation The basic DLT algorithm is never used with more than 4 point-correspondences This is because the algorithm performs better when all the terms of A has a similar scale Note that some of the terms will always be of scale 1 To achieve this, it is common to extend the algorithm with a normalization and a denormalization step Normalized Direct Linear Transform 1. Normalize the set of points u i = u i, v i T by computing a similarity transform T that translates the centroid to the origin and scales such that the average distance from the origin is 2 2. In the same way normalize the set of points u i = u i, v i T by computing a similarity transform TT 3. Apply the basic DLT algorithm on the normalized points to obtain a homography H 4. Denormalize to get the homography: H = TT 1 H T 12

13 Basic homography estimation Estimating the homography in a RANSAC scheme requires 1. A basic homography estimation method for n point-correspondences 2. A way to determine the inlier set of point-correspondences for a given homography For a point correspondence u i, v i uu i, vv i and homography H, we can choose from several errors Algebraic error: ε i = A i h where A i ui vi 1 vu i i vv i i v i = ui vi uu i i uv i i u i Geometric errors: 1. ε i = d Hu i, u i + d u i, H 1 uu i (Reprojection error) 2. ε i = d u i, H 1 uu i 3. ε i = d Hu i, uu i Notation Euclidean distance Inhomogenous Hu i Inhomogeneous H 1 u i d, Hu i H 1 uu i 13

14 Robust homography estimation RANSAC estimation of homography For a set of point-correspondences S = u i u i, perform N iterations, where N is determined adaptively 1. Estimate H ttt from 4 random correspondences u i u i using the basic DLT algorithm 2. Determine the set of inlier-correspondences S ttt = u i u i ssss tttt ε i < t Here one can choose ε i = d Hu i, u i + d u i, H 1 u i and t = 5.99σ where σ is the expected uncertainty in key-point positions 3. If S ttt > S II update N, homography and inlier set: H = H ttt, S II = S ttt Finally we would typically re-estimate H from all correspondences in S II Normalized DLT Minimize ϵ = ε i in an iterative optimization method like Levenberg Marquardt 14

15 Image mosaicing Let us compose these two images into a larger image 15

16 Image mosaicing Find key points and represent by descriptors 16

17 Image mosaicing Establish point-correspondences by matching descriptors Several wrong correspondences 17

18 Image mosaicing Establish point-correspondences by matching descriptors Several wrong correspondences 18

19 Image mosaicing H Estimate homography Hu = u OpenCV #include "opencv2/calib3d.hpp" cv::findhomography(srcpoints, dstpoints, CV_RANSAC); Matlab tform = estimategeometrictransform(srcpoints,dstpoints, projective ); 19

20 Image mosaicing Represent the images in common coordinates (Note the additional translation!) OpenCV #include "opencv2/calib3d.hpp" cv::warpperspective(img1, img2, H, output_size); Matlab img2 = imwarp(img1,tform); 20

21 Image mosaicing Now we can compose the images 21

22 Overwriting 22

23 Blending with a ramp 23

24 Blending with a ramp + histogram equalization 24

25 SVD Singular Value Decomposition The singular value decomposition of a real m n matrix A is a factorization A = USV T Here U is a orthogonal m m matrix, V is a orthogonal n n matrix and S is a real positive diagonal m n matrix The diagonal entries of S = diag s 1,, s min m,n are known as the singular values of A and the columns of U = u 1,, u m and V = v 1,, v n are known as the left and right singular vectors of A respectively How to use Matlab [U,S,V] = svd(a); Right singular vectors are columns in V OpenCV cv::svd::compute(a, S, U, Vtranspose, cv::svd::full_uv); Right singular vectors are rows in Vtranspose Eigen Eigen::JacobiSVD<Eigen::MatrixXd> svd(a, Eigen::ComputeFullU Eigen::ComputeFullV); Right singular vectors are columns in svd.matrixv() The nullspace of A is the span of the right singular vectors v i that corresponds to a zero singular value s i (or does not have a corresponding singular value) 25

26 SVD Singular Value Decomposition The singular value decomposition of a real m n matrix A is a factorization A = USV T Here U is a orthogonal m m matrix, V is a orthogonal n n matrix and S is a real positive diagonal m n matrix The diagonal entries of S = diag s 1,, s min m,n are known as the singular values of A and the columns of U = u 1,, u m and V = v 1,, v n are known as the left and right singular vectors of A respectively The nullspace of A is the span of the right singular vectors v i that corresponds to a zero singular value s i (or does not have a corresponding singular value) Applications of SVD Solving homogeneous linear equations like Ah = 0 Method For theoretical problems, h null A so h is a linear combination of the right singular vectors v i that correspond to a zero singular value s i h = k i v i ; k i R, s i = 0 (or missing) For practical problems, the presence of noise force us to expand the solution by including those right singular vectors that correspond to small singular values s i 0 h = k i v i ; k i R, s i 0 (or missing) 26

27 SVD Example Ax= 0 x y = 0 z From [U,S,V] = svd(a); we get 2 nonzero singular values = s = right singular vectors s v1 = v = v = U V = S = = Since v 3 does not have a corresponding singular value, x = v 3 is a non-trivial solution to Ax = 0 and x = k v 3 ; k R\ 0 is the family of all non-trivial solutions From this we see that A has: 2 left singular vectors u = = u

28 SVD Example Ax= x 0 y = z If this equation came from a practical problem, instead of looking for solutions to Ax = 0, we might be looking for the x that minimize Ax Since s 1 0, s 2 0, s 3 0, we would conclude that x = v 3 solves the equation in a least-squares sense This time singular value decomposition give us the following singular values and right singular vectors: s1 = s2 = s3 = v1 = v = v = Check: Av = This time all right singular vectors correspond to a non-zero singular value, so the equation does not have any non-trivial solutions! 28

29 Summary Homography Hu = u h 1 h 2 h 3 H = h 4 h 5 h 6 h 7 h 8 h 9 Automatic point-correspondences Improve estimate by normalized DLT on inliers or iterative methods for an even better estimate Additional reading Szeliski: Wrong correspondences are common RANSAC estimation Basic DLT (Direct Linear Transform) on 4 random correspondences Inliers determined from the reprojection error ε i = d Hu i, u i + d u i, H 1 u i 29