Augmented Reality VU Camera Registration. Prof. Vincent Lepetit

Transcription

1 Augmented Reality VU Camera Registration Prof. Vincent Lepetit

2 Different Approaches to Vision-based 3D Tracking [From D. Wagner] [From Drummond PAMI02] [From Davison ICCV01] Consider natural features Consider natural features Make use of visual markers No a priori 3D knowledge Use some 3D knowledge Use some 3D knowledge 13

3 Camera Model From World to Images 14

4 From the World to the Image World coordinate system M m What is the relation between the 3D coordinates of a point M and its correspondent m in the image captured by the camera? 15

5 Perspective Projection World coordinate system M m C Camera center The image formation is modeled as a perspective projection, which is realistic for standard cameras: The rays passing through a 3D point M and its correspondent m in the image all intersect at a single point C, the camera center. 16

6 Expressing M in the Camera Coordinates System World coordinate system M M cam m Z C Y X Camera coordinate system Step 1: Express the coordinates of M in the camera coordinates system as M cam. This transformation corresponds to a Euclidean displacement (a rotation plus a translation): M cam = RM + T where: R is a 3x3 rotation matrix, and T is a 3- vector. 18

7 Homogeneous Coordinates World coordinate system X X M = Y M Y = Z Z 1 M cam C Y Z Lets replace M by the 4- homogeneous vector : Just add a 1 as the fourth coordinate. m X Camera coordinate system Now, the Euclidean displacement can be expressed as an linear transformation instead of an affine one: M cam = RM + T X cam Y cam Z cam X ( ( = R ( Y ( + T Z M X cam Y cam Z cam X ( ( ( = ( R T Y ) ( Z( ( 1 ( ) M cam = R T M 19 (R T ) is a 3x4 matrix.

8 The matrix ( R T) = The External Calibration Matrix R 11 R 13 R 13 T 1 R 21 R 22 R 23 T 2 R 31 R 32 R 33 T 3 is called the External calibration matrix, or the extrinsic calibration matrix, or the matrix of external parameters. It can be parameterized by 6 values: 3 for the rotation, 3 for the translation. The parameterization of the translation is trivial, while parametrizing rotations is not. We will detail the possible parameterization of rotations in the 3D space. 20

9 21 Homogeneous Coordinates A mathematical trick to express non-linear transformations such as translation or projection as linear transformations. A 3- vector expressed in homogeneous coordinates becomes a 4-vector: To go back to the 3-vector, one just has to divide the 3 first coordinates by the last one. à A homogeneous vector is defined up to a scale factor: X Y Z kx ky kz k All the homogeneous vectors of the form kx ky kz k correspond to the same 3 vector X Y Z. The homogeneous vector u v w t corresponds to the 3 vector u t v t w t.

10 22 Homogeneous Coordinates In homogeneous coordinates, an Euclidean displacement can be written using a 4x4 matrix: The Computer Vision community uses a 3x4 matrix and considers that the 4th coordinate of is 1: k X k Y k Z k ( ( ( ( = R T ( ( ( ( kx ky kz k ( ( ( ( X cam Y cam Z cam = R T ( ) X Y Z 1 M

11 Projection M cam C Y Z X f Camera coordinate system m Z m X M cam Computation of the coordinates of m in the image plane, from M cam (expressed in the camera coordinates system): Simply use Thales theorem: f m X f = X Z m X = f X Z C m X X 23

12 From Projection to Image Coordinates of m in pixels? Image coordinate u m f m X = f X Z, m Y = f Y Z system m u =?, m v =? v 0 v u 0 1 k u 1 1 pixel k v C Camera coordinate system 24

13 - (u 0, v 0 ): Projection of the camera center C in the image coordinate system. - k u : Inverse of the size of one pixel along X axis. - k v : Same for Y axis. Image coordinate system v 0 u v f u 0 m m X = f X Z, m = f Y Y Z m u = u 0 + k u m X, m v = v 0 + k v m Y C Camera coordinate system 25

14 Putting the perspective projection and the transformation into pixel coordinates together: m X = f X Z, m Y = f Y Z m u = u 0 + k u m X, m v = v 0 + k v m Y In matrix form: u k u f 0 u 0 X v = 0 k v f v 0 Y w Z u v defines m in homogeneous coordinates w m u = u w = u + k f X 0 u Z m v = v w = v + k f Y 0 v Z 26

15 The matrix The Internal Calibration Matrix k u f 0 u 0 0 k v f v is called the Internal calibration matrix, or the intrinsic calibration matrix, or the calibration matrix, or the matrix of internal parameters. It can be parameterized by 4 values: α u, α v, u 0, v 0 : α u 0 u 0 ( 0 α v v 0 ( The number of parameters can be reduced under some assumptions: (u 0, v 0 ) is sometimes taken at the center of the image; Taking α u = α v assumes that the pixels are square. 27

16 28 The Full Transformation The two transformations are chained to form the full transformation from a 3D point in the world coordinate system to its projection in the image: The product of the internal calibration matrix and the external calibration matrix is a 3x4 matrix called the projection matrix. The projection matrix is defined up to a scale factor. u v w = k u f 0 u 0 0 k v f v R 11 R 13 R 13 T 1 R 21 R 22 R 23 T 2 R 31 R 32 R 33 T 3 X Y Z 1 = P 11 P 12 P 13 P 14 P 21 P 22 P 23 P 24 P 31 P 32 P 33 P 34 X Y Z 1 projection matrix

17 Notations R, T World coordinate system M C Camera center m m = A[ R T] M = PM A: Internal calibration matrix (3x3); R: Rotation matrix (3x3); T: Translation vector (3-); P: Projection matrix (3x4); M : world 3D point in homogenous coordinates (4-); m: image 2D point in homogenous coordinates (3-), projection of M in the image; 29

18 World coordinate system M R, t m Camera center C Camera coordinate system

19 Internal Parameters (A) A : Often assumed to be known in the following. Can be estimated beforehand using a calibration grid. m i = A[R T]M i 31

20 Camera Distortion The previous camera model does not take into account distortions due to fish eye lenses or bad quality lenses. Fish eye lenses are interesting because images have a larger field of view. Distortion can be removed before applying perspective projection model. 32

21 Distortion Estimation (u,v) (u c,v c ) ( u, v ) Radial + tangential distortions. Tangential distortion component has much less influenceà neglected. (u c,v c ) : Center of radial distortion. The correction is written: ( ) ( ) u = u c + L(r) u u c v = v c + L(r) v v c with ( ) 2 + ( v v c ) 2 and r 2 = u u c L(r) =1+ κ 1 r + κ 2 r 2 + κ 3 r Once the distortion parameters are estimated, the distortion can be compensated in two ways: Undistort the images, using a look-up table for efficiency; Distort the model projection when computing a reprojection error. 33

22 Camera Pose (Rotation + translation) Estimation Algorithms From 3D-2D points correspondences: DLT algorithm + Extraction of A, R, t; P3P algorithm; POSIT and others; Non-linear minimization. From a planar structure. 34

23 Linear Estimation of P from 3D/2D point correspondences (DLT algorithm) Each correspondence gives us 2 equations: m i = P P u i = 11 X i + P 12 Y i + P 13 Z i + P 14 P M 31 X i + P 32 Y i + P 33 Z i + P 34 i v i = P X + P Y + P Z + P 21 i 22 i 23 i 24 P 31 X i + P 32 Y i + P 33 Z i + P 34 P 11 X i + P 12 Y i + P 13 Z i + P 14 P 31 X i u i P 32 Y i u i P 33 Z i u i P 34 u i = 0 P 21 X i + P 22 Y i + P 23 Z i + P 24 P 31 X i v i P 32 Y i v i P 33 Z i v i P 34 v i = 0 X i Y i Z i X i u i Y i u i Z i u i u i ( X i Y i Z i 1 X i v i Y i v i Z i v i v i P P 34 ( ( = 0 Stacking all the equations into matrix B yields the linear system: B P P 34 = 0 35

24 DLT algorithm P P 34 ( ( B Question: Why is the null vector not a valid solution? P P 34 = 0 is the eigen vector of B corresponding to the smallest eigenvalue of B. 11 coefficients to estimate; each 3D-2D correspondence gives 2 equations: 6 3D-2D correspondences must be known. In practice, much more are needed to get a good estimation. 36

25 Extraction of A, R, and T from P 1. A is known. 2. A is not known. Trick: Consider the matrix made of the first 3 columns of P: P 3 = AR P = ( P 3 c 4 ) And compute P 3 P 3 T P 3 P 3 T = (AR)(AR) T = ARR T A T = AA T à since A is a upper-triangular matrix, it can be found using Choleski factorization of P 3 P 3 T ( AA T ) 1 is called the image of the absolute conic. 37

26 Extraction of R, and T from P Once A is known: [R t] A 1 P No guarantee that R is a rotation matrix: R still must be ortho-normalized. 38

27 Ortho-Normalization of R Lets Q = A -1 P 3 a first approximation of R. No guarantee that Q is a rotation matrix. How to find the best rotation matrix that approximates Q? 1. Singular Value Decomposition of Q: Q = USV T, where S = diag(s 1, s 2, s 3 ) U and V have orthogonal columns, so that U T U = V T V = I 2. R = UV T [From Zhang TR98] 39

28 Ortho-Normalization of R: Proof Lets Q = A -1 P 3 a first approximation of R. No guarantee that Q is a rotation matrix. How to find the best rotation matrix that approximates Q? - Take best in the sense of the smallest Frobenius norm of the difference R Q. R Q F 2 min R = trace R Q R Q F Minimizing R - Q 2 is equivalent to maximize trace(r T Q)) Let the SVD of Q be USV T, where S = diag(s 1, s 2, s 3 ): with Z =V T R T U. Maximum reached with Z = I R = UV T 2 subject to RR T = I (( ) T ( R Q) ) (definition of Frobenius norm) = trace(r T R Q T R R T Q + Q T Q) = 3 + trace Q T Q trace(r T Q) ( ) 2trace( R T Q) = trace(r T USV) = trace(v T RUS) = trace(zs) = z ii s i s i i i 40 [From Zhang TR98]

29 DLT : Summary Not a good idea when the internal parameters are known. 41

30 P-n-P Problem PnP: Perspective-n-Point Estimation of the Position and Orientation (R and T) from 2D/3D point correspondences when the internal parameters (A) are known. m i M i R, T 42

31 P-n-P Problem PnP: Perspective-n-Point Estimation of the Position and Orientation (R and T) from 2D/3D point correspondences when the internal parameters (A) are known. 3 correspondences are sufficient (6 equations for 6 parameters), BUT yield up to 4 solutions, generally only two. A fourth correspondence is needed to remove the ambiguity. M i R, T 43

32 The P3P Problem M 3 M 1 M2 m 1 m 3 m 2 C P3P: Pose estimation from 3 correspondences. Yield up to 4 solutions, generally only two. A fourth correspondence will be needed to remove the ambiguity. 44

33 M 3 M 1 R, t M2 m 1 m 3 m 2 C

34 The P3P Problem M i d ij Mj m i θ ij m j x i C Each pair of correspondences M i m i and M j m j gives a constraint on the (unknown) camera-point distances x i = M i - C and x j = M j - C : d ij 2 = x i 2 + x j 2 2 x i x j cosθ ij, where: d ij = M i - M j is the (known) distance between M i and M j ; θ ij is the (also known) angle sustended at the camera center by M i and M j. 46

35 Overview of the P3P M i d ij Mj m i θ ij m j x i C 1. Solve for the distances x i ; 2. The positions M i C of the points M i in the camera coordinates system can be computed; 3. R and T are computed as the Euclidean displacement from the M i to the M i C.. 47

36 Computation of cosθ ij m i θ ij m j C cosθ ij must be computed. It depends only on the (known) 2D positions m i and m j. ( cosθ ij = Cm j ) T ( Cm i ) ( Cm j ) T ( Cm i ) = Cm j Cm i Cm i = A 1 m i cosθ ij = m j T ωm i ( m T j ωm j ) 1 2 ( m T i ωm i ) ( Cm j ) T ( Cm j ) ( Cm i ) T ( Cm i ) 1 2 with ω = AAT ( ) -1 the image of the absolute conic. 49

37 Quadratic System d ij 2 = x i 2 + x j 2 2 x i x j cosθ ij f ij (x i, x j ) = x i 2 + x j 2 2 x i x j cosθ ij - d ij 2 = 0 f 12 (x 1, x 2 ) = 0 f 13 (x 1, x 3 ) = 0 Resultant g(x 1,x 2 ) [degree 4] f 23 (x 2,x 3 ) = 0 Resultant h(x 1 ) [degree 8] h(x 1 ): Polynomial of degree 8 in x 1 with (fortunately) only even terms Degree polynomial of degree 4 in x=x 12 : At most 4 solutions for x; Can be solved in closed form. x 1 is positive and x 1 = x. x 2 and x 3 are uniquely determined from x 1. To obtain a unique solution, add one more point: Solve the P3P problem for each subset of 3 points, and keep the common solution. 51

38 Resultant of Two Polynomials Sylvester resultant (for two polynomials of a single unknown): p 1 (x) = a m x m + a m-1 x m a 1 x + a 0 p 2 (x) = b n x n + b n-1 x n b 1 x + b 0 Syl(p 1, p 2 ) = a m a a m a 0 a m a 0 b n b b n b 0 b n b 0 p 1 and p 2 have a common root iff determinant(syl(p 1, p 2 )) = 0. determinant(syl(p 1, p 2 )) is called the Sylvester resultant of p 1 and p 2. 52

39 Example: System of two polynomials of two unknowns p 1 (x, y) = 6x 2 + 3xy - xy 2 + y + 1 p 2 (x, y) = x 2 y + 5x + 4y - 1 p 1 (x, y) = 0 p 2 (x, y) = 0 Lets considers y as a constant, and write these two polynomials as polynomials in x: p 1 (x, y) = 6x 2 + (3y - y 2 )x + (y + 1) p 2 (x, y) = y x 2 + 5x + (4y - 1) 6 0 y (3y y 2 ) 6 5 (y +1) (3y y 2 ) (4y 1) 0 (y +1) 0 0 y 5 (4y 1) determinant(syl(p 1, p 2 )) = 4y 6-20y y 4-310y y y + 36 is a polynomial in y only! = 0 First, solve 4y 6-20y y 4-310y y y + 36 = 0 Once y is known, x can be found from p 1 (x, y) or p 2 (x, y). 54

40 Estimating R and T The coordinates of the 3D points M i in the camera coordinates system can now be computed: M i M i C = x i A 1 m i m i x i C 55

41 Estimating R and T The positions M i C of the points M i in the camera coordinates system can now be computed: M i W R,T? M i C n M W = 1 W M i N W i = M W i M W M C = 1 n n i=1 v t.( Ru) = trace R t uv t C ( ) ( N i ) t W.( RN i ) i max R rotation matrix i C ( N i ) t W. RN i = trace R t C N i i ( ) ( ) t.n i W R = argmax trace(r t L) R rotation matrix R = UV t where L = USV t T = M C RM W n i=1 M i C ) = trace R t L ( N i C = M i C M C C ( ) where L = ( N i ) t W.N i i 56

42 What are the possible solutions? Special case: the center of projection lies on the plane perpendicular to the triangle: C M 2 M 1 M 3 [From The Perspective View of Three Points, Wolfe et al. PAMI91] A second solution can be obtained by rotating the triangle about the side M 1 M 2 57

43 What are the possible solutions? General case: M 2 C M 1 58

44 What are the possible solutions? General case: M 3 M 3 M 2 C M 1 [From The Perspective View of Three Points, Wolfe et al. PAMI91] The set of circles generates a surface. Each intersection between the line of sight [CM 3 ) and this surface corresponds to one solution to the P3P problem. 59

45 P3P: Summary Well adapted to the problem Closed-form solution. Only 3 points + one constraint needed to compute the pose. Inconvenient: Pose computed with only 3 points, can be inaccurate. à Pose refined by a non-linear estimation with all the points available. 60

46 PnP Lu-Hager-Mjolsness: One on the most efficient algorithms in the general case. Iterative algorithm. Requires an initialization. Pick a starting R 0 Repeat Compute T(R k ) [can be computed in closed-form] Compute N i (R k ) Compute L(R k ) Compute L(R k ) = U D V t Set R k+1 = V U t Until convergence Can converge into a local minimum. 61

47 PnP algorithm: Application [Moreno-Noguer et al. ICCV07] 62

48 64 Pose Estimation from a Plane Let assume that the homography between the plane and the plane image is known. H m= Hm ku kv k = a b c d e f g h i u v 1 m = PM with M a point on to the plane = A R 1 R 2 R 3 T [ ] X Y 0 1 = A R 1 R 2 T [ ] X Y 1 = H X Y 1

49 Pose Estimation from a Plane We have: H A[ R 1 R 2 T] R 1, R 2 and T can be retrieved from the product G = A -1 H G is defined up to a scale factor and R 1 and R 2 are not perfectly orthonormal. Normalization : l = G 1 G 2 R 1 = G 1 l, R 2 = G 2 l,t = G 3 l R 2 c c = R 1 + R 2, p = R 1 R 2, d = c p R 1 = 1 2 c c + d d (, R 2 = 1 2 c c d d ( p R 1 R 3 = R 1 R 2 + Refinement with a non-linear optimization. d 65

50 Pose Estimation from a Plane: Application [Benhimane et al. CVPR06] 66

51 Non-Linear Optimization Non-linear least-squares minimization: min 2 A[R T]M i m i R,T i Minimization of a physical, meaningful error (reprojection error, in pixels) Handle an arbitrary number of points. Gauss-Newton, Levenberg Marquardt... minimization (next lecture). 67