Camera Calibration (Trucco, Chapter 6) What is the goal of camera calibration? -Toproduce an estimate of the extrinsic and intrinsic camera parameters. Procedure -Given the correspondences beteen a set of point features in the orld (X, Y, Z ) and their projections in an image (x im, y im ), compute the intrinsic and extrinsic camera parameters. x im pixel frame P y im Camera Frame Yc X c Z c center of perspective projection optical axis y o x,oy image plane frame x principal point World Frame Y X Z Establishing the correspondences -Calibration methods rely on one or more images of a calibration pattern: () a 3D object of knon geometry. (2) it is located in a knon position in space. (3) it is generating image features hich can be located accurately.
-2- -Consider the above calibration pattern: *itconsists of to orthogonal grids. *equally spaced black squares dran on hite, perpendicular planes. *assume that the orld reference frame is centered at the loer left corner of the left grid, ith axes parallel to the three directions identified by the calibration pattern. * given the size of the planes, their angle, the number of squares etc. (all knon by construction), the coordinates of each vertex can be computed in the orld reference frame using trigonometry. *the projection of the vertices on the image can be found by intersecting the edge lines of the corresponding square sides (or through corner detection).
-3- Methods () Direct parameter calibration. Direct recovery of the intrinsic and extrinsic camera parameters. (2) Camera parameters through the projection matrix M = M in M ex = m m 2 m 3 m 2 m 22 m 32 m 3 m 23 m 33 m 4 m 24 m 34 (2.) Estimate the elements of the projection matrix. (2.2) Compute the intrinsic/extrinsic as closed-form functions of the entries of the projection matrix.
-4- Method : Direct Parameter Calibration -Weassume that the orld reference frame is knon (e.g., the origin is the middle loer corner of the calibration pattern). Revie of basic equations -From orld coordinates to camera coordinates (note that e have changed the order of rotation/translation): P c = R(P T )or P c = RP RT or P c = RP T -Inthe rest of this discussion, I ill replace T ith T : X c Y c Z c = r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 X Y Z + T x T y T z -From camera coordinates to pixel coordinates: x im = x/s x + o x = f s x X c Z c + o x y im = y/s y + o y = f s y Y c Z c + o y -Relating orld coordinates to pixel coordinates: x im o x = f /s x r X + r 2 Y + r 3 Z + T x r 3 X + r 32 Y + r 33 Z + T z y im o y = f /s y r 2 X + r 22 Y + r 23 Z + T y r 3 X + r 32 Y + r 33 Z + T z
Independent intrinsic parameters -5- -The five intrinsic parameters f, s x, s y, o x, o y are not independent. -Wecan define the folloing four independent parameters: f x = f /s x,the focal length in horizontal pixels = s y /s x (or = f x / f y ), aspect ratio (o x, o y ), image center coordinates Method : main steps () Assuming that o x and o y are knon, estimate all the remaining parameters. (2) Estimate o x and o y Step : estimate f x,, R, and T -Tosimplify notation, consider (x im o x, y im o y ) = (x, y) x = f x r X + r 2 Y + r 3 Z + T x r 3 X + r 32 Y + r 33 Z + T z y = f y r 2 X + r 22 Y + r 23 Z + T y r 3 X + r 32 Y + r 33 Z + T z -Using the fact that the above to equations have the same denominator, e get the folloing equation: x f y (r 2 X + r 22 Y + r 23 Z + T y ) = y f x (r X + r 2 Y + r 3 Z + T x ) Problem Statement Assuming that o x and o y are knon, compute f x,, R, and T from N corresponding pairs of points (Xi, Yi, Zi ), (x i, y i ), i =,..., N.
-6- Der ive a system of equations -Each pair of corresponding points leads to an equation: x i f y (r 2 X i + r 22 Y i + r 23 Z i + T y ) = y i f x (r X i + r 2 Y i + r 3 Z i + T x ) -Rerite the above equation as follos (i.e., divide by f y ): x i X i v + x i Y i v 2 + x i Z i v 3 + x i v 4 y i X i v 5 y i Y i v 6 y i Z i v 7 y i v 8 = here v = r 2 v 5 = r v 2 = r 22 v 6 = r 2 v 3 = r 23 v 7 = r 3 v 4 = T y v 8 = T x - N corresponding points lead to a homogeneous system of N equations ith 8 unknons: Av = here: A = x X x 2 X2 x N X N x Y x 2 Y2 x N Y N x Z x 2 Z2 x N Z N x x 2 x N y X y 2 X 2 y N X N y Y y 2 Y 2 y N Y N y Z y 2 Z 2 y N Z N y y 2 y N Solving the system -Itcan be shon that if N 7, then A has rank 7. -If A = UDV T,ehave discussed in class that the system has a nontrivial solution v hich is proportional to the column of V corresponding to the smallest singular value of A (i.e., the last column of V hich e denote as v): v = v ( is the scale factor) or v = v ( = / ) -Using the components of v and v: (v, v 2, v 3, v 4, v 5, v 6, v 7, v 8 ) = (r 2, r 22, r 23, T y, r, r 2, r 3, T x )
-7- Deter mine and v 2 + v2 2 + v2 3 = 2 (r2 2 + r2 22 + r2 23 ) = (r 2 2 + r 2 22 + r 2 23 = ) v 2 5 + v2 6 + v2 7 = 2 2 (r 2 + r2 2 + r2 3 ) = (r 2 + r 2 2 + r 2 3 = and >) Deter mine r 2, r 22, r 23, r, r 2, r 3, T y, T x -Wecan determine the above parameters, up to an unknon common sign. r 2 = / v r = / v 5 r 22 = / v 2 r 2 = / v 6 r 23 = / v 3 r 3 = / v 7 T y = / v 4 T x = / v 8 Deter mine r 3, r 32, r 33 -Can be estimated as the cross product of R and R 2 : R 3 = R xr 2 -The sign of R 3 is already fixed (the entries of R 3 remain unchanged if the signs of all the entries of R and R 2 are reversed). Ensur ing the orthogonality of R -The computation of R does not take into account explicitly the orthogonality constraints. -The estimate ˆR of R cannot be expected to be orthogonal (e.g., ˆR ˆR T = I). -Wecan "enforce" the orthogonality on ˆR by using its SVD: ˆR = UDV T -Replace D ith I, e.g., ˆR =UIV T ( ˆR ˆR T = I)
-8- Deter mine the sign of -Consider the folloing equations again: x = f /s x r X + r 2 Y + r 3 Z + T x r 3 X + r 32 Y + r 33 Z + T z = f /s x X c Z c y = f /s y r 2 X + r 22 Y + r 23 Z + T y r 3 X + r 32 r + r 33 Z + T z = f /s y Y c Z c -If Z c >, then x and r X + r 2 Y + r 3 Z + T x must have opposite signs (it is sufficient to check the sign for one of the points). if x(r X + r 2 Y + r 3 Z + T x )>,then reverse the signs of r, r 2, r 3,and T x else no further action is required -Similarly, if Z c >, then y and r 2 X + r 22 Y + r 23 Z + T x must have opposite signs (it is sufficient to check the sign for one of the points). if y(r 2 X + r 22 Y + r 23 Z + T y )>,then reverse the signs of r 2, r 22, r 23,and T y else no further action is required
-9- Deter mine T z and f x : -Consider the equation: -Let s rerite it in the form: x = f /s x r X + r 2 Y + r 3 Z + T x r 3 X + r 32 Y + r 33 Z + T z x(r 3 X + r 32 Y + r 33 Z + T z ) = f /s x (r X + r 2 Y + r 3 Z + T x ) -Wecan obtain T z and f x by solving a system of equations like the above, ritten for N points: A T z f x = b here A = x x 2 x N (r X + r 2 Y + r 3 Z + T x ) x (r 3 X + r 32 Y + r 33 Z + T x ) (r X2 + r 2 Y2 + r 3 Z2 + T x ) x 2 (r 3 X2 + r 32 Y2 + r 33 Z2 + T x ) b = (r X N + r 2 Y N + r 3 Z N + T x ) x N (r 3 X N + r 32 Y N + r 33 Z N + T x ) -Using SVD, the (least-squares) solution is: T z = (A T A) A T b f x Deter mine f y : -From f x = f /s x and f y = f /s y e have: f y = f x /
-- Step 2: estimate o x and o y -The computation of o x and o y ill be based on the folloing theorem: Orthocenter Theorem: Let T be the triangle on the image plane defined by the three vanishing points of three mutually orthogonal sets of parallel lines in space. The image center (o x, o y )isthe orthocenter of T. -Wecan use the same calibration pattern to compute three vanishing points (use three pairs of parallel lines defined by the sides of the planes). Note : it is important that the calibration pattern is imaged from a viepoint guaranteeing that none of the three mutually orthogonal directions ill be near parallel to the image plane! Note 2: to improve the accuracy ofthe image center computation, it is a good idea to estimate the center using several vies of the calibration pattern and average the results.
-- Method 2: Camera parameters through the projection matrix Revie of basic equations x h y h = M in M ex X Y Z = M X Y Z = m m 2 m 3 m 2 m 22 m 32 m 3 m 23 m 33 m 4 m 24 m 34 X Y Z x = x h = m X + m 2 Y + m 3 Z + m 4 m 3 X + m 32 Y + m 33 Z + m 34 y = y h = m 2 X + m 22 Y + m 23 Z + m 24 m 3 X + m 32 Y + m 33 Z + m 34 (Note: Ihave replaced x im ith x and y im ith y for simplicity) Step : solve for m ij s -The matrix M has independent entries (e.g., divide every entry by m ). -We ould need at least N =6 orld-image point correspondences to solve for the entries of M. m X i m 2 X i + m 2 Y i + m 3 Z i + m 4 m 3 x i X i m 32 x i Y i m 33 x i Z i + m 34 = + m 22 Y i + m 23 Z i + m 24 m 3 y i X i m 32 y i Y i m 33 y i Z i + m 34 = -These equations ill lead to a homogeneous system of equations: Am = here A = X X 2 X N Y Y2 Y N Z Z2 Z N X X2 X N Y Y2 Y N Z Z2 Z N x X y X x 2 X 2 y 2 X 2 x N X N y N X N x Y y Y x 2 Y 2 y 2 Y 2 x N Y N y N Y N x Z y Z x 2 Z 2 y 2 Z 2 x N Z N y N Z N x y x 2 y 2 x N y N
-2- -Itcan be shon that A has rank (for N ). -If A = UDV T,the system has a nontrivial solution m hich is proportional to the column of V corresponding to the smallest singular value of A (i.e., the last column of V denoted here as m): m = m ( is the scale factor) or m = m ( = / ) Step 2: find the intrinsic/extrinsic parameters using m ij s -The full expression for M is as follos: f x r + o x r 3 M = f y r 2 + o y r 3 r 3 f x r 2 + o x r 32 f y r 22 + o y r 32 r 32 f x r 3 + o x r 33 f y r 23 + o y r 33 r 33 f x T x + o x T z f y T y + o y T z T z -Let s define the folloing vectors: q = (m, m 2, m 3 ) T q 2 = (m 2, m 22, m 23 ) T q 3 = (m 3, m 32, m 33 ) T q 4 = (m 4, m 24, m 34 ) T -The solutions are as follos (see book for details): o x = q T q 3 o y = q T 2 q 3 f x = q T q o 2 x f y = q T 2 q 2 o 2 y -The rest parameters are easily computed. Question: ho ould you estimate the accuracy ofacalibration algorithm?
-3- Some comments -The precision of calibration depends on ho accurately the orld and image points are located. -Studying ho localization errors "propagate" to the estimates of the camera parameters is very important. -Although the to methods described here should produce the same results (at least theoretically), e usually obtain different solutions due to different error propagations. -Method 2 is simpler and should be preferred if e do not need to compute the intrinsic/extrinsic camera parameters explicitly.