3D from Photographs: Camera Calibration. Dr Francesco Banterle

Transcription

1 3D from Photographs: Camera Calibration Dr Francesco Banterle

2 3D from Photographs Automatic Matching of Images Camera Calibration Photographs Surface Reconstruction Dense Matching 3D model

3 3D from Photographs Automatic Matching of Images Camera Calibration Photographs Surface Reconstruction Dense Matching 3D model

4 Camera Pre-Calibration

5 Camera Model: Pinhole Camera Y c X c Image C Z c Plane Hole Y w X w f image-space Z w world-space

6 Camera Model: Pinhole Camera The perspective projection is defined as m = P M m 0 = m/m z P = K[I 0]G = K[R t] K = fk u 0 u 0 0 fk v v t = 2 3 t R = t 2 t 3 2 r > 1 4r > 2 r > Intrinsic Matrix Extrinsic Matrix

7 Pre-Calibration In some cases, when we know the camera, it is useful to avoid intrinsics matrix estimation: It is more precise. We reduce computations.

8 Pre-Calibration: Why? In some cases, when we know the camera, it is useful to avoid intrinsics matrix estimation: It is more precise. We reduce computations.

9 DLT: Direct Linear Transform

10 DLT: Direct Linear Transform Input: a photograph of a non-coplanar calibration with m 2D points with known 3D coordinates. Output: K of the camera.

11 DLT: Direct Linear Transform Y Z X

12 DLT: Direct Linear Transform P = 2 p > 1 4p > < u = p> 1 M p > 3 M p > 3 : v = p> 2 M p > 3 M

13 DLT: Direct Linear Transform ( p > M 1 i u i p > M 3 i =0 p > 2 M i v i p > 3 M i =0 m i =[u i,v i, 1] > $ M i =[x, y, z, 1] > 2D-3D matches

14 DLT: Direct Linear Transform This leads to a matrix: apple M > i 0 u i M > i 0 M > i v i M > i 2 4 p 1 p 2 p = 0 For each point, we need to stack this equations obtaining a matrix A.

15 DLT: Direct Linear Transform We obtain a 2m 12 linear system to solve. The minimum number of points to solve it is 6, but more points are required to have robust and stable solutions.

16 What s the problem with this method?

17 DLT: Direct Linear Transform DLT minimizes an algebraic error! It does not have geometric meaning!! Hang on, is it all wrong? Nope, we can use it as input for a non-linear method.

18 DLT: Non-linear Refinement The non-linear refinement minimizes (at least squares) the distance between 2D points of the image (mi) and projected 3D points (Mi): arg min P mx p > 1 M i i=1 p > 3 M i u i 2 + p > 2 M i p > 3 M i v i 2 Different methods for solving it; e.g., Nelder-Mead s method (MATLAB s fminsearch).

19 Now we have a nice matrix P

20 DLT: Direct Linear Transform Let s recap: K has to be upper-triangular. R is orthogonal. P = K[R t] =[K R K t] =[P 0 p 4 ]

21 DLT: Direct Linear Transform QR decomposition: A = O T where O is orthogonal and T is upper-triangular. In our case, we have: P 0 = K R! (P 0 ) 1 = R 1 K 1

22 DLT: Direct Linear Transform QR decomposition to P : [P 0 ] QR = O T In our case, we have: R = O 1 K = T 1 Note that there is a scale factor!

23 DLT: Direct Linear Transform The scale factor is due to the fact we do not know if are taking a shot to a large object from afar or to a small object in front of the camera! Case 1 Case 2

24 DLT: Direct Linear Transform It makes sense to fix the scale in K because R has to be an orthogonal matrix! This affects also t!

25 How do we compute t?

26 How do we compute t? t = K 1 p 4

27 The Sanity Check If we can have an estimation of K from camera parameters that are available in the camera specifications. K = 2 3 a 0 u 0 4a b v

28 The Sanity Check What do we need? Focal length of the camera in mm ( f ). Resolution of the picture in pixels ( w, h ). CCD/CMOS sensor size in mm ( ws, hs ).

29 The Sanity Check a = ( f w ) / ws. b = ( f h ) / hs. u0 = w / 2. v0 = h / 2.

30 The Sanity Check a = ( f w ) / ws. b = ( f h ) / hs. u0 = w / 2. v0 = h / 2. Assuming it in the center!

31 and what s about the radial distortion?

32 Estimating Radial Distortion Let s start with simple radial distortion (i.e., only a coefficient): ( u 0 =(u u 0 ) (1 + k 1 rd 2)+u 0 r 2 d = ( Can we solve it? v 0 =(v v 0 ) (1 + k 1 r 2 d )+v (u u0 ) + u 2 (v v0 ) u = f k u u = f k v v

33 Estimating Radial Distortion We have only one unknown, which is linear; i.e., k1: ( u 0 u (u u 0 ) rd 2 v 0 v (v v 0 ) rd 2 = k 1 = k 1 In theory, a single point is enough, but it is better to use more points to get a more robust solution.

34 Homography

35 2D Transformations We can have different type of transformation (defined by a matrix) of 2D points: Translation (2 degree of freedom [DoF]): It preserves orientation. Rigid/Euclidian (3 DoF); translation, and rotation: It preserves lengths. Similarity (4DoF); translation, rotation, and scaling: It preserves angles.

36 2D Transformations Affine (6 degree of freedom [DoF]): It reserves parallelism. Projective (8 DoF): It preserves straight lines.

37 2D Transformations: Homography Homography is defined as m = H M m = This is typically expressed as 2 4 x 0 y = 2 3 m 1 /m 3 4m 2 /m m H M where H is a 3 3 non-singular matrix with 8 DoF.

38 2D Transformations: Homography H M m

39 Homography Estimation m H M m = 2 4 x 0 y M = 2 3 x 4y5 1

40 Homography Estimation m H M m = 2 4 x 0 y M = 2 3 x 4y5 1

41 Homography Estimation m H M m = 2 4 x 0 y M = 2 3 x 4y5 1 x 0 = h 11x 1 + h 12 y 1 + h 13 h 31 x 1 + h 32 y 1 + h 33 y 0 = h 21x 1 + h 22 y 1 + h 23 h 31 x 1 + h 32 y 1 + h 33

42 Homography Estimation (h 31 x 1 + h 32 y 1 + h 33 ) x 0 (h 11 x 1 + h 12 y 1 + h 33 )=0 (h 31 x 1 + h 32 y 1 + h 33 ) y 0 (h 21 x 1 + h 22 y 1 + h 23 )=0

43 Homography Estimation (h 31 x 1 + h 32 y 1 + h 33 ) x 0 (h 11 x 1 + h 12 y 1 + h 33 )=0 (h 31 x 1 + h 32 y 1 + h 33 ) y 0 (h 21 x 1 + h 22 y 1 + h 23 )=0 Stacking multiple equations; one for each match (at least 5!)

44 Homography Estimation (h 31 x 1 + h 32 y 1 + h 33 ) x 0 (h 11 x 1 + h 12 y 1 + h 33 )=0 (h 31 x 1 + h 32 y 1 + h 33 ) y 0 (h 21 x 1 + h 22 y 1 + h 23 )=0 Stacking multiple equations; one for each match (at least 5!) A vec(h) =0 A is 2n 9

45 Homography Estimation Again, we have minimized an algebraic error!! Technically speaking, we should run a non-linear optimization.

46 Zhang s Algorithm

47 Zhang s Algorithm Input: a set of n photographs of a checkboard or other patterns. From these, we have to extract m points in each photograph! Output: K of the camera, and [R t] for each photographs.

48 Zhang s Algorithm A set of input images

49 Zhang s Algorithm 2 3 K = 4 fk u 0 u 0 0 fk v v

50 Zhang s Algorithm 2 3 K = 4 c u 0 0 v

51 Zhang s Algorithm 2 3 K = 4 c u 0 0 v

52 Zhang s Algorithm Assumption: We have a set of photographs of a plane so Z is equal 0. So we have 3D points defined as M = 2 3 x 6y

53 Zhang s Algorithm m =P M = =K [R t] 2 3 x 6y =

54 Zhang s Algorithm m 2 3 x =K [r 1 r 2 r 3 t] 6y = x =K [r 1 r 2 t] 4y5 1

55 Zhang s Algorithm m 2 3 x =K [r 1 r 2 r 3 t] 6y = x =K [r 1 r 2 t] 4y5 1

56 Zhang s Algorithm m 2 3 x =K [r 1 r 2 r 3 t] 6y = x =K [r 1 r 2 t] 4y5 1 H = K [r 1 r 2 t]

57 Zhang s Algorithm m 2 3 x =K [r 1 r 2 r 3 t] 6y = x =K [r 1 r 2 t] 4y5 1 It is a homography! H = K [r 1 r 2 t]

58 Zhang s Algorithm m 2 3 x =K [r 1 r 2 r 3 t] 6y = x =K [r 1 r 2 t] 4y5 1 It is a homography! H = K [r 1 r 2 t] H = h 1 h 2 h 3

59 Zhang s Algorithm Now that we know that we need homographies! What to do? For each photograph we compute the homography H between photographed checkerboard corners and its model.

60 Zhang s Algorithm H Model Photograph

61 Zhang s Algorithm Given that r1 and r2 are orthonormal, we have that: h > 1 K > K 1 h 2 =0 h > 1 K > K 1 h 1 = h > 2 K > K 1 h 2

62 Zhang s Algorithm B = K > K 1 = 1 α 2 c α 2 β c α 2 β cv 0 u 0 β α 2 β c 2 α 2 β β 2 c(cv 0 u 0 β) α 2 β 2 v 0 β 2 cv 0 u 0 β α 2 β c(cv 0 u 0 β) α 2 β 2 v 0 β 2 (cv 0 u 0 β) 2 α 2 β 2 + v2 0 β 2 +1 B is symmetric > defined only by six values: b =[B 1,1,B 1,2,B 2,2,B 1,3,B 2,3,B 3,3 ] >

63 Zhang s Algorithm B = K > K 1 = 1 α 2 c α 2 β c α 2 β cv 0 u 0 β α 2 β c 2 α 2 β β 2 c(cv 0 u 0 β) α 2 β 2 v 0 β 2 cv 0 u 0 β α 2 β c(cv 0 u 0 β) α 2 β 2 v 0 β 2 (cv 0 u 0 β) 2 α 2 β 2 + v2 0 β 2 +1 B is symmetric > defined only by six values: b =[B 1,1,B 1,2,B 2,2,B 1,3,B 2,3,B 3,3 ] >

64 Zhang s Algorithm h > i B h j = v > i,j b

65 Zhang s Algorithm h > i B h j = v > i,j b H = h 1 h 2 h 3 h i = h i1 h i2 h i3 > v ij = h i1 h j1 h i1 h j2 + h i2 h j1 h i2 h j2 h i3 h j1 + h i1 h j3 h i3 h j2 + h i2 h j3 h i3 h j

66 Zhang s Algorithm Given that r1 and r2 are orthonormal, we have that: h > 1 K > K 1 h 2 =0 h > 1 K > K 1 h 1 = h > 2 K > K 1 h 2 [ ] v T 12 (v 11 v 22 ) T b = 0

67 Zhang s Algorithm If n images of the model plane are observed, by stacking n of such equations: [ ] v T 12 (v 11 v 22 ) T b = 0 We obtain: V b = 0 V is 2n 6 matrix, so we need n > 2

68 Zhang s Algorithm At this point, we can compute elements of K as v 0 =(B 12 B 13 B 11 B 23 )/(B 11 B 22 B 2 12) λ = B 33 [B v 0 (B 12 B 13 B 11 B 23 )]/B 11 α = λ/b 11 β = λb 11 /(B 11 B 22 B12 2 ) c = B 12 α 2 β/λ u 0 = cv 0 /α B 13 α 2 /λ. 3 3

69 Zhang s Algorithm: Camera Pose Furthermore, we can extract the pose as r 1 = K 1 h 1 r 2 = K 1 h 2 r 3 = r 1 r 2 t = K 1 h 3

70 Zhang s Algorithm: Non-Linear Refinement So far, we have obtained a solution through minimizing an algebraic distance that is not physically meaningful. From that solution, we can use a non-linear method for minimizing the following error: nx mx km i,j m(k, R i, t i, M j )k 2 i=1 j=1

71 Zhang s Algorithm: Non-Linear Refinement So far, we have obtained a solution through minimizing an algebraic distance that is not physically meaningful. From that solution, we can use a non-linear method for minimizing the following error: nx mx i=1 j=1 km i,j m(k, R i, t i, M j )k 2 This is a function projecting Mj points given intrinsics and the pose!

72 Zhang s Algorithm: Optical Distortion What s about the parameters for modeling the radial distortion? As before, first algebraic solution, and then a nonlinear solution.

73 Zhang s Algorithm: Optical Distortion apple (u u0 )r 2 d (u u 0 )r 4 d (v v 0 )r 2 d (v v 0 )r 4 d apple k1 k 2 = apple u 0 v 0 u v D k = d k =(D > D) 1 D > d

74 Zhang s Algorithm: Non-Linear Refinement We extend the previous non-linear model to include optical distortion: nx mx km i,j m(k, R i, t i, k, M j )k 2 i=1 j=1

75 Zhang s Algorithm: Non-Linear Refinement We extend the previous non-linear model to include optical distortion: nx mx i=1 j=1 km i,j m(k, R i, t i, k, M j )k 2 This is a function projecting Mj points given intrinsics and the pose!

76 that s all folks!