Structure from Motion. CS4670/CS Kevin Matzen - April 15, 2016

Transcription

1 Structure from Motion CS4670/CS Kevin Matzen - April 15, 2016 Video credit: Agarwal, et. al. Building Rome in a Day, ICCV 2009

2 Roadmap What we ve seen so far Single view modeling (1 camera) Stereo modeling (2 cameras) Multi-view stereo (3+ cameras) How do we recover camera parameters necessary for MVS?

3 Wednesday s Lecture Assume we are always given the camera calibration. f1 T1 T2 f2 y x

4 Today s Lecture Assume we are always never given the camera calibration.???? y x

5 Calibration makes 3D reasoning possible! f1 x1 z x2 f2 1 2 b

6 Today s outline How can we calibrate our cameras? How can we calibrate a camera without photos of a calibration target? How can we automate this calibration at scale?

7 Projection Model

8 Projection Model Some 3D world-space point

9 Projection Model A 2D image-space projection Some 3D world-space point

10 Projection Model Calibration gives us these A 2D image-space projection Some 3D world-space point

11 Camera Calibration

12 Camera Calibration

13 Camera Calibration y (10, 12, 0) (0, 0, 0) x

14 DLT Method

15 DLT Method

16 DLT Method

17 DLT Method

18 Question: Is a single plane enough?

19 Question: Is a single plane enough? Assume plane is at Z = 0 (rotate and translate coordinates to make it so)

20 Question: Is a single plane enough? Columns are all 0 > Rank is at most 9 No, calibration target cannot be planar with DLT method. But we can combine many planes.

21

22 Non-Linear Method DLT method does not automatically give decomposition into extrinsics and intrinsics May wish to impose additional constraints on camera model (e.g. isotropic focal length, square pixels) Non-linearities such as radial distortion are not easily modeled with DLT

23 2 4 u i w i v i w i w i 3 5 = 2 4 f x 0 c x 0 f y c y r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z x i y i z i

24 2 4 u i w i v i w i w i 3 5 = 2 4 f x 0 c x 0 f y c y r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z x i y i z i D world-space point

25 2 4 u i w i v i w i w i 3 5 = 2 4 f x 0 c x 0 f y c y r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z x i y i z i Rotate and translate point into camera space 3D world-space point

26 2 4 u i w i v i w i w i 3 5 = 2 4 f x 0 c x 0 f y c y r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z x i y i z i Project point into image plane Rotate and translate point into camera space 3D world-space point

27 2 4 u i w i v i w i w i 3 5 = 2 4 f x 0 c x 0 f y c y r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z x i y i z i Project point into image plane Rotate and translate point into camera space 3D world-space point apple ui w i w i = Let s work through a simpler 2D version apple f c 0 1 apple cos( ) sin( ) tx sin( ) cos( ) t y 2 4 x i y i 1 3 5

28 apple ui w i w i = apple f c 0 1 apple cos( ) sin( ) tx sin( ) cos( ) t y 2 4 x i y i D point 1D projection

29 apple ui w i w i = apple f c 0 1 apple cos( ) sin( ) tx sin( ) cos( ) t y 2 4 x i y i apple ui w i w i = apple f c 0 1 apple cos( )xi sin( )y i + t x sin( )x i + cos( )y i + t y

30 apple ui w i w i = apple f c 0 1 apple cos( ) sin( ) tx sin( ) cos( ) t y 2 4 x i y i apple ui w i w i = apple f c 0 1 apple cos( )xi sin( )y i + t x sin( )x i + cos( )y i + t y u i w i w i = apple f(cos( )xi sin( )y i + t x )+c(sin( )x i + cos( )y i + t y ) sin( )x i + cos( )y i + t y

31 apple ui w i w i = apple f c 0 1 apple cos( ) sin( ) tx sin( ) cos( ) t y 2 4 x i y i apple ui w i w i = apple f c 0 1 apple cos( )xi sin( )y i + t x sin( )x i + cos( )y i + t y u i w i w i = apple f(cos( )xi sin( )y i + t x )+c(sin( )x i + cos( )y i + t y ) sin( )x i + cos( )y i + t y u i = f(cos( )x i sin( )y i + t x )+c(sin( )x i + cos( )y i + t y ) sin( )x i + cos( )y i + t y

32 h(f,c,,t x,t y,x i,y i )= f(cos( )x i sin( )y i + t x )+c(sin( )x i + cos( )y i + t y ) sin( )x i + cos( )y i + t y

33 h(f,c,,t x,t y,x i,y i )= f(cos( )x i sin( )y i + t x )+c(sin( )x i + cos( )y i + t y ) sin( )x i + cos( )y i + t y L(f,c,,t x,t y )= X i (u i h(f,c,,t x,t y,x i,y i )) 2

34 h(f,c,,t x,t y,x i,y i )= f(cos( )x i sin( )y i + t x )+c(sin( )x i + cos( )y i + t y ) sin( )x i + cos( )y i + t y L(f,c,,t x,t y )= X i (u i h(f,c,,t x,t y,x i,y i )) 2 argmin L(f,c,,t x,t y ) f,c,,t x,t y

35 h(f,c,,t x,t y,x i,y i )= f(cos( )x i sin( )y i + t x )+c(sin( )x i + cos( )y i + t y ) sin( )x i + cos( )y i + t y L(f,c,,t x,t y )= X i (u i h(f,c,,t x,t y,x i,y i )) 2 argmin L(f,c,,t x,t y ) f,c,,t x,t y Apply non-linear optimization method. @t y

36 What if we don t have a target?

37 What if we don t have a target? The world is our calibration target!

38 What if we don t have a target? The world is our calibration target! But we don t know the position of all points in the world.

39 Structure from Motion Key goals of SfM: Use approximate camera calibrations to match features and triangulate approximate 3D points Use approximate 3D points to improve approximate camera calibrations Chicken-and-egg problem Can extend and use our non-linear optimization framework Requires a good initialization

40 SfM building blocks What do we need from our CV toolbox? Keypoint detection Descriptor matching F-matrix estimation Ray triangulation Camera projection Non-linear optimization Useful metadata Focal length guess (EXIF tags)

41 Given: 1 2 Images 1 and 2 Focal length guesses

42 1. Compute feature 1 2 matches and F- matrix

43 2. Use approx K s 1 to get E-matrix 2 E = K2 T FK1

44 3. Decompose E 1 into relative pose 2 E = R[t]x

45 1 4. Triangulate features 2

46 1 5. Apply non-linear optimization 2

47 h(f,c,,t x,t y,x i,y i )= f(cos( )x i sin( )y i + t x )+c(sin( )x i + cos( )y i + t y ) sin( )x i + cos( )y i + t y L(f,c,,t x,t y )= X i (u i h(f,c,,t x,t y,x i,y i )) 2 argmin L(f,c,,t x,t y ) f,c,,t x,t y

48 h(f,c,,t x,t y,x i,y i )= f(cos( )x i sin( )y i + t x )+c(sin( )x i + cos( )y i + t y ) sin( )x i + cos( )y i + t y (f,c,,t x,t y, (x 1,y 1 ),...,(x n,y n )) = X i (u i h(f,c,,t x,t y,x i,y i )) 2 argmin f,c,,t x,t y,(x 1,y 1 ),...,(x n,y n ) L(f,c,,t x,t y, (x 1,y 1 ),...,(x n,y n )) Doesn t make sense for 1 camera

49 h(f,c,,t x,t y,x i,y i )= f(cos( )x i sin( )y i + t x )+c(sin( )x i + cos( )y i + t y ) sin( )x i + cos( )y i + t y L(K 1,...,K m, (x 1,y 1 ),...,(x n,y n )) = X i X j w i,j (u i h(k j, (x i,y i ))) 2 argmin K 1,...,K m,(x 1,y 1 ),...,(x n,y n ) L(K 1,...,K m, (x 1,y 1 ),...,(x n,y n )) Called Bundle Adjustment

50 Camera sets can be incrementally built up Essential matrix

51 Camera sets can be incrementally built up Perspective n Point method

52 Camera sets can be incrementally built up Perspective n Point method

53 Camera sets can be incrementally built up Perspective n Point method

54 Dubrovnik - Incremental Bundle Adjustment

55 Dubrovnik

56 Sacré-Cœur

57 SfM Ambiguities x = PX

58 SfM Ambiguities x = PX x =(PQ)(Q 1 X)

59 SfM Ambiguities x = PX x =(PQ)(Q 1 X) x = K(TQ)(Q 1 X)

60 SfM Ambiguities x = PX x =(PQ)(Q 1 X) x = K(TQ)(Q 1 X) T is a rigid body transformation If we want TQ to be a RBT, then Q could be an RBT > If we rotate and translate all the points, everything works out if we rotate and translate all the cameras.

61 SfM Ambiguities x = PX x =(PS 1 )(SX)

62 SfM Ambiguities x = PX x =(PS 1 )(SX) x = K(TS 1 )(SX)

63 SfM Ambiguities x = PX x =(PS 1 )(SX) x = K(TS 1 )(SX) x = K(S 1 TT 0 )(SX) x =(KS 1 )(TT 0 )(SX)

64 SfM Ambiguities x = PX x =(PS 1 )(SX) x = K(TS 1 )(SX) x = K(S 1 TT 0 )(SX) x =(KS 1 )(TT 0 )(SX) x =(S 1 K)(TT 0 )(SX) Sx = K(TT 0 )(SX)

65 SfM Ambiguities Sx = Sx = K(TT 0 )(SX) suw uw 4 svw 5 = 4 vw 5 = x sw w

66 SfM Ambiguities Sx = Sx = K(TT 0 )(SX) suw uw 4 svw 5 = 4 vw 5 = x sw w > If we scale all the points, everything works out if we move the camera positions.

67

68 SfM Ambiguities x = PX x =(PQ)(Q 1 X) In this case Q is a general similarity transform. We resolve the ambiguity often by placing one camera at the origin facing some direction and a second camera at fixed offset from the first.

69 Applications

70 Internet-scale 3D

71 Snavely, et. al. Finding Paths through the World's Photos. SIGGRAPH 2008.

72

73