Structure from Motion. Read Chapter 7 in Szeliski s book

Transcription

1 Structure from Motion Read Chapter 7 in Szeliski s book

2 Schedule (tentative) 2 # date topic 1 Sep.22 Introduction and geometry 2 Sep.29 Invariant features 3 Oct.6 Camera models and calibration 4 Oct.13 Multiple-view geometry 5 Oct.20 Model fitting (RANSAC, EM, ) 6 Oct.27 Stereo Matching 7 Nov.3 2D Shape Matching 8 Nov.10 Segmentation 9 Nov.17 Shape from Silhouettes 10 Nov.24 Structure from motion 11 Dec.1 Optical Flow 12 Dec.8 Tracking (Kalman, particle filter) 13 Dec.15 Object category recognition 14 Dec.22 Specific object recognition

3 Today s class Structure from motion factorization sequential bundle adjustment

4 Factorization Factorise observations in structure of the scene and motion/calibration of the camera Use all points in all images at the same time Affine factorisation Projective factorisation

5 Affine camera The affine projection equations are 1 = j j j y i x i ij ij Z Y X P P y x = j j j y i x i ij ij Z Y X P P y x ~ ~ 4 4 = = j j j y i x i ij ij y i ij x i ij Z Y X P P y x P y P x how to find the origin? or for that matter a 3D reference point? affine projection preserves center of gravity = i ij ij ij x x x~ = i ij ij ij y y y~

6 Orthographic factorization The orthographic projection equations are where n j m i j i ij,..., 1, 1,...,, M m = = = P All equations can be collected for all i and j where [ ] n m mn m m n n,...,m M,M,, m m m m m m m m m = = = M P P P P m m = PM M ~ ~ m = = = j j j j y i x i i ij ij ij Z Y X, P P, y x P Note that P and M are resp. 2mx3 and 3xn matrices and therefore the rank of m is at most 3 (Tomasi Kanade 92)

7 Orthographic factorization Factorize m through singular value decomposition An affine reconstruction is obtained as follows T V U m Σ = T V M U P Σ = = ~, ~ (Tomasi Kanade 92) [ ] n m mn m m n n M,..., M M, m m m m m m m m m min P P P Closest rank-3 approximation yields MLE!

8 0 ~ ~ 1 ~ ~ 1 ~ ~ = = = T T T T T T y i x i y i y i x i x i P P P P P P A A A A A A 0 ~ ~ 1 ~ ~ 1 ~ ~ = = = T T T y i x i y i y i x i x i P P P P P P C C C A metric reconstruction is obtained as follows Where A is computed from Orthographic factorization Factorize m through singular value decomposition An affine reconstruction is obtained as follows T V U m Σ = T V M U P Σ = = ~, ~ AM M PA P ~, ~ 1 = = = = = T T T y i x i y i y i x i x i P P P P P P 3 linear equations per view on symmetric matrix C (6DOF) A can be obtained from C through Cholesky factorisation and inversion (Tomasi Kanade 92)

9 Examples Tomasi Kanade 92, Poelman & Kanade 94

10 Examples Tomasi Kanade 92, Poelman & Kanade 94

11 Examples Tomasi Kanade 92, Poelman & Kanade 94

12 Examples Tomasi Kanade 92, Poelman & Kanade 94

13 Perspective factorization The camera equations for a fixed image i can be written in matrix form as where m λ ij mij = Pi M j, i = 1,..., m, j = 1,..., m i m i Λi = P M i [ m ] [ ] i1,mi2,...,mim, M 1, M2,..., Mm Λ = diag ( λ,λ,...,λ ) = M = i i1 i2 im

14 Perspective factorization All equations can be collected for all i as where m = PM = Λ Λ Λ = m n n P P P P m m m m..., In these formulas m are known, but Λ i,p and M are unknown Observe that PM is a product of a 3mx4 matrix and a 4xn matrix, i.e. it is a rank-4 matrix

15 Perspective factorization algorithm Assume that Λ i are known, then PM is known. Use the singular value decomposition PM=UΣ V T In the noise-free case Σ=diag(σ 1,σ 2,σ 3,σ 4,0,,0) and a reconstruction can be obtained by setting: P=the first four columns of UΣ. M=the first four rows of V.

16 Iterative perspective factorization When Λ i are unknown the following algorithm can be used: 1. Set λ ij =1 (affine approximation). 2. Factorize PM and obtain an estimate of P and M. If σ 5 is sufficiently small then STOP. 3. Use m, P and M to estimate Λ i from the camera equations (linearly) m i Λ i =P i M 4. Goto 2. In general the algorithm minimizes the proximity measure P(Λ,P,M)=σ 5 Note that structure and motion recovered up to an arbitrary projective transformation

17 Further Factorization work Factorization with uncertainty (Irani & Anandan, IJCV 02) Factorization for dynamic scenes (Costeira and Kanade 94) (Bregler et al. 00, Brand 01) (Yan and Pollefeys, 05/ 06)

18 juxtapose x, y coordinates for all points and cameras Multi-Camera Factorizations (Static) affine cameras Rigidly moving object Camera calibration using rigid motion 2D feature point trajectories as input No feature point correspondences between different camera views required (Angst & Pollefeys ICCV09) for all frames rank 13 all tracks of all affine cameras form rank 13 subspace!

19

20 Multi-Camera Factorizations Factorization technique 3 rd order tensor with rank 13 constraint (Angst & Pollefeys ICCV09) C : camera matrices M : motion matrix S : Shape matrix

21 Multi-Camera Factorizations Experiment on real data (Angst & Pollefeys ICCV09) Minimal cases extra camera: 1 point

22 practical structure and motion recovery from images Obtain reliable matches using matching or tracking and 2/3-view relations Compute initial structure and motion Refine structure and motion Auto-calibrate Refine metric structure and motion

23 Sequential Structure and Motion Computation Initialize Motion (P 1,P 2 compatibel with F) Initialize Structure (minimize reprojection error) Extend motion (compute pose through matches seen in 2 or more previous views) Extend structure (Initialize new structure, refine existing structure)

24 Computation of initial structure and motion according to Hartley and Zisserman this area is still to some extend a black-art All features not visible in all images No direct method (factorization not applicable) Build partial reconstructions and assemble (more views is more stable, but less corresp.) 1) Sequential structure and motion recovery 2) Hierarchical structure and motion recovery

25 Sequential structure and motion recovery Initialize structure and motion from two views For each additional view Determine pose Refine and extend structure Determine correspondences robustly by jointly estimating matches and epipolar geometry

26 Initial structure and motion Epipolar geometry Projective calibration m T 2 Fm1 = 0 P P 1 2 = = [ I 0] [ e] F ea e] T x + compatible with F Yields correct projective camera setup (Faugeras 92,Hartley 92) Obtain structure through triangulation Use reprojection error for minimization Avoid measurements in projective space

27 Determine pose towards existing structure M 2D-3D 2D-3D m i m i+1 2D-2D new view xi = Pi X(x1,..., xi 1) Compute Pi+1 using robust approach (6-point RANSAC) Extend and refine reconstruction

28 Compute P with 6-point RANSAC Generate hypothesis using 6 points Count inliers Projection error d X x,..., x ( Pi ( 1 i 1 ), xi ) < t? -1 3D error d ( x ), X < t Λ ( )? Pi i 3D Back-projection error d( F x, x ) < t?, j i ij i j < Re-projection error d( Pi X( x1,..., xi 1, xi ), xi ) < t Projection error with covariance d ( X( x,..., x ), x ) t Λ Pi 1 i 1 i < Expensive testing? Abort early if not promising Verify at random, abort if e.g. P(wrong)>0.95 (Chum and Matas, BMVC 02)

29 Calibrated structure from motion Equations more complicated, but less degeneracies For calibrated cameras: 5-point relative motion (5DOF) Nister CVPR03 3-point pose estimation (6DOF) Haralick et al. IJCV94 D. Nistér, An efficient solution to the five-point relative pose problem, In Proc. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2003), Volume 2, pages , R. Haralick, C. Lee, K. Ottenberg, M. Nolle. Review and Analysis of Solutions of the Three Point Perspective Pose Estimation Problem. Int l Journal of Computer Vision, 13, 3, , 1994.

30 5-point relative motion (Nister, CVPR03) Linear equations for 5 points Linear solution space scale does not matter, choose Non-linear constraints 10 cubic polynomials

31 5-point relative motion (Nister, CVPR03) Perform Gauss-Jordan elimination on polynomials represents polynomial of degree n in z -z -z -z

32 Three points perspective pose p3p (Haralick et al., IJCV94) All techniques yield 4 th order polynomial Haralick et al. recommends using Finsterwalder s technique as it yields the best results numerically

33 Minimal solvers Lot s of recent activity using Groebner bases: Henrik Stewénius, David Nistér, Fredrik Kahl, Frederik Schaffalitzky: A Minimal Solution for Relative Pose with Unknown Focal Length, CVPR H. Stewénius, D. Nistér, M. Oskarsson, and K. Åström. Solutions to minimal generalized relative pose problems. Omnivis D. Nistér, A Minimal solution to the generalised 3-point pose problem, CVPR 2004 Martin Bujnak, Zuzana Kukelova, Tomás Pajdla: A general solution to the P4P problem for camera with unknown focal length. CVPR Brian Clipp, Christopher Zach, Jan-Michael Frahm and Marc Pollefeys, A New Minimal Solution to the Relative Pose of a Calibrated Stereo Camera with Small Field of View Overlap, ICCV Zuzana Kukelova, Martin Bujnak, Tomás Pajdla: Automatic Generator of Minimal Problem Solvers. ECCV F. Fraundorfer, P. Tanskanen and M. Pollefeys. A Minimal Case Solution to the Calibrated Relative Pose Problem for the Case of Two Known Orientation Angles. ECCV 2010.

34 Minimal relative pose with know vertical Fraundorfer, Tanskanen and Pollefeys, ECCV2010 -g Vertical direction can often be estimated inertial sensor vanishing point 40 5 linear unknowns linear 5 point algorithm 3 unknowns quartic 3 point algorithm

35 Richard Szeliski Incremental Structure from Motion Photo Tourism Noah Snavely, Steve Seitz, Rick Szeliski University of Washington

36 Incremental structure from motion Automatically select an initial pair of images

37 Incremental structure from motion

38 Incremental structure from motion

39 Hierarchical structure and motion recovery Compute 2-view Compute 3-view Stitch 3-view reconstructions Merge and refine reconstruction F T H PM

40 Hierarchical structure from motion Farenzena, Fusiello, and Gherardi, Structure-and-motion Pipeline on a Hierarchical Cluster Tree, 3DIM 2009, October 2009 Ric ICVSS

41 Hierarchical structure from motion Ric ICVSS

42 Hierarchical structure from motion Ric ICVSS

43 Stitching 3-view reconstructions Different possibilities 1. Align (P 2,P 3 )with(p 1,P 2 ) -1-1 arg min d A( P2, P' 1 H ) + d A( P3, P' 2 H ) H 2. Align X,X (and C C ) arg min d A( X j, HX' j ) H j Minimize reproj. error arg min d( PH X' j, x j ) H j + d( P' HX j, x' j ) j 4. MLE (merge) arg min d( PX, x ) P,X j j j

44 More issues Repetition ambiguities Large scale reconstructions Marc Pollefeys ICVSS

45 Disambiguating visual relations using loop constraints (Zach et al CVPR 10) 51

46 Repetition and symmetry detection (Wu et al ECCV 10) 52

47 Video-only large-scale reconstruction? Challenge: Error accumulation yields drift of relative scale, orientation and position Solution: Cancel drift by closing loops (e.g. at intersections) Need to visually recognize locations 53

48 Solving 3D puzzles with VIPs SIFT features Extracted from 2D images Variation due to viewpoint VIP features (Wu et al., CVPR08) Extracted from 3D model Viewpoint invariant 54

49 55 am

50 Where am I? What am I looking at? Image query 3D Database SIFT VIP x Location recognition (Baatz et al., ECCV2010) 56

51 Projective ambiguity reconstruction from uncalibrated images only projective ambiguity on reconstruction m = 1 P M = ( PT )( T M) = P M Similarity (7DOF) Affine (12DOF) 57 Projective (15DOF)

52 Euclidean projection matrix Factorization of Euclidean projection matrix Intrinsics: K = f x f s y cx c y 1 (camera geometry) Extrinsics: ( R,t) (camera motion) Note: every projection matrix can be factorized, but only meaningful for euclidean projection matrices 58

53 The Absolute Quadric Eliminate extrinsics from equation Equivalent to projection of quadric Absolute Quadric A + B + C = 0 Absolute Quadric also exists in projective world KK T = PΩ * T 1 * T T T P = ( PT T )( TΩ = P Ω * P Transforming world so that reduces ambiguity to metric 59 T )( T * P ) Ω Ω AX + BY + CZ + D = * 0

54 Absolute quadric and self-calibration Projection equation: ω i = P Ω P = i T i K K Translate constraints on K through projection equation to constraints on Ω * i T i ω* Ω* Absolute Quadric = calibration object which is always present but can only be observed through constraints on the intrinsics 60

55 f K = Practical linear self-calibration y x s f cx c y 1 KK T Don t treat all constraints equal ˆ f ˆ * T 2 = PΩ P 0 f 0 after normalization! (relatively accurate for most cameras) (only rough aproximation, but still usefull to avoid degenerate configurations) when fixating point at image-center not only absolute quadric diag(1,1,1,0) satisfies ICCV98 eqs., but also diag(1,1,1,a), i.e. real or imaginary spheres! (Pollefeys et al., ECCV 02) ( T ) ( T PΩ P ) 11 PΩ P 1 ( T Ω ) 0.01 P P 12 = 0 1 ( T PΩ P ) 13 = ( T PΩ P ) = = ( T ) ( T PΩ P ) 11 PΩ P 33 = 0 ( T Ω ) ( T P P PΩ P ) = 0 Recent related work (Gurdjos et al. ICCV 09)

56 Upgrade from projective to metric Locate Ω * in projective reconstruction (self-calibration) Problem: not always unique (Critical Motion Sequences) Transform projective reconstruction to bring Ω * in canonical position 62

57 Refining structure and motion Minimize reprojection error min m n Pˆ k,mˆ i k = 1 i = 1 ( m,pˆ Mˆ ) Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) D ki Huge problem but can be solved efficiently (Bundle adjustment) k i 2

58 Non-linear least-squares X = f (P) Newton iteration argmin P X f (P) Levenberg-Marquardt Sparse Levenberg-Marquardt

59 Newton iteration Taylor approximation f (P + ) (P0 ) + J X f (P ) 1 0 f Jacobian J = X P X f (P 1) X f (P0 ) J = e0 J T J = J T e = 0 ( T ) -1 T J J J e0 J P i+ 1 = Pi + ( T ) -1 T = J J J e0 normal eq. = ( T -1 ) -1 T -1 J Σ J J Σ e0

60 Levenberg-Marquardt Normal equations J T J = N' = N = T J e 0 J T e Augmented normal equations 0 N' = J T J + λdiag(j T J) 3 λ 0 = 10 success : λi + 1 = λi /10 accept failure : λ = 10 solve again i λ i λ small ~ Newton (quadratic convergence) λ large ~ descent (guaranteed decrease)

61 Levenberg-Marquardt Requirements for minimization Function to compute f Start value P 0 Optionally, function to compute J (but numerical ok, too)

62 Bundle Adjustment What makes this non-linear minimization hard? many more parameters: potentially slow poorer conditioning (high correlation) potentially lots of outliers gauge (coordinate) freedom Richard Szeliski

63

64

65 Chained transformations Think of the optimization as a set of transformations along with their derivatives Richard Szeliski

66 Levenberg-Marquardt Iterative non-linear least squares [Press 92] Linearize measurement equations Substitute into log-likelihood equation: quadratic cost function in m Richard Szeliski

67 Robust error models Outlier rejection use robust penalty applied to each set of joint measurements for extremely bad data, use random sampling [RANSAC, Fischler & Bolles, CACM 81] Richard Szeliski

68 Levenberg-Marquardt Iterative non-linear least squares [Press 92] Solve for minimum Hessian: error: Richard Szeliski

69 Chained transformations Only some derivatives are non-zero: j th camera, i th point Richard Szeliski

70 Multi-camera rig Easy extension of generalized bundler: Richard Szeliski

71 Lots of parameters: sparsity Only a few entries in Jacobian are non-zero Richard Szeliski

72 Exploiting sparsity Schur complement to get reduced camera matrix Richard Szeliski

73 Sparse Levenberg-Marquardt N 3 complexity for solving prohibitive for large problems = N' -1 J T e0 (100 views 10,000 points ~30,000 unknowns) Partition parameters partition A partition B (only dependent on A and itself)

74 Sparse bundle adjustment residuals: normal equations: with note: tie points should be in partition A

75 Sparse bundle adjustment normal equations: modified normal equations: solve in two parts:

76 Sparse bundle adjustment Jacobian of m n k = 1 i= 1 D ( m, Pˆ ( Mˆ ) ki k i 2 has sparse block structure im.pts. view 1 P 1 P 2 P 3 M T J = N = J J = U 1 U 2 W U 3 12xm 3xn (in general much larger) W T Needed for non-linear minimization V

77 Sparse bundle adjustment Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m I 0 WV I 1 N = U-WV -1 W T Allows much more efficient computations e.g. 100 views,10000 points, solve ±1000x1000, not ±30000x30000 Often still band diagonal use sparse linear algebra algorithms W T 11xm V 3xn

78 Sparse bundle adjustment normal equations: modified normal equations: solve in two parts:

79 Sparse bundle adjustment Covariance estimation a b = = ( -1 U WV W) Y T a Y + V Y = WV ab = - a Y

80 Direct vs. iterative solvers How do they work? When should you use them? Richard Szeliski

81 Large reduced camera matrices Use iterative preconditioned conjugate gradient [Jeong, Nister, Steedly, Szeliski, Kweon, CVPR 2009] [Agarwal, Snavely, Seitz, and Szeliski, ECCV 2009] Block diagonal works well Sometimes partial Cholesky does too Direct solvers for small problems Richard Szeliski

82 Richard Szeliski Large-scale structure from motion

83 Large-scale reconstruction Most of the models shown so far have had ~500 images How do we scale from 100s to 10,000s of images? Observation: Internet collections represent very non-uniform samplings of viewpoint Richard Szeliski

84 Skeletal graphs for efficient structure from motion Noah Snavely, Steve Seitz, Richard Szeliski CVPR 2008

85 Skeletal set Goal: select a smaller set of images to reconstruct, without sacrificing quality of reconstruction Two problems: How do we measure quality? How do we find a subset of images with bounded quality loss?

86 Skeletal set Goal: given an image graph, select a small set S of important images to reconstruct, bounding the loss in quality of the reconstruction Reconstruct the skeletal set S Estimate the remaining images with much faster pose estimation steps

87 Properties of the skeletal set Should touch all parts of Dominating set Should form a single reconstruction Connected dominating set Should result in an accurate reconstruction?

88 Example Full graph Skeletal graph

89 Representing information in a graph What kind of information? No absolute information about camera positions Each edge provides information about the relative positions of two images but not all edges are equally informative We model information with the uncertainty (covariance) in pairwise camera positions

90 Estimating certainty Usually measured with covariance matrix SfM has thousands or millions of parameters We only measure covariance in camera positions (3x3 matrix for every camera)

91 Pantheon Full graph Skeletal graph (t=16)

92 Pantheon Skeletal reconstruction 101 images After adding leaves 579 images After final optimization 579 images

93 Trafalgar Square 2973 images registered (277 in skeletal set)

94 Trafalgar Square

95 Running time 180 (~10 days) (~30 days) hours Full reconstruction Skeletal reconstruction Skeletal reconstruction + BA Stonehenge St. Peters Pantheon Pisa Trafalgar

96 Building Rome in a Day Sameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz, Richard Szeliski ICCV 2009

97 Distributed matching pipeline

98 Query expansion

99 Results: Dubrovnik

100

101 Results: Rome

102 Changchang Wu s SfM code for iconic graph uses 5-point+RANSAC for 2-view initialization uses 3-point+RANSAC for adding views performs bundle adjustment For additional images use 3-point+RANSAC pose estimation

103 Rome on a cloudless day GIST & clustering (1h35) (Frahm et al. ECCV 2010) Dense Reconstruction (1h58) SIFT & Geometric verification (11h36) Some numbers 1PC SfM & Bundle (8h35) 2.88M images 100k clusters 22k SfM with 307k images 63k 3D models Largest model 5700 images Total time 23h53

104 Structure from motion: limitations Very difficult to reliably estimate metric structure and motion unless: large (x or y) rotation or large field of view and depth variation Camera calibration important for Euclidean reconstructions Need good feature tracker

105 Related problems On-line structure from motion and SLaM (Simultaneous Localization and Mapping) Kalman filter (linear) Particle filters (non-linear)

106 Visual SLAM Real-time Stereo Visual SLAM (Clipp et al., IROS2010) 113

107 Open challenges Large scale structure from motion Complete building Complete city

108 Open source vode available, e.g. Christopher Zach (SfM, Bundle adjustment) Photo Tourism Bundler:

109 Next week: Optical Flow 11