Robust Camera Location Estimation by Convex Programming

Transcription

1 Robust Camera Location Estimation by Convex Programming Onur Özyeşil and Amit Singer INTECH Investment Management LLC 1 PACM and Department of Mathematics, Princeton University SIAM IS /24/2016, Albuquerque NM 1 Work conducted as part of Ozyesil s Ph.D. work at Princeton University O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

2 Table of contents 1 Structure from Motion (SfM) Problem 2 The Abstract Problem 3 Well-posedness of the Location Estimation Problem 4 Robust Location Estimation from Pairwise Directions 5 Robust Pairwise Direction Estimation 6 Experimental Results O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

3 SfM Problem Structure from Motion (SfM) Problem Given a collection of 2D photos of a 3D object, recover the 3D structure by estimating the camera motion, i.e. camera locations and orientations? 3D Structure Camera Locations = We are primarily interested in the camera location estimation part O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

4 SfM Problem Structure from Motion Classical Approach Find corresponding points between images, estimate relative poses Estimate camera orientations and locations, i.e. camera motion Estimate the 3D structure (e.g., by reprojection error minimization) O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

5 SfM Problem Structure from Motion Classical Approach Find corresponding points between images, estimate relative poses Estimate camera orientations and locations, i.e. camera motion Estimate the 3D structure (e.g., by reprojection error minimization) Previous Methods Incremental methods: Incorporate images one by one (or in small groups) to maintain efficiency prone to accumulation of errors Joint structure and motion estimation: Computationally hard, usually non-convex methods, no guarantees of convergence to global optima Orientation estimation methods: Relatively stable and efficient solvers O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

6 SfM Problem Structure from Motion Classical Approach Find corresponding points between images, estimate relative poses Estimate camera orientations and locations, i.e. camera motion Estimate the 3D structure (e.g., by reprojection error minimization) Previous Methods Incremental methods: Incorporate images one by one (or in small groups) to maintain efficiency prone to accumulation of errors Joint structure and motion estimation: Computationally hard, usually non-convex methods, no guarantees of convergence to global optima Orientation estimation methods: Relatively stable and efficient solvers Global Location Estimation Ill-conditioned problem (because of undetermined relative scales) Current methods: Usually not well-formulated, not stable, inefficient O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

7 The Abstract Problem Problem: Location Estimation from Pairwise Directions Estimate the locations t 1, t 2,..., t n R d, for arbitrary d 2, from a subset of (noisy) measurements of the pairwise directions, where the direction between t i and t j is given by the unit norm vector γ ij : γ ij = t i t j t i t j Pairwise Directions Locations t 1 γ 12 γ 15 t 2 γ 12 γ 13 γ 24 γ15 γ 25 γ 46 γ 45 γ 36 t 5 γ 25 t γ γ γ 46 A (noiseless) instance in R 3, with n = 6 locations and m = 8 pairwise directions. t 4 γ 24 γ 13 t 3 O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

8 Well-posedness Well-posedness of the Location Estimation Problem We represent the total pairwise information using a graph G t = (V t, E t ) and endow each edge (i, j) with the direction measurement γ ij O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

9 Well-posedness Well-posedness of the Location Estimation Problem We represent the total pairwise information using a graph G t = (V t, E t ) and endow each edge (i, j) with the direction measurement γ ij Fundamental Questions Is the problem well-posed, i.e. do we have enough information to estimate the locations {t i} i Vt stably (or, exactly in the noiseless case)? O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

10 Well-posedness Well-posedness of the Location Estimation Problem We represent the total pairwise information using a graph G t = (V t, E t ) and endow each edge (i, j) with the direction measurement γ ij Fundamental Questions Is the problem well-posed, i.e. do we have enough information to estimate the locations {t i} i Vt stably (or, exactly in the noiseless case)? What does well-posedness depend on, is it a (generic) property of G t? O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

11 Well-posedness Well-posedness of the Location Estimation Problem We represent the total pairwise information using a graph G t = (V t, E t ) and endow each edge (i, j) with the direction measurement γ ij Fundamental Questions Is the problem well-posed, i.e. do we have enough information to estimate the locations {t i} i Vt stably (or, exactly in the noiseless case)? What does well-posedness depend on, is it a (generic) property of G t? Can it be decided efficiently? O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

12 Well-posedness Well-posedness of the Location Estimation Problem We represent the total pairwise information using a graph G t = (V t, E t ) and endow each edge (i, j) with the direction measurement γ ij Fundamental Questions Is the problem well-posed, i.e. do we have enough information to estimate the locations {t i} i Vt stably (or, exactly in the noiseless case)? What does well-posedness depend on, is it a (generic) property of G t? Can it be decided efficiently? What can we do if an instance is not well-posed? O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

13 Well-posedness Well-posedness of the Location Estimation Problem We represent the total pairwise information using a graph G t = (V t, E t ) and endow each edge (i, j) with the direction measurement γ ij Fundamental Questions Is the problem well-posed, i.e. do we have enough information to estimate the locations {t i} i Vt stably (or, exactly in the noiseless case)? What does well-posedness depend on, is it a (generic) property of G t? Can it be decided efficiently? What can we do if an instance is not well-posed? O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

14 Well-posedness Well-posedness of the Location Estimation Problem We represent the total pairwise information using a graph G t = (V t, E t ) and endow each edge (i, j) with the direction measurement γ ij Fundamental Questions Is the problem well-posed, i.e. do we have enough information to estimate the locations {t i} i Vt stably (or, exactly in the noiseless case)? What does well-posedness depend on, is it a (generic) property of G t? Can it be decided efficiently? What can we do if an instance is not well-posed? Well-posedness was previously studied in various contexts, under the general title of parallel rigidity theory. O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

15 Well-posedness Well-posedness: Simple Examples Consider the following noiseless instance: G t = (V t, E t ) { ij } (i,j)2et O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

16 Well-posedness Well-posedness: Simple Examples Consider the following noiseless instance: t 1 t 2 t 3 G t = (V t, E t ) { ij } (i,j)2et t 4 t 5 O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

17 Well-posedness Well-posedness: Simple Examples Consider the following noiseless instance: G t = (V t, E t ) { ij } (i,j)2et 23 t 1 t 2 t 3 t 4 t 5 t 1 t 2 t 3 t 4 t 5 O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

18 Well-posedness Well-posedness: Simple Examples Consider the following noiseless instance: G t = (V t, E t ) { ij } (i,j)2et 23 t 1 t 2 t 3 t 4 t 5 t 1 t 2 t 3 t 4 t 5 Well-posedness depends on the dimension: t 1 t 2 Well-posed in R 3, but not in R 2 t 3 t 4 O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

19 Well-posedness Main Results of Parallel Rigidity Theory Unique Realization of Locations Unique solution exists if and only if the formation is parallel rigid. O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

20 Well-posedness Main Results of Parallel Rigidity Theory Unique Realization of Locations Unique solution exists if and only if the formation is parallel rigid. Rigidity is Generic Parallel rigidity is a generic property, i.e. it is a function of G t only. O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

21 Well-posedness Main Results of Parallel Rigidity Theory Unique Realization of Locations Unique solution exists if and only if the formation is parallel rigid. Rigidity is Generic Parallel rigidity is a generic property, i.e. it is a function of G t only. Theorem (Efficient Decidability, Whiteley, 1987) For a graph G = (V, E), let (d 1)E denote the set consisting of (d 1) copies of each edge in E. Then, G is generically parallel rigid in R d if and only if there exists a nonempty set D (d 1)E, with D = d V (d + 1), such that for all subsets D of D, D d V (D ) (d + 1), where V (D ) denotes the vertex set of the edges in D. O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

22 Well-posedness Main Results of Parallel Rigidity Theory Unique Realization of Locations Unique solution exists if and only if the formation is parallel rigid. Rigidity is Generic Parallel rigidity is a generic property, i.e. it is a function of G t only. Theorem (Efficient Decidability, Whiteley, 1987) For a graph G = (V, E), let (d 1)E denote the set consisting of (d 1) copies of each edge in E. Then, G is generically parallel rigid in R d if and only if there exists a nonempty set D (d 1)E, with D = d V (d + 1), such that for all subsets D of D, D d V (D ) (d + 1), where V (D ) denotes the vertex set of the edges in D. Maximally Parallel Rigid Components If G t is not parallel rigid, we can efficiently find maximally parallel rigid subgraphs. O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

23 Robust Location Estimation Robust Location Estimation from Pairwise Directions Objective Minimize the effects of direction measurements with large errors, i.e. outlier directions, in location estimation, by maintaining computational efficiency O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

24 Robust Location Estimation Robust Location Estimation from Pairwise Directions Objective Minimize the effects of direction measurements with large errors, i.e. outlier directions, in location estimation, by maintaining computational efficiency Recipe Formulation based on a robust cost function, approximate by convex programming O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

25 Robust Location Estimation Robust Location Estimation from Pairwise Directions Objective Minimize the effects of direction measurements with large errors, i.e. outlier directions, in location estimation, by maintaining computational efficiency Recipe Formulation based on a robust cost function, approximate by convex programming Linearization of Pairwise Directions γ ij = ti tj t i t + j ɛγ ij ɛ t ij = t i t j d ijγ ij where d ij = t i t j and ɛ t ij = t i t j ɛ γ ij O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

26 Robust Location Estimation Robust Location Estimation from Pairwise Directions Objective Minimize the effects of direction measurements with large errors, i.e. outlier directions, in location estimation, by maintaining computational efficiency Recipe Formulation based on a robust cost function, approximate by convex programming Linearization of Pairwise Directions γ ij = ti tj t i t + j ɛγ ij ɛ t ij = t i t j d ijγ ij where d ij = t i t j and ɛ t ij = t i t j ɛ γ ij Idea Large ɛ γ ij induces large ɛt ij exchange robustness to ɛ γ ij with robustness to ɛt ij O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

27 Robust Location Estimation Least Unsquared Deviations (LUD) formulation Relax the non-convex constraint d ij = t i t j to obtain: minimize {t i} i Vt R d {d ij} (i,j) Et subject to (i,j) E t t i t j d ij γ ij i V t t i = 0 ; d ij c, (i, j) E t Inspired by convex programs developed for robust signal recovery in the presence of outliers, exact signal recovery from partially corrupted data Prevents collapsing solutions via the constraint d ij c O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

28 Robust Location Estimation IRLS for LUD Initialize: w 0 ij = 1, (i, j) E t for r = 0, 1,... do Compute {ˆt r+1 r+1 i }, { ˆd ij } by solving the QP: { minimize} wij r t i t j d ij γ ij 2 ti=0, d (i,j) E ij 1 t ( ) w r+1 ij ˆt r+1 i ˆt r+1 r+1 2 1/2 j ˆd ij γ ij + δ O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

29 Robust Direction Estimation Robust Pairwise Direction Estimation Pinhole Camera Model P p i p j t i t j For a camera C i = (R i, t i, f i) and P R 3 : Represent P in i th coordinate system: P i = Ri T (P t i) = (Pi x, P y i, P i z ) T Project onto i th image plane: q i = (f i/pi z )(Pi x, P y i )T R 2 O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

30 Robust Direction Estimation Robust Pairwise Direction Estimation Pinhole Camera Model t i p i P p j Pairwise Directions from Epipolar Constraints t j For a pair (C i, C j ) with given (R i, R j ), (f i, f j ) and a set {(q k i, qk j )}m ij k=1 of corresponding points: Fact: P, t i, t j are coplanar (equiv. to epipolar constraint) [(P t i ) (P t j )] T (t i t j ) = 0 [ ] (R i ηi k R jηj k )T (t i t j ) = 0 (for ηi k.= q k i /f i ) 1 For a camera C i = (R i, t i, f i) and P R 3 : Represent P in i th coordinate system: P i = R T i (P t i) = (P x i, P y i, P z i ) T Project onto i th image plane: q i = (f i/p z i )(P x i, P y i )T R 2 γ 0 ij ν 5 ij ν 4 ij ν 3 ij ν 1 ij ν 2 ij (ν k ij )T (t i t j ) = 0, (for ν k ij.= R iη k i R jη k j R i η k i R jη k j ) ν 6 ij ν 7 ij O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

31 Robust Direction Estimation Robust Directions from Noisy 2D Subspace Samples Given noisy R i s, f i s and q k i s, we essentially obtain noisy samples ˆνk ij s from the 2D subspace orthogonal to t i t j O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

32 Robust Direction Estimation Robust Directions from Noisy 2D Subspace Samples Given noisy R i s, f i s and q k i s, we essentially obtain noisy samples ˆνk ij s from the 2D subspace orthogonal to t i t j To maintain robustness to outliers among ˆν ij k s, we estimate the lines γij 0 using the non-convex problem (solved via a heuristic IRLS method): minimize γ 0 ij R3 m ij γij 0, ˆνk ij k=1 subject to γ 0 ij = 1 O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

33 Robust Direction Estimation Robust Directions from Noisy 2D Subspace Samples Given noisy R i s, f i s and q k i s, we essentially obtain noisy samples ˆνk ij s from the 2D subspace orthogonal to t i t j To maintain robustness to outliers among ˆν ij k s, we estimate the lines γij 0 using the non-convex problem (solved via a heuristic IRLS method): minimize γ 0 ij R3 m ij γij 0, ˆνk ij k=1 subject to γ 0 ij = 1 The heuristic IRLS method is not guaranteed to converge (because of non-convexity) However, we empirically observed high quality line estimates Computationally much more efficient (relative to previous methods with similar accuracy) Pairwise directions are computed from the lines using the fact that the 3D points should lie in front of the cameras. [WS14] K. Wilson and N. Snavely, Robust global translations with 1DSfM, ECCV Madrid Metropolis Piazza del Popolo Notre Dame Vienna Cathedral Robust Directions PCA Directions Histogram plots of direction errors as compared to simpler PCA method (for datasets from [WS14]) O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

34 Experiments Syncthetic Data Experiments Measurement Model Measurement graphs G t = (V t, E t ) are random graphs drawn from Erdös-Rényi model, i.e. each (i, j) is in the edge set E t with probability q, independently of all other edges. Noise model: Given a set of locations {t i } n i=1 Rd and G t = (V t, E t ), for each (i, j) E t, we let γ ij = γ ij / γ ij, where { γij U, w.p. p γ ij = (t i t j )/ t i t j + σγij G w.p. 1 p Here, {γ U ij } (i,j) E t are i.i.d. Unif(S d 1 ), {γ G ij } (i,j) E t and t i s are i.i.d. N (0, I 3 ). O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

35 Experiments Synthetic Data: Relatively High Robustness to Outliers Performance Measure Normalized root mean square error (NRMSE) of the estimates ˆt i w.r.t. the original locations t i (after the removal of global scale and translation, and for t 0 denoting the center of t i s.) i NRMSE({ˆt i }) = ˆt i t i 2 i t i t 0 2 O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

36 Experiments Synthetic Data: Relatively High Robustness to Outliers Performance Measure Normalized root mean square error (NRMSE) of the estimates ˆt i w.r.t. the original locations t i (after the removal of global scale and translation, and for t 0 denoting the center of t i s.) i NRMSE({ˆt i }) = ˆt i t i 2 i t i t 0 2 Methods for comparison: Least Squares (LS) method [AKK12] M. Arie-Nachimson, S. Kovalsky, I. Kemelmacher-Shlizerman, AS, and R. Basri, Global motion estimation from point matches, 3DimPVT, [BAT04] M. Brand, M. Antone, and S. Teller, Spectral solution of large-scale extrinsic camera calibration as a graph embedding problem, ECCV, Constrained Least Squares (CLS) method [TV14] R. Tron and R. Vidal, Distributed 3-D localization of camera sensor networks from 2-D image measurements, IEEE Trans. on Auto. Cont., Semidefinite Relaxation (SDR) method [OSB15] OÖ, AS, and R. Basri, Stable camera motion estimation using convex programming, SIAM J. Imaging Sci., 8(2): , NRMSE NRMSE q = 0.25, p = 0 LUD CLS SDR LS σ q = 0.1, σ = 0 LUD CLS SDR LS p q = 0.25, p = 0.05 LUD CLS SDR LS σ q = 0.25, σ = 0 LUD CLS SDR LS p q = 0.25, p = 0.2 LUD CLS SDR LS σ 1 q = 0.5, σ = 0 LUD CLS SDR LS p O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

37 Experiments Exact Recovery with Partially Corrupted Directions Empirical observation: LUD can recover locations exactly with partially corrupted data 0.5 d = 2, n = d = 2, n = p p q q d = 3, n = d = 3, n = p p q q The color intensity of each pixel represents log 10 (NRMSE), depending on the edge probability q (x-axis), and the outlier probability p (y-axis). NRMSE values are averaged over 10 trials. O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

38 Experiments Real Data Experiments: Internet Photo Collections We tested our methods on internet photo collections from [WS14] and observed that: The LUD solver is more robust to outliers as compared to existing methods, and is more efficient as compared to methods with similar accuracy The robust direction estimation by IRLS further improves robustness to outliers Snapshots of selected 3D structures computed using the LUD solver (before bundle adjustment) O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

39 Experiments Real Datasets: Comparison of Estimation Accuracy and Efficiency Estimation errors in meters (Bundler sequential SfM [SSS06] is taken as ground truth): Name Dataset LUD CLS [TV14] SDR [OSB15] 1DSfM [WS14] LS [G01] Size Initial After BA Initial After BA Initial After BA Init. After BA After BA PCA Robust Robust Robust Robust Robust Robust m Nc ẽ ê ẽ ê Nc ẽ ê ẽ ê Nc ẽ ê ẽ ê Nc ẽ ê ẽ Nc ẽ ê Nc ẽ Piazza del Popolo NYC Library Metropolis Yorkminster Tower of London Montreal N. D Notre Dame Alamo e Vienna Cathedral e Running times in seconds: LUD CLS [TV14] SDR [OSB15] 1DSfM [WS14] [G01] [SSS06] Dataset T R T G T γ T t T BA T tot T t T BA T tot T t T BA T tot T R T γ T t T BA T tot T tot T tot Piazza del Popolo NYC Library Metropolis Yorkminster Tower of London Montreal N. D Notre Dame Alamo Vienna Cathedral [SSS06] N. Snavely, S. M. Seitz, and R. Szeliski, Photo tourism: exploring photo collections in 3D, SIGGRAPH, [G01] V. M. Govindu, Combining two-view constraints for motion estimation, CVPR, O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

40 Experiments Real Data Experiments: 3D Surface Reconstructions Sample 3D Reconstructions O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20

41 Acknowledgements THANK YOU!!! Acknowledgements: O. Ozyesil and A. Singer Robust Camera Location Estimation SIAM IS / 20