arxiv: v1 [cs.cv] 4 Apr 2019

Transcription

1 Algebraic Characterization of Essential Matrices and heir Averaging in Multiview Settings Yoni Kasten* Amnon Geifman* Meirav Galun Ronen Basri Weizmann Institute of Science arxiv: v1 [cscv 4 Apr 019 Abstract Essential matrix averaging, ie, the tas of recovering camera locations and orientations in calibrated, multiview settings, is a first step in global approaches to Euclidean structure from motion A common approach to essential matrix averaging is to separately solve for camera orientations and subsequently for camera positions his paper presents a novel approach that solves simultaneously for both camera orientations and positions We offer a complete characterization of the algebraic conditions that enable a unique Euclidean reconstruction of n cameras from a collection of ( n ) essential matrices We next use these conditions to formulate essential matrix averaging as a constrained optimization problem, allowing us to recover a consistent set of essential matrices given a (possibly partial) set of measured essential matrices computed independently for pairs of images We finally use the recovered essential matrices to determine the global positions and orientations of the n cameras We test our method on common SfM datasets, demonstrating high accuracy while maintaining efficiency and robustness, compared to existing methods 1 Introduction What algebraic conditions mae a collection of ( n ) essential matrices consistent, in the sense that there exist n Euclidean camera matrices that produce them? his fundamental question has not yet been answered in the literature It is well nown that ( 3 ) = 3 fundamental matrices are consistent if, and only if, the epipole of the third view is transferred correctly between each pair of views, ie, for every 1 i,j, 3, e i F ije j = 0 Recent wor [1 presented a set of sufficient and necessary algebraic conditions that mae ( n ) fundamental matrices in general position consistent One could expect that essential matrices that fulfill those same conditions would be consistent with respect to Euclidean camera matrices However, these conditions are not enough and can be contradicted 1 Establishing consistency constraints for essential matrices is an important step toward producing essential matrix averaging algorithms GivennimagesI 1,,I n, a common approach for global Structure from Motion (SfM) begins by robustly estimating essential matrices between pairs of views, {E ij }, from which an estimate of the relative pairwise rotations {R ij } and translations {t ij } are extracted Motion averaging then is performed typically in two steps: first the absolute camera orientations{r i } are solved by averaging the relative rotations hen, using the relative translations and the recovered absolute orientations, the absolute camera positions {t i } are recovered Finally, the obtained solution is refined by bundle adjustment Our goal in this paper is to establish a complete set of necessary and sufficient conditions for the consistency of essential matrices and to use these conditions to formulate a one-step algorithm for averaging essential matrices o achieve this goal we investigate an object called the n-views essential matrix, which is obtained by stacing the ( n ) essential matrices into a 3n 3n matrix whose i,j th 3 3 bloc is the essential matrixe ij relating thei th and thej th frames We prove that, in addition to projective consistency (introduced in [1), this matrix must have three pairs of eigenvalues each of the same magnitude but opposite signs, and its eigenvectors directly encode camera parameters We use these results to introduce an essential matrix averaging algorithm Given a noisy estimate of a subset of ( ) n essential matrices, our algorithm sees to find the nearest consistent set of essential matrices We formulate this problem as constrained optimization and solve it using ADMM We then incorporate this algorithm in a global SfM pipeline and evaluate our pipeline on the datasets of [, showing superior accuracies relative to state of the art methods on almost all image collections while also maintaining efficiency *Equal contributors 1 A counter example is provided in the supplementary material 1

2 Related wor Approaches for Euclidean motion averaging can be divided into two main categories: Incremental methods [13, 17, 1, 11, 3 begin with a small subset of frames and produce an initial reconstruction he rest of cameras are then used sequentially for reconstruction hese methods are very successful and quite robust However, they have to apply bundle adjustment refinement at every step to prevent camera drift Consequently, these method are computationally demanding when applied to large data sets Global methods [,, 7, 10, 15, in contrast, recover the motion parameters simultaneously for all the frames ypical global SfM pipelines proceed by applying a camera orientation solver, followed by a location solver Global orientation solvers [, 14, 1, 8, 4 solve for the absolute orientations of the cameras given relative rotation measurements between pairs of images [, 14 derive closed form solutions that minimize a least squares objective constructed from the pairwise relative orientations hese methods are very efficient but due to the relaxed orthonormality requirement, the result is usually suboptimal Other methods [1, 8, 4 utilize the Lie algebra structure of the rotation group to perform rotation averaging in SO(3) hese methods, however, often converge to local minima Recently, [6 has used the strong duality principle to find the global minimum under certain conditions Global location solvers [,, 15 assume nown camera orientations and solve for the absolute positions of the cameras by using the noisy relative translations [ uses point correspondences to find a least squares solution for the absolute positions [ formulates a highly non convex objective and solves for the absolute translations utilizing the Levenberg-Marquet algorithm with random initialization [15 uses a similar objective while formulating a convex relaxation [5 uses linear global method that minimizes geometric error in triplet of views while considering feature tracs All the aforementioned methods are highly dependent on accurate estimation of the absolute rotations of the cameras, which result from a rotation averaging method Integrative Methods: [19 formulates the problem of fundamental and essential averaging as global optimization through minimizing the epipolar transfer error While their method manages to improve the consistency of viewing graphs, it is unable to generate a consistent reconstruction, and so it requires postprocessing steps of both rotation and translation averaging [16 has introduced the concept of a multi-view essential/fundamental matrix heir wor, however, established only a partial list of constraints Moreover, their use of a complicated non-convex objective allowed them to only refine a complete reconstruction [1 has introduced a complete set of consistency conditions for fundamental matrices hey formulate a robust optimization objective and demonstrate state of the art projective reconstructions heir method, however, is limited to projective settings and is inapplicable to calibrated settings, ie, for Euclidean reconstruction [1 suggested a method that optimizes first for camera positions and then for their orientations, and as a post processing simultaneously optimizes for both However, this method is sensitive to outliers Recent wor explored the properties of the manifold of essential matrices [0 heir characterization, however, is suitable only for a single essential matrix and not for general multiview settings Our paper extends the wor of [1, 16 by introducing a complete set of necessary and sufficient conditions for consistency of multiview essential matrices and by proposing an efficient and robust optimization algorithm for essential matrix averaging that incorporates these conditions 3 heory Let I 1,,I n denote a collection of n images of a static scene captured respectively by cameras P 1,,P n Each camera P i is represented by a 3 4 matrix P i = K i Ri [I, t i where K i is a 3 3calibration matrix, t i R 3 and R i SO(3) denote the position and orientation of P i, respectively, in some global coordinate system We further denote V i = K i Ri, so P i = V i [I, t i Consequently, let X = (X,Y,Z) be a scene point in the global coordinate system Its projection onto I i is given by x i = X i /Z i, wherex i = (X i,y i,z i ) = K i Ri (X t i) We denote the fundamental matrix and the essential matrix between images I i and I j by F ij and E ij respectively It was shown in [ thate ij andf ij can be written as E ij = R i ( i j )R j F ij = Ki E ij Kj 1 = V i ( i j )Vj where i = [t i hroughout this paper we assume that all calibration matrices are nown, so our wor deals with solving the problem of Euclidean SfM he derivations in this paper adopt the definitions nview fundamental matrix and consistent n-view fundamental matrix from [1 We first repeat these definitions, for the sae of clarity, and then define analogous definitions for the calibrated case In the definitions below we denote the space of all the3n 3n symmetric matrices bys 3n Definition 1 A matrix F S 3n, whose 3 3 blocs are denoted byf ij, is called an n-view fundamental matrix if i j {1,,n},ran(F ij ) = and if ii = 0 Definition An n-view fundamental matrix F is called consistent if there exist camera matrices P 1,,P n of the formp i = V i [I,t i such thatf ij = V i ([t i [t j )Vj

3 Definition 3 A matrix E S 3n, whose 3 3 blocs are denoted bye ij, is called an n-view essential matrix if i j {1n}, ran(e ij ) =, the two singular values of E ij are equal, and ie ii = 0 Definition 4 An n-view essential matrix E is called consistent if there existnrotation matrices{r i } n i=1 andnvectors {t i } n i=1 such thate ij = R i ([t i [t j )R j Note that any (consistent) n-view essential matrix is also a (consistent) n-view fundamental matrix In [1 necessary and sufficient conditions for the consistency of the n- view fundamental matrix were proved he main theoretical contribution of [1 is summarized in heorem 1 For the consistency of n-view essential matrix, a partial set of necessary conditions were derived in [16 hose are summarized below in heorem heorem 1 An n-view fundamental matrix F is consistent with a set of n cameras whose centers are not all collinear if, and only if, the following conditions hold: 1 Ran(F) = 6 andf has exactly 3 positive and 3 negative eigenvalues Ran(F i ) = 3 for all i = 1,,n, where F i is the 3 3ni th bloc row of F heorem Let E be a consistent n-view essential matrix, associated with rotation matrices{r i } n i=1 and camera centers{t i } n i=1 E satisfies the following conditions 1 E can be formulated as A+A wherea = UV andu,v R 3n 3 V = R 1 R n U = with i = [t i and n i=1 t i = 0 R 1 1 R n n Each column of U is orthogonal to each column of V, ie,v U = ran(v)=3 4 If not all {t i } n i=1 are collinear, then ran(u) and ran(a) = 3 Moreover, if the (thin) SVD of A is A = Û V ˆ, with Û, ˆV R 3n 3 and R 3 3 then the (thin) SVD ofe is implying ran(e) = 6 Û 31 Main theoretical results In this section we derive and prove necessary and sufficient conditions for the consistency of n-view essential matrices in terms of their spectral decomposition hese conditions, in turn, will be used later to formulate a constrained optimization problem and to extract the motion parameters from a consistent n-view essential matrix E heorem 3 LetE S 3n be a consistentn-view fundamental matrix with a set ofncameras whose centers are not all collinear We denote by +, R 3 3 the diagonal matrices with the 3 positive and 3 negative eigenvalues of E, respectively hen, the following conditions are equivalent: 1 E is a consistentn-view essential matrix he (thin) SVD of E can be written in the form [Û, ˆV ( )[ + ˆV + Û with Û, ˆV R 3n 3 such that each 3 3 bloc of ˆV, ˆV i, i = 1,,n, is an n-scaled rotation matrix, ie, ˆV i = 1 ˆRi n, where ˆR i SO(3) We say that ˆV is a bloc rotation matrix 3 + = and the (thin) spectral decomposition of E is of the form ( )[ + X [X,Y Y such that 05(X +Y) is a bloc rotation matrix Proof (1) () Assume that E is a consistent n-view essential matrix hen, according to hm, A + A witha = UV andu,v R 3n 3 with V = R 1 R n U = R 1 1 R n n where i = [t i SinceA = UV andran(a) = 3, then A A = VU UV anda A 0 with ran(a A) = 3 (A and A A share the same null space) First, we construct a spectral decomposition toa A, relying on the special properties of U and V We have ran(u) = 3, and therefore U U, which is a 3 3, symmetric positive semi-definite matrix, is of full ran Its spectral decomposition is of the form U U = PDP, where P SO(3) (Spectral decomposition guarantees that P O(3) However, P can be replaced by P if det(p) = 1) D R 3 3 is a diagonal matrix consisting of the (positive) eigenvalues ofu U his spectral decomposition yields the following decomposition A A = VPDP V (1) 3

4 Now, note that R 1 P V VP = P [ R 1 R n R n = P (ni 3 3 )P = ni 3 3 P By a simple manipulation (1) becomes a spectral decomposition ( ) ( ) 1 1 A A = n V P(nD)P n V () On the other hand, the (thin) SVD of A is of the form A = ÛˆV, whereû, ˆV R 3n 3, R 3 3 his means that A A = ˆV ˆV (3) Due to the uniqueness of the eigenvector decomposition, () and (3) collapse to the same eigenvector decomposition, up to some global rotation,h SO(3), that is 1 n VP = ˆVH, which means that ˆV i = 1 n R i PH Since R i,p,h SO(3), then ˆR i =: R i PH SO(3), showing that ˆV is a bloc rotation matrix Finally, by hm, the (thin) SVD ofe is of the form Û and according to Lemma 5, the eigenvalues of E are and Since the elements on the diagonal of are positive, ande is symmetric with exactly 3 positive eigenvalues + and 3 negative eigenvalues, it follows that = + and = concluding the proof () (1) Let E be a consistent n-view fundamental matrix that satisfies condition () We would lie to show that E is a consistent n-view essential matrix By condition () E can be written as (4) Û +ˆV + ˆV + Û = Ū ˆV + ˆV U, (5) where Ū = Û + with ˆV i = 1 n ˆRi, ˆR i SO(3) By definitione ii = 0, and this implies thatūiˆv i is sew symmetric Using Lemma 4, Ūi = ˆV iˆi for some sew symmetric matrix ˆ i = [ˆt i PluggingŪi and ˆV i in (5) yields E ij = ŪiˆV j + ˆV i Ū j = 1 n ˆR iˆi ˆR j 1 n ˆR iˆj ˆR j = R i ([t i [t j )R j, wherer i = ˆR i andt i = nˆt 1 i, concluding the proof () (3) Let E be an n-view fundamental matrix which satisfies condition () his means that the Û (thin) SVD of E can be written in the form [Û, ˆV ( )[ + ˆV, where ˆV is a bloc rota- + tion matrix hen, by Lemma 5, ( the (thin) spectral )[ decomposition of E is [X,Y + X + Y, where X, Y are the eigenvectors of E satisfying X = 05( Û + ˆV) and Y = 05(ˆV Û) Since ˆV = 05(X + Y), and by condition () ˆV is a bloc rotation matrix, the claim is confirmed and also = + (3) () Let E be a consistent n-view fundamental matrix satisfying condition (3), ie, its (thin) ( spectral decomposition )[ is of the form + X [X,Y Y, where 05(X+Y) is a bloc rotation matrix Since + =, we can use Lemma 5, which implies that the (thin) SVD is of the form [Û, ˆV ( )[ + ˆV, where + ˆV = 05(X +Y), concluding the proof Û Corollary 1 Euclidean reconstruction Let E be a consistent n-view essential matrix with 6 distinct eigenvalues, then it is possible to explicitly determine R 1 R n and t 1,,t n that are consistent with respect to E Proof he claim is justified by the following construction, which relies on the spectral characterizations of hm 3 1 Calculate the eigenvectors X,Y of E, and the corresponding three positive eigenvalues, +, and three negative eigenvalues,, respectively o realize condition (3) of hm 3, there are 8 possible choices of 05(X + YI s ), where I s = ± ±1 0, due to the sign ambiguity of 0 0 ±1 each eigenvector hen, I s is chosen such that 05(X+YIs ) is bloc rotation matrix up to a global sign which can be removed 3 his spectral decomposition determines directly an SVD decomposition in the form of condition () of hm 3 We would lie to emphasize that due to the multiplicity of singular values, a standard SVD method which is performed directly on E, in general, will not produce this special SVD pattern 4 Having the relation E ij = Ûi +ˆV j + ˆV i + Ûj and sincee ii = 0 we get thatûi +ˆV i is sew symmetric We denoteūi = Ûi + and, by Lemma 4, it holds that ˆ i = Ū i ˆV 1 i 4

5 5 Finally, for i = 1,,,n, define R i =: nˆv i and t i =: 1 nˆt i and it holds that E ij = R i ([t i [t j )R j his construction yields{r i } i=n i=1 and{t i} i=n i=1 which are consistent with respect to E and are unique up to a global similarity transformation 3 Supporting lemmas Lemma 4 [1 Let A,B R 3 3 with ran(a) =, ran(b) = 3 and AB is sew symmetric, then = B 1 A is sew symmetric Lemma 5 Let E S 3n of ran(6), and R 3 3, a diagonal matrix, with positive elements on the diagonal Let X,Y,U,V R 3n 3, and we define the mapping(x,y) (U,V) : X = 05(Û + ˆV), Y = 05(ˆV Û), Û = 05(X Y), ˆV = 05(X +Y) hen, the (thin) SVD ofe is of the form Û if and only if the (thin) spectral decomposition ofe is of the form ( [X,Y )[ X Y Proof he proof is provided in the supplementary material 4 Method Given images I 1,,I n, we assume a standard robust method is used to estimate the pairwise essential matrices, which we denote by Ω = {Êij} In practice, only a small subset of the pairwise essential matrices are estimated, due to occlusion, large viewpoint and brightness changes as well as objects motion, and in addition the available estimates are noisy Our goal therefore is to find a consistent n-view essential matrix E S 3n that is as similar to the measured essential matrices as possible o mae an n-view essential matrix consistent, its blocs of pairwise essential matrices must each be scaled by an unnown factor [16 proposed an optimization scheme that explicitly sees for the unnown scale factors, yielding a nonlinear, ran-constrained optimization formulation he success of this approach critically depends on the quality of its initialization, which in experiments was obtained by applying another, state of the art SfM method More recently, [1 proposed an analogous approach for projective SfM hey showed that a consistent 3-view fundamental matrix, which uniquely determines camera matrices (up to a projective ambiguity) from a triplet of images, is invariant to scaling of its pairwise fundamental matrices his allowed them to formulate an optimization problem that sees 3-view fundamental matrices that are both consistent and compatible, while avoiding the need to explicitly optimize for the scale factors In this paper, we introduce an optimization scheme that is analogous to that of [1, but adapted to calibrated settings In particular, our scheme uses the algebraic constraints derived in hm 3 to enforce the consistency of noisy, and possibly partial essential matrices Similar to [1, our method simultaneously enforces consistency of camera triplets attached rigidly to each other, allowing us to avoid optimizing explicitly for the unnown scales of the estimated essential matrices (o that end we further extend hm 3 to handle scaled rotations for image triplets, see supplementary material for details) Our formulation, however, is more involved than in [1 due to the additional spectral constraints required for Euclidean reconstruction In the rest of this section we present our constrained optimization formulation and propose an ADMM-based solution scheme Subsequently, we discuss how to select minimal subsets of triplets to speed up the optimization Finally, we show how the results of our optimization can be used to reconstruct the absolute orientations and positions of the n cameras 41 Optimization In multi-view settings, it is common to define a viewing graph G = (V,W), with nodes v 1,,v n, corresponding to the n cameras, and w ij W if Êij is one of the estimated pairwise essential matrices Let τ denote a collection of m 3-cliques of cameras where m ( n 3) he collection may be the full set of the 3-cliques in G, or a chosen subset as described in Sec 4 We index the elements of τ by = 1,,m, whereτ() denote the th triplet he collection τ determines a partial selection of measured essential matrices, Ω, that plays a role in the optimization problem, where it holds that if Êij Ω thenê ij = Êji Ω We define the measurements matrix Ê S3n to have Ê ij as its (i,j) th bloc if Êij Ω and in the rest of its blocs In our optimization problem we loo fore that is as close as possible toê under the constraints that its9 9 blocs, induced by{τ()} m =1 and denoted by{e τ()} m =1, are consistent 3-view essential matrices In general, such E is inconsistent and incomplete, but as we explain in Sec 43 it is possible to retrieve a set of n absolute rotations and translations that is compatible with its essential matrices up to scale, which in turn implies that the completion of the missing entries is consistent 5

6 We formulate our constrained optimization as follows (i) Solving for E minimize E subject to m E τ() Êτ() F (6) =1 E E ii = ran(e τ() ) = 6 + (E τ() ) = (E τ() ) X(E τ() )+Y(E τ() ) is a bloc rotation matrix, E (t) = argmin E subject to m =1 + α 1 E τ() Êτ() F + α E B (t 1) D (t 1) E ii = E τ() +Γ (t 1) E τ() +Φ (t 1) F F (8) with i = 1,,n and = 1,,m, where + (E τ() ), (E τ() ) R 3 denote the 3 largest (descending order) and 3 smallest (ascending order) eigenvalues of E τ() respectively and X(E τ() ) R 9 3 and Y(E τ() ) R 9 3 are their corresponding eigenvectors Solving (6) is challenging due to its ran and spectral decomposition constraints We solve this optimization problem using ADMM o that end, as part of the ADMM method [3 4m auxiliary matrix variables of size 9 9 are added: m variables duplicating {E τ() } m =1, denoted B = {B } m =1 andd = {D } m =1, as well asm Lagrange multipliers, Γ = {Γ } m =1 and Φ = {Φ } m =1 his yields the following constrained optimization problem max Γ,Φ min E,B,D subject to m L(E τ(),b,γ,d,φ ) (7) =1 E E ii = ran(b ) = 6 + (B ) = (B ) ran(d ) = 6 X(D )+Y(D ) is a bloc rotation matrix withi = 1,,n and = 1,,m, where α 1 L(E τ(),b,γ,d,φ ) = B E τ() +Γ F + α E τ() Êτ() + F D E τ() +Φ F We initialize the auxiliary variables at t = 0 with B (0) (0) (0) = Êτ(),D = Êτ(),Γ = 0,Φ (0) = 0 hen, we solve the optimization problem iteratively by alternating between the following four steps his is a convex quadratic problem and can be solved efficiently, using a closed form solution (ii) Solving forb For all = 1,,m B (t) = argmin B subject to ran(b ) = 6 B E (t) τ() +Γ(t 1) F (9) + (B ) = (B ) he minimizer for this sub-problem is obtained in the following way By construction, E (t) τ() Γ(t 1) is a symmetric matrix, and we denote its (full) spectral decomposition by UU, where U R 9 9 and R 9 9 is a diagonal matrix in which the eigenvalues are arranged in a descending order hen, the update is B (t) = U U, (10) where R 9 9 is a diagonal matrix with the entries ii = { 1 ( ii 10 ii ) i 4,5,6 0 i = 4,5,6 (iii) Solving ford For all = 1,,m D (t) = argmin D subject to ran(d ) = 6 (11) D E (t) τ() +Φ(t 1) F (1) X(D )+Y(D ) is a bloc rotation matrix We minimize this sub-problem by an iterative process, which we repeat until convergence We begin with D = E (t) τ() Φ(t 1), which is symmetric by construction We apply spectral decomposition to D, and extract X(D ), Y(D ), + (D ) and (D ) Assuming no eigenvalue multiplicities, the eigenvectors are determined uniquely up to a sign (this argument is justified in 51) We denote byi s, a diagonal matrix of size3 3, such that each diagonal element is either 1 or -1 here are eight configurations for I s from which we select the best, in the sense that on average 6

7 each3 3 bloc, of the form, 05[X +YI s i,i = 1,,3, is close to scaled rotation, using the following score, 3 diag((xi Is +Y i I s ) (X i +Y i I s )) = argmax I s (X i=1 i +Y i I s ) (X i +Y i I s ) F Next, let V = V 1 be the proj of 05(X + YIs) such V V 3 that V i is the closest scaled rotation to 05[X + YIs i Projection to scaled SO(3) is obtained through removal of the singular values from the SVD decomposition, setting the scale factor to the average of the singular values, and possibly negating the scale factor to mae the determinant positive LetU = 05(X YIs), and X = 05(U+V) andỹ = 05(V U) We then update the value ofd to be the symmetric matrix D = [ X,Ỹ ( + )[ X Ỹ and repeat these steps until convergence (iv) UpdatingΓ,Φ For all = 1,,m Γ (t) = Γ(t 1) +B (t) E(t) τ() (13) Φ (t) = Φ(t 1) +D (t) E(t) τ() (14) 4 Graph construction and outliers removal As explained above, to apply our optimization algorithm, it is required to determine a collection τ of camera triplets, which is a subset of the given camera triplets he selection of a subset allows for better efficiency and robustness Similarly to [13, 1, we consider a weighted viewing graph G whose weights for each edgew ij is assigned to be the number of the inlier matches relatingi i andi j We begin by selecting disjoint maximal spanning trees from G, from which we extract an initial subset of triplets We then, remove near collinear and inconsistent triplets We next build a triplet graphg whose nodes, which represent image triplets, are connected by an edge whenever two triplets share the same two cameras Finally, we greedily remove nodes from G starting with the least consistent triplet (using the rotation consistency score defined below), a node is removed as long as the connectivity ofg is preserved and the total number of cameras associated with G does not decrease o define collinear and consistency scores for each triplet we denote the angles in the triangle formed by three cameras i,j, by θ i,θ j,θ respectively We measure each angle using the nown relative translations t ij,t i,t j, ie, t ij t i cosθ i = ( t ij t i ) hen, the collinearity score of cameras {i,j,} is the minimal angle in {θ i,θ j,θ } he consistency score of translations is defined by θ i +θ j +θ π and the consistency score of rotations by R ij R j R i I Our Method Chatterjee et al [4 Martinec et al [14 Data Set R f R d R f R d R f R d Vienna Cathedral Piazza del Popolo NYC Library Alamo Metropolis Yorminster Montreal ND ower of London Ellis Island Notre Dame able 1 Errors in estimated camera orientations, compared to ground truth measurements R f denotes the mean Frobenius norm error, averaged over the different cameras, andr d is the mean angular error in degrees Empty cells represent missing information 43 Location and orientation retrieval After solving (6), we extract frome the collection of 3- view essential matrices {E τ() } m =1, which, due to the optimization, are consistent wrt scaled rotations Next, using Corollary 1 with additional bloc normalizing at step 3, three rotations{r τ() 1,R τ(),r τ() 3 } and three translations {t τ() 1,t τ(),t τ() 3 } are extracted from each E τ(), which are uniquely defined up to a similarity transformation Now, any two triplets in τ that share two cameras a,b agree on E ab Since the sign of E ab is fixed, it determines the cameras a, b up to unique configurations [9 herefore, each one of the two triplets must agree with one of the two configurations As a result, assuming that both triplets defines the same configuration for a and b, there is a unique similarity transformation between the two triplets In practice, in our experiments we observe that this is always the case By the construction process described in Sec 4, the collection of triplets τ form a connected triplet graph It is therefore possible to traverse the graph and apply a similarity transformation on the three cameras of each new node τ(), and as a result bring all the cameras to a common Euclidean frame 5 Experiments o evaluate our approach, we implemented the SfM pipeline described next and tested it on ten challenging collections of unordered internet photographs of various sizes from [ Each of these datasets is provided with ground truth camera parameters We use our method to recover camera parameters and compare them to the parameters obtained by existing methods, before and after bundle adjustment (BA) 51 SfM pipeline he input to our algorithm is a set of independently estimated pairwise essential matrices, along with the number of inlier matches for each pairwise essential matrix We build a triplet graphg as we describe in Sec 4, removing any triplet whose (a) collinearity score is below 017 radians, 7

8 Our Method LUD [15 1DSFM [ Cui [5 Data set Nc x x x BA x BA Nr x x x BA x BA Nr x x BA x BA Nr x x BA x BA Nr Vienna Cathedral e Piazza del Popolo NYC Library Alamo e Metropolis Yorminster Montreal ND ower of London Ellis Island Notre Dame able Camera positions error in meters evaluated on the data sets of [ N c is the number of images in each dataset, x, x are the mean and median error respectively before bundle adjustment, and x BA, x BA are the mean and median errors after bundle adjustment N r are the number of reconstructed cameras Empty cells represent missing information Our Method LUD [15 1DSFM[ Cui [5 Data set R+ BA ot R+ BA ot R+ BA ot R+ BA ot Vienna Cathedral Piazza del Popolo NYC Library Alamo Metropolis Yorminster Montreal ND ower of London Ellis Island Notre Dame able 3 Running time in seconds for results in able R+ denote the time for motion averaging (for other methods rotation and translation estimation) BA the time for bundle adjustment and ot is the total running time of the method, including the additional time for building the triangle cover Empty cells represent image collections not tested by the authors In addition, Cui [5 does not report results before BA (b) rotation consistency score exceeds 11, or (c) translation consistency score exceeds 1 radians he final connected graphg defines the collectionτ of triplets of cameras Next we apply our optimization algorithm as is described in Sec 41 During optimization we observed that the eigenvalues of B and D (9) were always distinct for = 1,,m his means that the optimization variables, E τ(), indeed converge to 3-view consistent essential matrices with distinct eigenvalues At this stage, we follow Corollary 1 to recover camera positions and orientations for each triplet of cameras and align all the recovered camera matrices by similarity transformations, as is described in Sec 43 his yields a set of cameras with absolute positions and orientations he obtained camera parameters are finally refined using BA 5 Results o evaluate our recovered camera orientations, we compare our results to those obtained with the methods of Chatterjee et al [4 and Martinec et al [14 For a fair comparison we evaluate these methods on same subset of images used in our method Moreover, since in contrast to our method these solvers do not estimate camera positions we evaluate the results before BA he results are summarized in able 1 Our method outperforms these two solvers in nine out of the ten datasets o evaluate our recovered camera positions we compare our method to the following position solvers: Cui et al [5, 1DSFM [ and LUD [15 he results are summarized in able Note that [15, 5 use point matches in their pipelines, while both our method and [ do not use point correspondences until the final BA In general, the latter approaches allow for faster optimization, but result in inaccuracies before BA On the other hand, it allows for greater improvement in the final BA, compared to [15, 5 Indeed, as can be seen in the table, while our method surpasses all the other methods before bundle adjustment in three out of the ten datasets (according to mean error), it achieves superior performance on eight out of ten after BA able 3 further compares execution times, before and after BA, showing that our method is efficient, compared to existing methods 53 echnical details For Bundle Adjustment we used the heia standard SfM library [18 Camera position results for [15, 5, in ables and 3 are taen from the papers We ran our experiments on an Intel(R)-i7 360GHz with Windows Bundle Adjustment was performed on a Linux machine Intel(R) Xeon(R) 30GHz with 16 cores 6 Conclusion We have provided in this paper algebraic conditions for the consistency of essential matrices in multiview settings and an algorithm for their averaging given noisy and partial measurements In future research we will see to further incorporate collinear camera triplets in the averaging algorithm, explore numerical properties, and design online consistency enforcement algorithms for SLAM settings Acnowledgment his research was supported partly by the Minerva foundation with funding from the Federal German Ministry for Education and Research 8

9 References [1 S Agarwal, N Snavely, I Simon, S M Seitz, and R Szelisi Building rome in a day In 009 IEEE 1th international conference on computer vision, pages 7 79 IEEE, 009 [ M Arie-Nachimson, S Z Kovalsy, I Kemelmacher- Shlizerman, A Singer, and R Basri Global motion estimation from point matches In 01 Second Int Conf on 3D Imaging, Modeling, Processing, Visualization & ransmission, pages IEEE, 01 [3 S Boyd, N Parih, E Chu, B Peleato, J Ecstein, et al Distributed optimization and statistical learning via the alternating direction method of multipliers Foundations and rends in Machine learning, 3(1):1 1, [4 A Chatterjee and V M Govindu Robust relative rotation averaging IEEE ransactions on Pattern Analysis and Machine Intelligence, 40(4):958 97, 018, 7, 8 [5 Z Cui, N Jiang, C ang, and P an Linear global translation estimation with feature tracs ArXiv preprint, ArXiv: , 015, 8 [6 A Erisson, C Olsson, F Kahl, and -J Chin Rotation averaging and strong duality In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages , 018 [7 Goldstein, P Hand, C Lee, V Voroninsi, and S Soatto Shapefit and shapeic for robust, scalable structure from motion In European Conf on Computer Vision, pages Springer, 016 [8 R Hartley, J rumpf, Y Dai, and H Li Rotation averaging International Journal of Computer Vision, 103(3):67 305, 013 [9 R Hartley and A Zisserman Multiple view geometry in computer vision Cambridge University Press, [10 N Jiang, Z Cui, and P an A global linear method for camera pose registration In IEEE Int Conf on Computer Vision, pages , 013 [11 Y Kasten, M Galun, and R Basri Resultant based incremental recovery of camera pose from pairwise matches arxiv preprint arxiv: , 019 [1 Y Kasten, A Geifman, M Galun, and R Basri Gpsfm: Global projective sfm using algebraic constraints on multi-view fundamental matrices ArXiv preprint, ArXiv: , 018 1,, 3, 5, 7 [13 M Klopschitz, A Irschara, G Reitmayr, and D Schmalstieg Robust incremental structure from motion In Proc 3DPV, volume, pages 1 8, 010, 7 [14 D Martinec and Pajdla Robust rotation and translation estimation in multiview reconstruction In 007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1 8 IEEE, 007, 7, 8 [15 O Ozyesil and A Singer Robust camera location estimation by convex programming In IEEE Conf on Computer Vision and Pattern Recognition, pages , 015, 8 [16 S Sengupta, Amir, M Galun, Goldstein, D W Jacobs, A Singer, and R Basri A new ran constraint on multiview fundamental matrices, and its application to camera location recovery In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages , 017, 3, 5 [17 N Snavely, S M Seitz, and R Szelisi Modeling the world from internet photo collections International Journal of Computer Vision, 80():189 10, 008 [18 C Sweeney heia multiview geometry library: utorial & reference 8 [19 C Sweeney, Sattler, Hollerer, M ur, and M Pollefeys Optimizing the viewing graph for structure-frommotion In IEEE Int Conf on Computer Vision, pages , 015 [0 R ron and K Daniilidis he space of essential matrices as a Riemannian quotient manifold SIAM Journal on Imaging Sciences, 10(3): , 017 [1 R ron and R Vidal Distributed image-based 3-d localization of camera sensor networs In Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 009 8th Chinese Control Conference, pages IEEE, 009 [ K Wilson and N Snavely Robust global translations with 1dsfm In European Conf on Computer Vision, pages Springer, 014 1,, 7, 8 [3 C Wu owards linear-time incremental structure from motion In 013 International Conference on 3D Vision-3DV 013, pages IEEE, 013 9

10 Algebraic Characterization of Essential Matrices and heir Averaging in Multiview Settings Supplementary Material Yoni Kasten* Amnon Geifman* Meirav Galun Ronen Basri Weizmann Institute of Science 1 Counter example In the introduction of the paper we claim that a consistent n-view fundamental matrix whose 3 3 blocs form essential matrices does not necessarily form a consistent n-view essential matrix We justify this argument by a constructing a counter example for the case n = 3 First note that the following observation is true If t,t 3,a,b R 3 andr,r 3 SO(3) and we set and V 1 = I 3 3,V = R,V 3 = R 3 +at 3,t 1 = F ij = V i ([t i t j )V j, i,j = 1,,3 then by construction 0 F 1 F 13 F = F 1 0 F 3 F13 F3 0 is a consistent n-view fundamental matrix, unless the outer productat 3 is such that it reduces the full ran ofv 3 to Examine each of the bloc matricesf ij : F 1 = V 1 [t 1 t V = [ t R F 13 = V 1 [t 1 t 3 V 3 = [ t 3 (R 3 +t 3 a ) = [ t 3 R 3 F 3 = V [t t 3 V 3 = R [t t 3 (R 3 +t 3 a ) It follows from this derivation thatf 1 andf 13 are essential matrices Next, we use the identity [Rt = R[t R, which holds for any R SO(3) and t R 3, to show that for anyb R 3 F 3 = [R (t t 3 ) R (R 3 +t 3 a ) = [R (t t 3 ) (R (R 3 +t 3 a )+R (t t 3 )b ) = [R (t t 3 ) R (R 3 +t 3 a +(t t 3 )b ) he trivial choice of a = b = 0 yields an essential matrix F 3 which is consistent with the essential matrices F 1 and F 13 Our aim is to mae a specific choice of R,R 3,t,t 3,a,b such that F 3 will be an essential matrix that is inconsistent with the essential matrices F 13 and F 1 11 Choosing R,R 3,t,t 3,a,b We first give some intuition for the way we set the values ofr,r 3,t,t 3,a,b We loo at the term R (R 3 +t 3 a +(t t 3 )b ) (15) and wish to set the values such that this term does not collapse to RR 3, but is still in the form of RR, for some R SO(3) o that end, we loo for two rotation matrices R and R such that M = R R has ran, and set the values such that the term in (15) will be equal torr Following the construction of M, the SVD of M is of the form M = U σ σ 0 V and we set t 3 = u 1 σ 1,a = v 1,b = v,t = t 3 u σ (16) herefore, R R = M = u 1 σ 1 v 1 +u σ v = t 3a +(t 3 t )b Now we set R to be some rotation matrix and R 3 = R Clearly, which means yielding t 3 a +(t 3 t )b = R R R R 3 R R = R (R 3 +t 3 a +(t 3 t )b ), F 3 = [R (t t 3 ) R R, which is an essential matrix that in general is inconsistent withf 1,F 13 10

11 1 echnical details of the code provided for demonstrating a counter example We first randomly sampled two rotation matrices R,R until we obtained M = R R of ran, and we set R 3 = R hen, we sampled randomly a rotation matrix, and we set R hese selections are stored in a file called counter datamat In the code we assign values to t,t 3,R 3,a,b, according to (16) We verify that indeed F 1,F 13 and F 3 are essential matrices By construction, the 3-view fundamental matrix F is consistent In order to verify that the essential matrices are not consistent we extract the relative rotations from them Each essential matrix defines two possible relative rotations We evaluate the relation R 1 R 3 R 31 for each of the 8 choices of triplet of relative rotations, and verify that none of them closes a loop, ie the following always holds R 1 R 3 R 31 I An interesting observation In addition, we verify in the code that for any choice of signs for the eigenvectors of F, X,Y, it turns out that 05(X+Y)is indeed not a bloc rotation matrix Interestingly, it holds that + =, which shows that by itself this condition, in this case, is not sufficient for defining an appropriate consistent essential matrix o conclude, requiring from a set of essential matrices to fulfill the sufficient conditions for consistent n-view fundamental matrix does not provide sufficient conditions for generating from this set, a consistent essential matrix he counter example demonstrates the inconsistency of the essential matrices, as well as the violation of our conditions on the consistency of an n-view essential matrix Proof of Lemma 5 Below we prove Lemma 5 from the paper (which we rename here to be Lemma 6) Lemma 6 Let E S 3n of ran(6), and R 3 3, a diagonal matrix, with positive elements on the diagonal Let X,Y,U,V R 3n 3, and we define the mapping(x,y) (U,V) : X = 05(Û + ˆV), Y = 05(ˆV Û), Û = 05(X Y), ˆV = 05(X +Y) hen, the (thin) SVD ofe is of the form Û if and only if the (thin) spectral decomposition ofe is of the form ( [X,Y )[ X Y Proof ( ) Û = ÛˆV + ˆVÛ = 05 (Û + V)(Û ˆ + ˆV) 05 (ˆV Û)(ˆV Û) = ( ) [Û 05[Û + ˆV, ˆV Û + ˆV, ˆV Û = ( )[ X [X,Y Y (17) where X = 05(Û + ˆV) and Y = [ 05(ˆV Û) Since, Û [ X [Û, ˆV ˆV = I 6 6, it yields Y [X,Y = I 6 6, concluding that the last term in (17) is indeed (thin) spectral decomposition of E ( ) ( ) [X,Y [X,Y = XX YY = 05(X +Y)(X Y) +05(X Y)(X +Y) = ( ) 05[X Y,X +Y [X +Y,X Y = ( )[ ˆV [Û, ˆV (18) Û where Û = 05(X Y) and ˆV = 05(X + Y) he same argument for orthogonality wors here, showing that indeed that last term in (18) is SVD of E 3 Handling scaled rotations Our optimization enforces the consistency of camera triplets, while allowing the essential matrices to be scaled arbitrarily his is possible, because our theory can be generalized to handle scaled rotations Below we generalize Definition 4 from the paper to allow essential matrices of the form E ij = α i R i ([t i [t j )R j α j and prove that the main theorem, ie, heorem 3 from the paper, holds for this generalization as well his argument is then used to justify our treatment of camera triplets Definition 5 An n-view essential matrix E is called scaled consistent if there exist n rotation matrices {R i } n i=1, n vectors {t i } n i=1 and n non-zero scalars {α i} n i=1 such that E ij = α i Ri ([t i [t j )R j α j he following theorem is a generalized version of hm from the paper, and the derivations here are inspired by the derivations made in [16 heorem 7 Let E be a scaled consistent n-view essential matrix, associated with scaled rotation matrices {α i R i } n i=1, α i 0 and camera centers {t i } n i=1 E satisfies the following conditions 11

12 1 E can be formulated as A+A wherea = UV andu,v R 3n 3 V = α 1 R 1 α n R n U = α 1 R 1 1 α n R n n with i = [t i and wlog n i=1 α i t i = 0 Each column of U is orthogonal to each column of V, ie,v U = ran(v)=3 4 If not all {t i } n i=1 are collinear, then ran(u) and ran(a) = 3 Moreover, if the (thin) SVD of A is A = Û V ˆ, with Û, ˆV R 3n 3 and R 3 3 then the (thin) SVD ofe is implying ran(e) = 6 Û Proof 1 he decomposition is a straightforward result from Def 5 Moreover, any global translation of all the camera centers, will not change the values of the entries of E In particular, if we denote the camera centers by{ t i } n i=1 and they are translated to their new α positiont i = t i αi t i, then α i i t i = 0 By the decomposition above, we have that i=1 i=1 i=n V U = α i R iri i=n i = [ α i t i = 0 which concludes that each column of U is orthogonal to each column ofv 3 ran(v) = 3 since each bloc of V is of ran 3 and V has 3 columns 4 Assume by contradiction that ran(u) < 3 hen, t R 3,t 0, st Ut = 0 his implies that t i t = 0 for all i = 1,,n his implies that all the t i s are parallel to t, violating our assumption that not all camera locations are collinear Consequently ran(u) = 3 and therefore also ran(a) = 3 Finally, leta = ÛˆV the SVD ofa SinceA = UV we get that Span(U) = Span(Û),Span(V) = Span(ˆV) hen, since A+A, we get ÛˆV + ˆVÛ = [ Û ˆV ( 0 0 )[ˆV Û Following the result that the columns of U are orthogonal to those ofv, it turns out that [ Û ˆV is column orthogonal, concluding that the form above is the SVD ofe, andran(e) = 6 Next, we show that hm 3 in the paper is also applicable with the generalized definition, ie, Def 5 heorem 8 LetE S 3n be a consistentn-view fundamental matrix with a set ofncameras whose centers are not all collinear We denote by +, R 3 3 the diagonal matrices with the 3 positive and 3 negative eigenvalues of E, respectively he following conditions are equivalent: 1 E is a scaled consistentn-view essential matrix he (thin) SVD of E can be written in the form [Û, ˆV ( )[ + ˆV + Û with Û, ˆV R 3n 3 such that each 3 3 bloc of ˆV, ˆV i, i = 1,,n, is a scaled rotation matrix, ie, ˆV i = ˆα i ˆRi, where ˆR i SO(3) and ˆα i 0 We say that ˆV is a scaled bloc rotation matrix 3 + = and the (thin) spectral decomposition of E is of the form ( )[ + X [X,Y Y such that 05(X +Y) is a scaled bloc rotation matrix Proof (1) () Assume that E is a scaled consistent n- view essential matrix hen, according to hm 7, A+A witha = UV andu,v R 3n 3 with α 1 R1 V = α n Rn U = α 1 R 1 1 α n R n n where i = [t i SinceA = UV andran(a) = 3, then A A = VU UV anda A 0 with ran(a A) = 3 (A and A A share the same null space) First, we construct a spectral decomposition toa A, relying on the special properties of U and V We have ran(u) = 3, and therefore U U, which is a 3 3, symmetric positive semi-definite 1

13 matrix, is of full ran Its spectral decomposition is of the form U U = PDP, where P SO(3) (Spectral decomposition guarantees that P O(3) However, P can be replaced by P if det(p) = 1) D R 3 3 is a diagonal matrix consisting of the (positive) eigenvalues ofu U his spectral decomposition yields the following decomposition A A = VPDP V (19) Now, note that P V VP = P [ α 1 R 1 α 1 R1 α n R n P α n Rn n n = P ( α i )I 3 3P = ( α i )I 3 3 i=1 i=1 Letα = n i=1 α i By a simple manipulation (19) becomes a spectral decomposition A A = 1 α VP(αD)P V 1 α (0) On the other hand, the (thin) SVD of A is of the form A = ÛˆV, whereû, ˆV R 3n 3, R 3 3 his means that A A = ˆV ˆV (1) Due to the uniqueness of the eigenvector decomposition, (0) and (1) collapse to the same eigenvector decomposition, up to some global rotation, H SO(3), that is 1 α VP = ˆVH, which means that ˆV i = α i α R i PH () Since Ri,P,H α SO(3), then setting ˆα i =: αi and ˆR i =: Ri PH SO(3), shows that ˆV is a scaled bloc rotation matrix Finally, by hm 7, the (thin) SVD of E is of the form Û (3) and according to Lemma 6, the eigenvalues of E are and Since the elements on the diagonal of are positive, ande is symmetric with exactly 3 positive eigenvalues + and 3 negative eigenvalues, it follows that = + and = concluding the proof () (1) Let E be a consistent n-view fundamental matrix that satisfies condition () We would lie to show that E is a scaled consistent n-view essential matrix By condition ()E can be written as Û +ˆV + ˆV + Û = Ū ˆV + ˆV U (4) where Ū = Û + with ˆV i = ˆα i ˆRi, ˆRi SO(3) By definition E ii = 0, and this implies that ŪiˆV i is a sew symmetric matrix Using Lemma 4 in the paper, Ūi = ˆV i ˆi for some sew symmetric matrix ˆ i = [ˆt i Plugging Ūi and ˆV i in (4) yields E ij = ŪiˆV j + ˆV i Ū j = ˆα i ˆα j ˆRiˆi ˆR j ˆα i ˆα j ˆRiˆj ˆR j = ˆα i R i ([t i [t j )R j ˆα j where R i = ˆR i, α i = ˆα i and t i = ˆt i, concluding the proof Finally, the derivation of the equivalence() (3) is exactly as in hm 3 in the paper Corollary Let E be a scaled consistent n-view matrix, then 1 he scale of each bloc ˆV i in the scaled bloc rotation matrix ˆV can be calculated using by ˆα i = (det(ˆv i )) 1 3 Let E be a scaled consistent n-view essential matrix hen, the transformationdiag( 1ˆ 1 α 1 I 3 3,, αˆ n I 3 3 ) E diag( 1ˆα 1 1 I 3 3,, αˆ n I 3 3 ) transform E to be a consistentn-view essential matrix Corollary 3 A scaled consistent 3-view essential matrix is invariant to pairwise scaling Proof his proof is inspired by the proof of Corollary, presented in [1 Let E be a scaled consistent 3- view essential matrix whose blocs are defined as E ij = α i Ri ( i j )R j α j, and let Ẽ be a 9 9 matrix whose blocs are defined to beẽij = s ij E ij wheres ij 0 are arbitrary pairwise scale factors Without loss of generality we can assume that the number of negative scale factors is even (otherwise we can multiply the entire matrix by -1) herefore, s 1 = ( s1s13 s 3 ) 1, s = ( s3s1 s 13 ) 1, and s3 = ( s13s3 s 1 ) 1 determine real values such that s 1 s = s 1, s 1 s 3 = s 13, ands s 3 = s 3 Let α i = s i α i fori = 1,,3, we get that Ẽ ij = s ij E ij = α i R i ( i j )R j α j (5) Hence, Ẽ is a scaled consistent 3-view essential matrix 13