M3: Multiple View Geometry L18: Projective Structure from Motion: Iterative Algorithm based on Factorization Based on Sections 13.4 C. V. Jawahar jawahar-at-iiit.net Mar 2005: 1
Review: Reconstruction from Two Calibrated Cameras If a world point P is imaged at p and p in two cameras M and M, zp = MP or p MP = 0 zp = M P or p M P = 0 World point can be computed as the solution of the homogeneous system of equation [p] M [p ] M P = 0 C. V. Jawahar jawahar-at-iiit.net Mar 2005: 2
Ill posed SFM Problem Given the images of multiple world points in two or cameras zp i = MP i zp i = M P i we are interested in finding out the structure P i and motion M i. C. V. Jawahar jawahar-at-iiit.net Mar 2005: 3
Ambiguity of Projective Reconstruction If M i and P j are the solutions of the above mentioned equations, then M i = M i Q and P j = Q 1 P j are also acceptable solutions. The matrix Q is only defined upto scale and has 15 unknowns. The system of equations with m cameras and n points will have solution only if 2mn 11m + 3n 15 When m = 2, we need seven points. C. V. Jawahar jawahar-at-iiit.net Mar 2005: 4
Method 1: Five Point Correspondence and Epipole Define a projective frame of reference and compute structure/motion in this coordinate system. C. V. Jawahar jawahar-at-iiit.net Mar 2005: 5
Method 2: Cameras/Motion from Fundamental Matrix Let F be a fundamental matrix and S be a skew-symmetric matrix. Define the pair of camera matrices, M = [I 0] and M = [SF e ] where e is the epipole such that e T F = 0 and assume the M so defined is a valid camera matrix with rank -3. Then F is the fundamental matrix relating the views. C. V. Jawahar jawahar-at-iiit.net Mar 2005: 6
Projective Structure and Motion from Multiple Images Frequent situation. Multilinear constraints are not useful for views > 4. Robust computation and minimisation of error propagation. C. V. Jawahar jawahar-at-iiit.net Mar 2005: 7
Imaging Equations Basic Perspective Imaging Equation: zp = MP If there are m points and n cameras z ij p ij = M i P j Or D = MP where P = (P 1, P 2... P n ) z 11 p 11... z 1n p 1n D =......... z m1 p m1... z mn p mn and M = M 1... M m C. V. Jawahar jawahar-at-iiit.net Mar 2005: 8
Rank and Factorization of D D is the product of two matrices 3m 4 and 4 n. Therefore rank of D is 4. If we knew z ij, we could have thought of factoring D into M and P However, we do not know the depths z ij. Or else this factorization is much more involved that the affine factorization. What about the solution of the following minimisation problem? E = D MP 2 = j E j = ij z ij p ij M i P j 2 C. V. Jawahar jawahar-at-iiit.net Mar 2005: 9
Unfortunately NOT. The above problem is ill posed z ij = 0; M i = 0 and P j = 0 is an acceptable minima!! There are infact many more non-meaningful trivial solutions. Solution is to impose additional constraints: say that columns of the matrix D, i.e., d j have unit norm. C. V. Jawahar jawahar-at-iiit.net Mar 2005: 10
An important class of Iterative Solution Procedures Fix the value of z j = [z 1j,... z mj ] Factorize and Compute M and P. Using the values of M and P, update the projective depth z ij C. V. Jawahar jawahar-at-iiit.net Mar 2005: 11
Condition for Minima E j = m z ij p ij M i P j 2 i=1 Differentiating with respect to P j and equating to zero. Or E j P j = 2 m M T i (z ij p ij M i P j ) = 0 i=1 M T d j = M T MP j P j = M + d j C. V. Jawahar jawahar-at-iiit.net Mar 2005: 12
If M = UWV T, its pseudo inverse is VW 1 U T. The objective function to be minimised reduces to E j = m z ij p j M i P j 2 i=1 E j = (I MM + )d j 2 E j = [I UU T ]d j 2 = 1 Ud j 2 Minimisation of E j with respect to z ij and P j is equivalent to maximisation of Ud j 2 C. V. Jawahar jawahar-at-iiit.net Mar 2005: 13
Define Q j as Q j = p 1j 0... 0 0 p 2j 0............ 0 0... p mj such that d j = Q j z j C. V. Jawahar jawahar-at-iiit.net Mar 2005: 14
Minimisation of E j is equivalent to maximisation of R j z j 2 with respect to z j under the constraint Q j z j 2 = 1, where R = U T Q j i.e., minimise R j z j 2 + λ(1 Q j z j 2 ) Solution to the above minimisation problem (by a direct differentiation and equating to zero) is given by the largest scalar which satisfies R T j Rz j = λq T J Q j z j This is a generalised eigen value problem with available numerical solutions. C. V. Jawahar jawahar-at-iiit.net Mar 2005: 15
Factorization Algorithm for Projective Shape from Motion 1. Compute an initial estimate of the projective depths z ij 2. Normalize each column of the data matrix D. 3. Repeat (a) Use SVD to compute 3m 4 matrix M and the 4 n matrix P than minimize D MP 2. (b) for j = 1 to n, compute the matrices R j and Q j and find the value of z j that minimizes R j z j 2 under the constraint Q j z j 2 = 1 as the solution of a generalized eigen value problem. (c) Update the value of D accordingly. Until converges C. V. Jawahar jawahar-at-iiit.net Mar 2005: 16
Convergence of the Above Algorithm C. V. Jawahar jawahar-at-iiit.net Mar 2005: 17
Euclidean Upgrade of the Projective Structure and Motion C. V. Jawahar jawahar-at-iiit.net Mar 2005: 18
Next Module: Processing Dynamic Scenes C. V. Jawahar jawahar-at-iiit.net Mar 2005: 19