Triangulation
Perspective projection in homogenous coordinates ~x img I 0 apple R t 0 T 1 ~x w ~x img R t ~x w
Matrix transformations in 2D ~x img K R t ~x w K = 2 3 1 0 t u 40 1 t v 5 0 0 1 Translation K = K = 2 3 s x t u 4 0 s y t v 5 0 0 1 Added skew if image x and y axes are not perpendicular 2 3 s x 0 t u 4 0 s y t v 5 0 0 1 Scaling of Image x and y (conversion from meters to pixels )
Final perspective projection Camera extrinsics: where your camera is relative to the world. Changes if you move the camera ~x img K R t ~x w Camera intrinsics: how your camera handles pixel. Changes if you change your camera ~x img P~x w
Final perspective projection ~x img K R t ~x w Camera parameters ~x img P~x w
Goal: find the parameters of the camera ~x img K R t ~x w Why? Tells you where the camera is relative to the world/particular objects Equivalently, tells you where objects are relative to the camera Can allow you to render new objects into the scene
Y Y Z O O X X Z
~x img P~x w Suppose we know that (X,Y,Z) in the world projects to (x,y) in the image. How many equations does this provide? 2 3 x 4y5 P 1 2 3 X 6Y 7 4Z 5 1 Need to convert equivalence into equality.
~x img P~x w Suppose we know that (X,Y,Z) in the world projects to (x,y) in the image. How many equations does this provide? Note: λ is unknown 2 4 3 x y5 = P 2 3 X 6Y 7 4Z 5 1
~x img P~x w Suppose we know that (X,Y,Z) in the world projects to (x,y) in the image. How many equations does this provide? 2 4 3 x y5 = 2 2 3 P 11 P 12 P 13 P 14 4P 21 P 22 P 23 P 5 24 P 31 P 32 P 33 P 34 X 6Y 4Z 1 3 7 5
~x img P~x w Suppose we know that (X,Y,Z) in the world projects to (x,y) in the image. How many equations does this provide? x = P 11 X + P 12 Y + P 13 Z + P 14 y = P 21 X + P 22 Y + P 23 Z + P 24 = P 31 X + P 32 Y + P 33 Z + P 34
Suppose we know that (X,Y,Z) in the world projects to (x,y) in the image. How many equations does this provide? 2 equations! ~x img P~x w (P 31 X + P 32 Y + P 33 Z + P 34 )x = P 11 X + P 12 Y + P 13 Z + P 14 (P 31 X + P 32 Y + P 33 Z + P 34 )y = P 21 X + P 22 Y + P 23 Z + P 24 Are the equations linear in the parameters? How many equations do we need?
(P 31 X + P 32 Y + P 33 Z + P 34 )x = P 11 X + P 12 Y + P 13 Z + P 14 (P 31 X + P 32 Y + P 33 Z + P 34 )y = P 21 X + P 22 Y + P 23 Z + P 24 XxP 31 + YxP 32 + ZxP 33 + xp 34 XP 11 Y P 12 ZP 13 P 14 =0 In matrix vector form: Ap = 0 6 points give 12 equations, 12 variables to solve for But can only solve upto scale
In matrix vector form: Ap = 0 We want non-trivial solutions If p is a solution, αp is a solution too Let s just search for a solution with unit norm Ap =0 s.t kpk =1 How do you solve this?
In matrix vector form: Ap = 0 We want non-trivial solutions If p is a solution, αp is a solution too Let s just search for a solution with unit norm Ap =0 s.t kpk =1
What happens if there are more than 6 points? What if there is noise in the point locations? min kapk 2 p Ap =0 s.t kpk =1
What happens if there are more than 6 points? What if there is noise in the point locations? min p T A T Ap p Ap =0 s.t kpk =1 Look at eigenvector of A T A with the smallest eigenvalue!
>=6 points with known 3D coordinates + known image coordinates XxP 31 + YxP 32 + ZxP 33 + xp 34 XP 11 Y P 12 ZP 13 P 14 =0 In matrix vector form: want Ap = 0 Resilience to noise: min p p T A T Ap s.t kpk =1 Look at eigenvector of A T A with the smallest eigenvalue!
We need 6 world points for which we know image locations Would any 6 points work? What if all 6 points are the same? Need at least 6 non-coplanar points!
Y Y Z O O X X Z
~x img P~x w ~x img K R t ~x w How do we get K, R and t from P? Need to make some assumptions about K What if K is identity?
~x img P~x w ~x img K R t ~x w How do we get K, R and t from P? Need to make some assumptions about K What if K is upper triangular? 2 3 s x t u K = 4 0 s y t v 5 0 0 1 Added skew if image x and y axes are not perpendicular
How do we get K, R and t from P? Need to make some assumptions about K What if K is upper triangular? 2 3 s x t u K = 4 0 s y t v 5 P = K [ R t] 0 0 1 First 3 x 3 matrix of P is KR RQ decomposition: decomposes an n x n matrix into product of upper triangular and rotation matrix
How do we get K, R and t from P? Need to make some assumptions about K What if K is upper triangular? P = K [ R t] First 3 x 3 matrix of P is KR RQ decomposition: decomposes an n x n matrix into product of upper triangular and rotation matrix t = K -1 P[:,2] ß last column of P
Camera calibration and pose estimation
Triangulation Suppose we have two cameras Calibrated: parameters known And a pair of corresponding pixels Find 3D location of point!
Triangulation Suppose we have two cameras Calibrated: parameters known And a pair of corresponding pixels Find 3D location of point! P (1) P (2) (x 1,y 1 ) (x 2,y 2 )
Triangulation 2 3 x 1 4y 5 1 1 2 3 x 2 4y 5 2 1 ~x (1) img P (1) ~x w ~x (2) img P (2) ~x w 2 3 X 6Y 7 4Z 5 1
Triangulation ~x (1) img P (1) ~x w x 1 = P (1) 11 X + P (1) 12 Y + P (1) 13 Z + P (1) 14 y 1 = P (1) 21 X + P (1) 22 Y + P (1) 23 Z + P (1) 24 = P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34 (P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34 )x 1 = P (1) 11 X + P (1) 12 Y + P (1) 13 Z + P (1) 14 X(P (1) 31 x 1 P (1) (1) 11 ) + Y (P 32 x 1 P (1) (1) 12 ) + Z(P 33 x 1 P (1) 13 ) + (P (1) 34 x 1 P (1) 14 ) =0
X(P (1) X(P (1) Triangulation 1 image gives 2 equations Need 2 images! ~x (1) img P (1) ~x w 31 x 1 P (1) (1) 11 ) + Y (P 32 x 1 P (1) (1) 12 ) + Z(P 33 x 1 P (1) (1) 13 ) + (P 34 x 1 P (1) 14 ) =0 31 y 1 P (1) (1) 21 ) + Y (P 32 y 1 P (1) (1) 22 ) + Z(P 33 y 1 P (1) (1) 23 ) + (P 34 y 1 P (1) 24 ) =0 Solve linear equations to get 3D point location
Linear vs non-linear optimization x 1 = P (1) 11 X + P (1) 12 Y + P (1) 13 Z + P (1) 14 y 1 = P (1) 21 X + P (1) 22 Y + P (1) 23 Z + P (1) 24 = P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34 x 1 = P (1) 11 X + P (1) 12 Y + P (1) 13 Z + P (1) 14 P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34 y 1 = P (1) 21 X + P (1) 22 Y + P (1) 23 Z + P (1) 24 P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34
Linear vs non-linear optimization x 1 = P (1) 11 X + P (1) 12 Y + P (1) 13 Z + P (1) 14 P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34 y 1 = P (1) 21 X + P (1) 22 Y + P (1) 23 Z + P (1) 24 P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34 P (1) 11 (x X + P (1) 12 Y + P (1) 13 Z + P (1) 14 1 P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34 P (1) 21 +(y X + P (1) 22 Y + P (1) 23 Z + P (1) 24 1 P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34 Reprojection error ) 2 ) 2
Linear vs non-linear optimization P (1) 11 (x X + P (1) 12 Y + P (1) 13 Z + P (1) 14 1 P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34 P (1) 21 +(y X + P (1) 22 Y + P (1) 23 Z + P (1) 24 1 P (1) 31 X + P (1) 32 Y + P (1) 33 Z + P (1) 34 Reprojection error Reprojection error is the squared error between the true image coordinates of a point and the projected coordinates of hypothesized 3D point Actual error we care about Minimize total sum of reprojection error across all images Non-linear optimization ) 2 ) 2