GEOMETRY FOR 3D COMPUTER VISION

Transcription

1 GEOMETRY FOR 3D COMPUTER VISION 1/29 Válav Hlaváč Czeh Tehnial University, Faulty of Eletrial Engineering Department of Cybernetis, Center for Mahine Pereption Praha 2, Karlovo nám. 13, Czeh Republi LECTURE PLAN 1. Brief introdution to my group Center for Mahine Pereption. 2. Mathematial model of a single perspetive amera. 3. Epipolar onstraint. 4. Correspondene problem. 5. Results: state-of-the-art stereo, unalibrated 3D reonstrution, VR model.

2 CENTER FOR MACHINE PERCEPTION Researh group, head Prof. Válav Hlaváč, established 1986 as omputer vision lab, under the name CMP sine staff (11 2 Prof., 1 Asso. Prof., 3 PhD, 7 MS); out of it 2 mathematiians, 2 physiists, 8 engineers) + 8 full time PhD students. Interests: omputer vision, pattern reognition, mathematial models for treating unertainty. Links to industry mainly via a spin-off ompany Neovision Prague (10 people). E.g. Samsung, Boeing, Texas Instruments, Robert Bosh, Kyoera, Hitahi. 2/29

3 MAIN RUNNING PROJECTS AtIPret (R&D, , IST ) Interpreting and Understanding Ativities of Expert Operators for Teahing and Eduation (V. Hlaváč, J. Matas). ISAAC (Trial, 2002, IST ) Inspeting Sewerage Systems And Image Analysis by Computer (V. Hlaváč). Reonstrution of 3D sene from multiple unalibrated views (V. Hlaváč). Computational stereo (R. Šára). Omni-diretional vision. (T. Pajdla). Authentiation based on fae reognition (J. Matas). Pattern reognition theory (V. Hlaváč). 3/29

4 V. Hlaváč, books 4/29 Šonka M., Hlaváč V., Boyle R.B.: Image Analysis, Proessing and Mahine Vision, 2nd edition, PWS Boston, USA, 1999 (China edition 2002), 800 p, USD 105. Shlesinger M.I., Hlaváč V.: Ten Letures on Statistial and Strutural Pattern Reognition Kluwer, Dordreht, May 2002, EUR 165.

5 BASICS OF PROJECTIVE GEOMETRY Pinhole model - the simplest geometrial model of human eye, photographi and TV amera. 5/29 Perspetive projetion, also entral projetion. Parallel lines in the world do not remain parallel in the image (e.g., view along the straight setion of a railroad). Foal point Image plane Horizon Vanishing point Optial axis Base plane

6 PROJECTIVE SPACE 6/29 Consider (n + 1) dimensional vetor spae without its origin, R n+1 {(0,..., 0)}. Define an equivalene relation [x 1,..., x n+1 ] T [x 1,..., x n+1] T iff α 0 : [x 1,..., x n+1 ] T = α [x 1,..., x n+1] T Projetive spae P n is the quotient spae of this equivalene relation. Points in the projetive spae are expressed in homogeneous o-ordinates (alled also projetive oordinates) x = [x 1,..., x n, 1] T.

7 RELATION BETWEEN EUCLIDEAN AND PROJECTIVE SPACES 7/29 Consider Eulidean spae R n. The one-to-one mapping from the R n into P n [x 1,..., x n ] T [x 1,..., x n, 1] T Projetive points [x 1,..., x n, 0] T do not have an Eulidean ounterpart and represent points at infinity in a partiular diretion. Consider [x 1,..., x n, 0] T as a limiting ase of [x 1,..., x n, α] T that is projetively equivalent to [x 1 /α,..., x n /α, 1] T, and assume that α 0. This orresponds to a point in R n going to infinity in the diretion of the radius vetor [x 1 /α,..., x n /α] R n.

8 PROJECTIVE TRANSFORMATION (also CO-LINEATION) 8/29 Co-lineation is any mapping P n P n. Defined by a regular (n + 1) (n + 1) matrix A, ỹ = A x. Matrix A is defined up to a sale fator. Co-lineations map hyperplanes to hyperplanes. A speial ase is the mapping of lines to lines that is often used in omputer vision.

9 SINGLE PERSPECTIVE CAMERA, pinhole model Optial axis Z w O w Y w 9/29 X w Sene point X X w World Eulidean oordinate system t X Foal point C Z O Camera Eulidean oordinate system Y R Image affine oordinate system Z w i X f Optial ray X Image plane i π O i u v Y i U, u ~ Prinipal point = [0, 0, - f ] U 0 U 0a = [u, v, 0] 0 0 T T

10 CAMERA: P 3 P 2 10/29 A sene point X w in the world Eulidean o-ordinate system is a 3 1 vetor. The same point X in the amera Eulidean o-ordinate system is transformed by translation t (vetor) and rotation R (orthogonal matrix). X = x y z = R (X w t)

11 CAMERA: P 3 P 2 (2) 11/29 The point X is projeted to the image plane π as point U. Z x z X f - f x z U = [ fx z, fy z, f ] T, U0a = [u 0, v 0, 0] T.

12 CAMERA: P 3 P 2 (3) 12/29 Projeted point in the 2D image plane π in homogeneous o-ordinates ũ = U V W = = a b u 0 0 v fa fb u 0 0 f v fx z fy z 1 x z y z 1 2D Eulidean ounterpart is u = [u, v] T = [ U W, V W ]T.

13 CALIBRATION MATRIX K 13/29 z ũ = z fa fb u 0 0 f v x z y z 1 = fa fb u 0 0 f v x y z = fa fb u 0 0 f v R (X w t) = KR (X w t) Calibration parameters: intrinsi (matrix K) vs. extrinsi (vetor t, matrix R).

14 PROJECTION MATRIX M 14/29 ũ = U V W = 1 z KR (X w t) = [KR K R t] = M [ Xw 1 ] [ Xw 1 ] = M X w

15 SINGLE CAMERA CALIBRATION, overview 15/29 Intrinsi parameters only - seeking matrix K. Intrinsi + extrinsi parameters - seeking matrix M. 1. Known sene: A set of n non-degenerate (not o-planar) points in the 3D world (e.g., a alibration objet), and the orresponding 2D image points are known. Eah orrespondene between a 3D sene and 2D image point provides one equation [ ] Xj α j ũ j = M Unknown sene: More views are needed to alibrate the amera. The intrinsi amera parameters will not hange for different views, and the orrespondene between image points in different views must be established.

16 CALIBRATION FROM UNKNOWN SCENE (ont.) X 16/29 K R, t K Known amera motion: Three ases aording to the known motion onstraint: (a) Both rotation and translation, general ase. (b) Pure rotation () Pure translation, a linear solution proposed by [Pajdla, Hlaváč 1995]. 2. Unknown amera motion: The most general ase, sometimes alled amera self-alibration. At least three views are needed and the solution is nonlinear. Numerially hard.

17 CAMERA CALIBRATION FROM A KNOWN SCENE (1) 17/29 Typially a two stage proess. 1. Estimate the projetion matrix M is estimated from the o-ordinates of points with known sene positions. 2. The extrinsi and intrinsi parameters are estimated from M. Note: The seond step is not always needed the ase of stereo vision is an example.

18 CAMERA CALIBRATION FROM A KNOWN SCENE (2) Eah orrespondene between sene point X = [x, y, z] T and 2D image point [u, v] T gives one equation αu αv α αu αv α = = m 11 m 12 m 13 m 14 m 21 m 22 m 23 m 24 m 31 m 32 m 33 m 34 x y z 1 m 11 x + m 12 y + m 13 z + m 14 m 21 x + m 22 y + m 23 z + m 24 m 31 x + m 32 y + m 33 z + m 34 18/29

19 CAMERA CALIBRATION FROM A KNOWN SCENE (3) 19/29 u(m 31 x + m 32 y + m 33 z + m 34 ) = m 11 x + m 12 y + m 13 z + m 14 v(m 31 x + m 32 y + m 33 z + m 34 ) = m 21 x + m 22 y + m 23 z + m 24 Two linear equations, eah in 12 unknowns m 11,..., m 34, for eah known orresponding sene and image point (atually only 11 unknowns due to unknown saling). 6 orresponding points needed, at least. If n suh points are available, we an write it as a 2n 12 matrix. x y z ux uy uz u x y z. 1 vx vy vz v m 11 m 12. m 34 = 0 Overonstraint linear system. Robust least squares. Result = M.

20 SVD Singular Value Deomposition SVD is a linear algebra tehnique for solving linear equations in the least square sense. SVD works for singular matries or matries numerially lose to singular. Contained, e.g., in MATLAB. Any m n matrix A, m n an be fatorized as A = UDV T. U has orthonormal olumns, D is non-negative diagonal, and V T has orthonormal rows. SVD loates the losest possible solution in a least square sense. Sometimes need for the losest singular matrix to the original matrix A this dereases the rank from n to n 1. Replae the smallest diagonal element of D by zero. This new matrix is the losest to the original one with respet to the Frobenius norm (whih is alulated as a sum of the squared values of all matrix elements). 20/29

21 SEPARATION OF EXTRINSIC PARAMETERS FROM M Given: projetion matrix M 21/29 Output: rotation matrix R and translation vetor t). M = [KR KR t] = [A b] The 3 3 submatrix is denoted as A, and the rightmost olumn as b. Translation vetor t is easy; A = KR, t = A 1 b. Rotation matrix R. Reall that the alibration matrix K is upper triangular and the rotation matrix is orthogonal. The QR fatorization method or SVD will deompose A into a produt and hene reover K and R.

22 RADIAL DISTORTION AND DE-CENTERING 22/29 PINCUSHION BARREL Often modelled as rotationally symmetri by polynomials. u, v - orret image o-ordinates ũ, ṽ - measured unorreted image o-ordinates û 0, ˆv 0 - estimate of the position of the prinipal point ũ = x û 0, ṽ = y ˆv 0

23 RADIAL DISTORTION (2) 23/29 u = ũ + δu, v = ṽ + δv δu = (ũ u p )(κ 1 r 2 + κ 2 r 4 + κ 3 r 6 ) δv = (ṽ v p )(κ 1 r 2 + κ 2 r 4 + κ 3 r 6 ) r 2 is the square of the radial distane from the enter of the image. r 2 = (ũ u p ) 2 + (ũ u p ) 2 u p, v p are orretions to û 0, ˆv 0 u 0 = û 0 + u p v 0 = ˆv 0 + v p

24 GEOMETRY OF 2 CAMERAS 24/29 Epipoles e, e, epipolar lines l, l. e, e, l, l, C, C, X lie in a single plane. Epipolar geometry. Seeking orrespondenes between two 1D signals. Bilinear relation between u, u.

25 FUNDAMENTAL MATRIX (1) 25/29 Left projetion u and right projetion u of the sene point X. u [K 0] [ X 1 ] u [K R K R t] = K X, [ X 1 = K (R X R t) = K X ] Coplanarity of X, X and t. Distinguish o-ordinates of the left and right ameras by the subsript L, R. Vetor produt.

26 FUNDAMENTAL MATRIX (2) 26/29 Coordinates rotation X R = R X L, and hene X L = R 1 X R. Coplanarity onstraint X T L (t X L) = 0. Preparing for substitution X L = K 1 u, X R = (K ) 1 u, and X L = R 1 (K ) 1 u. Epipolar onstraint in vetor form (K 1 u) T (t R 1 (K ) 1 u ) = 0. Equation is homogeneous with respet to t, so the sale is not determined. Absolute sale annot be reovered without yardstik.

27 FUNDAMENTAL MATRIX (3) 27/29 Replaement of a vetor produt by a matrix multipliation. The translation vetor is t = [t x, t y, t z ] T, and a skew symmetri matrix S(t) (i.e., S T = S) an be reated from it if t 0. S(t) = 0 t z t y t z 0 t x t y t x 0 Note that rank(s) = 2 if and only if t 0.

28 FUNDAMENTAL MATRIX (4) 28/29 The vetor produt an be replaed by the multipliation of two matries. For any regular matrix A, we have t A = S(t) A. Thus we an rewrite the epipolar onstraint in a vetor form (K 1 u) T (S(t) R 1 (K ) 1 u ) = 0, u T (K 1 ) T S(t)R 1 (K ) 1 u = 0.

29 FUNDAMENTAL MATRIX (5) 29/29 The middle part an be onentrated into a single matrix F alled the fundamental matrix of two views, F = (K 1 ) T S(t)R 1 (K ) 1. With the substitution for F we finally get the bilinear relation (sometimes named after Longuet-Higgins) between any two views u T F u = 0. It an be seen that the fundamental matrix F aptures all information that an be reovered from a pair of images if the orrespondene problem is solved.

30

31

32 Foal point Image plane Horizon Vanishing point Optial axis Base plane

33 Optial axis Z w O w Y w Sene point X w X X w World Eulidean oordinate system t X Foal point C Z O Camera Eulidean oordinate system Y R Image affine oordinate system Z w i X f Optial ray X Image plane i π O i u v Y i U, u ~ Prinipal point = [0, 0, - f ] U 0 U 0a = [u, v, 0] 0 0 T T

34 Z x z X f - f x z

35 X K R, t K 1 2

36

37 PINCUSHION BARREL

38