MIN-Fakultät Fachbereich Informatik Arbeitsbereich SAV/BV (KOGS) Image Processing 1 (IP1) Bildverarbeitung 1 Lecture 18 Mo<on Analysis 2 Winter Semester 2014/15 Dr. Benjamin Seppke Prof. Siegfried S<ehl
IP1 Lecture 18: Mo<on Analysis 2 Opcal Flow and Segmentaon The op<cal flow smoothness constraint is not valid at occluding boundaries (silhoueqes). In order to inhibit the constraint, one may try to segment the image based on op<cal flow discon<nui<es while performing the itera<ons. (From B.K.P. Horn, Robot Vision, 1986) Checkered sphere rota<ng before randomly textured background 11.12.14 1. iteraon 4. iteraon 16. iteraon 64. iteraon final result ideal result University of Hamburg, Dept. Informatics 2
Opcal Flow PaGerns Complex op<cal flow fields may be segmented into components which show a consistent qualita<ve paqern. Qualitave flow pagerns: translaon at constant distance translaon in depth rotaon at constant distance rotaon about axis parallel to image plane General transla<on results in a flow paqern with a focus of expansion (FOE): FOE As the direc<on of mo<on changes, the FOE changes its loca<on. 3
Opcal Flow and 3D Moon I In general, op<cal flow may be caused by an unknown 3D mo<on of an unkown surface. How do the flow components u, v depend on the 3D mo<on parameters? Assume camera mo<on in a sta<c scene, op<cal axis = z- axis, rota<on about the origin. rotaon y axis ω x x v opcal center z image plane at f=1 2D velocity opcal axis 3D velocity of a point is determined by rota<onal velocity and transla<onal velocity t: v = t ω r r y v = u v 3D velocity v = x scene point r = y z ω u v w 4
Opcal Flow and 3D Moon II By taking the component form of v = t ω r with, t T = ( t x t y t z ) and r T = ( x y z) and compu<ng the perspec<ve projec<on we get u = x z xz z = t x 2 z b + c y ( x t z z a y + b x ( v = y z yz z = t y 2 z c x + a ( y t z z a y + b x ( Observa<on of u and v at loca<on (x, y ) gives 2 equa<ons for 7 unknowns. Note that mo<on of a point at distance kz with transla<on kt and the same rota<on ω will give the same op<cal flow, k any scale factor. The transla<onal and rota<onal parts may be separated: u translation = t x + x t z z v translation = t + y t y z z u rotation =a x y -b( x 2 +1)+ c y v rotation =a( y 2 +1)-b x y + c x For pure transla<on we have 2 equa<ons for 3 unknows (z fixed arbitrarily). ω T = a b c ( ) 5
3D Moon Analysis Based on 2D Point Displacements 2D displacements of points observed on an unknown moving rigid body may provide informa<on about the 3D structure of the points the 3D mo<on parameters Cases of interest: sta<onary camera, moving object(s) moving camera, sta<onary object(s) moving camera, moving object(s) Rotang cylinder experiment by S. Ullman (1981) camera moon parameters may be known 6
Structure from Moon I Ullman showed 1979 that the spa<al structure of 4 rigidly connected non- coplanar points may be recovered from 3 orthographic projec<ons. x 1 O z x C c b a a 1 B y A o projecon plane P 1 y 1 O, A, B, C 4 rigid points a, b, c vectors to A, B, C Π 1, Π 2, Π 3 projec<on planes The 3 projec<on planes intersect and form a tetrahedron. u 12, u 23, u 31 are unit vectors along the intersec<ons. The idea is to determine the from the observed coordinates x i, y i coordinate axes of P i a i, b i, c i coordinate pairs of points A, B, C in projec<on plane Π i The problem is to determine the spa<al orienta<ons of Π 1, Π 2, Π 3 from the 9 projec<on coordinate pairs a i, b i, c i, i = 1, 2, 3. u 23 ui, j a i, b i, c i u 12 O u31 7
The projec<on coordinates are: a 1x = a T x1 Structure from Moon II a 1y = a T y1 b 1x = b T x1 b 1y = b T y1 c 1x = c T x1 Since each u ij u ij = α ij xi + β y i u ij = γ ij x j +δ y j Mul<plying with c 1y = c T y1 lies in both planes Π i and Π j, it can be wriqen as ( a i ) T, ( b i ) T, ( c i ) T α ij xi + β y i = γ ij x j +δ y j we get α ij a ix + βa iy α ij b ix + βb iy α ij c ix + βc iy = γ ij a jx +δa jy = γ ij b jx +δb jy = γ ij c jx +δc jy Solve for α ij, β ij, γ ij, δ ij. using the constraints α ij 2 + β ij 2 = 1 and γ ij 2 + δ ij 2 = 1 8
Structure from Moon III From the coefficients α ij, β ij, γ ij, δ ij one can compute the distances between the 3 unit vectors u 12, u 23, u 31 : d 1 = u 23 u 12 = ( α 23 α 12 ) x i + ( β 23 β 12 ) y i = ( α 23 α 12 ) 2 + ( β 23 β 12 ) 2 d 2 = ( α 31 α 23 ) 2 + ( β 31 β 23 ) 2 d 3 = ( α 12 α 31 ) 2 + ( β 12 β 31 ) 2 Hence the rela<ve angles of the projec<on planes are determined. d 1 d 2 The spa<al posi<ons of A, B, C rela<ve to the projec<on planes (and to the origin O) can be determined by intersec<ng the projec<on rays perpendicular on the projected points a i, b i, c i. d 3 O 9
Perspecve 3D Analysis of Point Displacements rela<ve mo<on of one rigid object and one camera observa<on of P points in M views For P points in M images we have 3MP 6(M-1) MP 3(M-1)P 3MP v p For each point in 2 consecu<ve images we have: v p,m+1 = R m vp,m + t m mo<on equa<on v p,m = λ p,m vp,m projec<on equa<on unknown 3D point coordinates unkown mo<on parameters R m and unknown projec<on parameters λ p,m mo<on equa<ons projec<on equa<ons 1 arbitrary scaling parameter v p,m 2 equa<ons unknowns à P 3+ à 2M 3 tm M P 2 5 3 4 4 4 5 4 v 2,m v 1,m v3,m v 1,mv3,m v 2,m 10
Essenal Matrix Geometrical constraints derived from 2 views of a point in mo<on mo<on between image m and m+1 may be decomposed into 1. rota<on R m about origin of coordinate system (= op<cal center) 2. transla<on t m observa<ons are given by direc<on vectors and nm+1 along projec<on rays. n m R m nm, t m and nm+1 are coplanar: tm R m nm Aier some manipula<on: E = essen<al matrix with E m = and y n m ( ) T n m+1 = 0 n m E m n m+1 = 0 x n m+1 t m r 1 tm r 2 tm r 3 v m R m t m R m = vm+1 r 1 r2 r3 z 11
( n m ) T E m nm+1 = 0 But: Solving for the Essenal Matrix formally one equa<on for 9 unknowns e ij only 6 degrees of freedom (3 rota<on angles, 3 transla<on components) e ij can only be determined up to a scale factor Basic solu<on approach: observe P points in 2 views, P >> 8 fix e 11 arbitrarily solve an overconstrained system of equa<ons for the other 8 unknown coefficients e ij E may be decomposed into S and R by Singular Value Decomposi<on (SVD). E may be wrigen as E = S R -1 with R = rot. matrix and Note: S (and therefore E) has rank 2 S = 0 t z t y t z 0 t x t y t x 0 12
Singular Value Decomposion of E Any m n matrix A, m n, may be decomposed as A = U D V T where U has orthonormal columns m n D is non- nega<ve diagonal n n V T has orthonormal rows n n This can be applied to E to give E = U D V T with: R = U G V T or R = U G T V T S = V Z V T 0 1 0 where G = 1 0 0 and Z 0 0 1 = 0 1 0 1 0 0 0 0 1 13
Alternave 3D Moon Constraint Nagel and Neumann 82 y Consider 2 views of 3 points v p,m, p = 1... 3, m = 1, 2 The planes through all intersect in t m R m np,m and à the normals of the planes are coplanar np,m+1 Coplanarity condi<on for 3 vectors a, b, c : n p,m hence: Nonlinear equa<on with 3 unknown rota<on parameters. => Observa<on of at least 5 points required to solve for the unknowns. x n p,m+1 v p,m a b ( ) T c = 0 R m n1,m n (( 1,m+1 ) ( R m n2,m n 2,m+1 )) T ( R m n3,m n 3,m+1 )= 0 R m t m vp,m+1 z 14
Reminder: Homogeneous Coordinates (N+1)- dimensional nota<on for points in N- dimensional Euclidean space allows to express projec<on and transla<on as linear opera<ons Normal coordinates: v T = ( x y z) Homogeneous coordinates: v T = wx wy wz w w 0 is arbitrary constant ( ) Rota<on and transla<on in homogeneous coordinates: R t v = A v with A = 0 1 Projec<on in homogeneous coordinates: f 0 0 v = B v with B = 0 f 0 0 0 1 Divide the first N components by the (N+1)th component to recover normal coordinates 15
From Homogeneous World Coordinates to Homogeneous Image Coordinates v T = ( x y z) v p x p y p ( ) T = ( ) w x p w y p w = K R K t fa fb x p0 K = 0 fc y p0 0 0 1 t M = 3 x 4 projecve matrix scene coordinates image coordinates R, extrinsic camera parameters x y z 1 = M x y z 1 intrinsic camera parameters (camera calibraon matrix ) v p = M v fa = scaling in x P - axis fc = scaling in y P - axis fb = slant of axes x P0, y P0 = principal point (opcal center in image plane) 16