Robert Collins CSE486, Penn State. Lecture 25: Structure from Motion

Transcription

1 Lecture 25: Structure from Motion

2 Structure from Motion Given a set of flow fields or displacement vectors from a moving camera over time, determine: the sequence of camera poses the 3D structure of the scene Sequence of camera poses Scene structure

3 SFM Killer App Match Move Track a set of feature points through a movie sequence Deduce where the cameras are and the 3D locations of the points that were tracked Render snthetic objects with respect to the deduced 3D geometr of the scene / cameras

4 Match Move Eamples Harts War and Graham Kimpton eamples from MatchMover Professional galler. Coprighted.

5 Factorization Tomasi and Kanade, Shape and Motion from Image Streams under Orthograph, International Journal of Computer Vision (IJCV), Vol 9, pp.37-54, 992. Goal: combine point correspondence information from multiple points over multiple frames to solve for scene structure and camera motion (structure from motion) Approach: numericall stable approach based on using SVD to factor matri of observed point positions. Historical significance: until that time, most SFM work dealt with minimal configurations, and noise-free data. Factorization was one of the first practical SFM algorithms

6 CSE486, Penn Recall State : World to Camera Transform P C = R ( P W - C ) C P C P C P z r r 2 r 3 r 2 r 22 r 32 r 3 r 23 r 33 c c c z W P W P W P z P C = M et. P W

7 Perspective Projection X Y O Z p f P f f X Z Y Z Non-linear linear equations An An point on the ra OP has image p!!

8 Perspective Projection f f X Z Y Z Perspective Projection : parallel lines appear to meet at a vanishing point; farther objects seem smaller O.Camps, PSU

9 Simplification: Weak Perspective f X Zo f Y Zo Weak perspective = Parallel projection (parallel lines remain parallel) + Scaling to simulate change in size due to object distance.

10 Simpler: Orthographic Projection X Y Pure parallel projection. Highl simplified case where we even ignore the scaling due to distance.

11 Robert Collins Perspective Matri Equation (Camera Coordinates) Z Y f Z X f Using homogeneous coordinates: Using homogeneous coordinates: z z Z Y X f f z

12 Robert Collins Weak Perspective Approimation Using homogeneous coordinates: Using homogeneous coordinates: z z Z Y X f f z o Z X f o Z Y f Z Z Z

13 Robert Collins Let s Consider Orthographic Using homogeneous coordinates: Using homogeneous coordinates: z z Z Y X f f z X Y Z

14 Combine with Eternal Params r 2 r r 3 c r 2 r 22 r 23 c r 3 r 32 r 33 cz W P W P W P z r r 2 r 2 r 22 r 3 r 23 c c c z W P W P W P z

15 Combine with Eternal Params r r 2 r 2 r 22 r 3 r 23 c c c z W P W P W P z r r 2 r 3 r 2 r 22 r 23 W P W P P W z c c c z

16 Orthographic: Algebraic Equation i T r r 2 r 3 r 2 r 22 r 23 j T P W P W P P W z T c c c z = i T ( P - T ) = j T ( P - T )

17 Multiple Points, Multiple Frames Notation (attack of the killer subscripts) = i T ( P - T ) = j T ( P - T ) N points P P 2 P j P N F frames i i 2 i i i F j j 2 j i j F T T 2 T i T F ij = i it ( P j - T i ) ij = j it ( P j - T i ) Eq T&V book

18 Factorization Approach ij = i it ( P j - T i ) ij = j it ( P j - T i ) N points P P 2 P j P N (We want to recover these) Note that absolute position of the set of points is something that cannot be uniquel recovered, so First Trick: set the origin of the world coordinate sstem to be the center of pass of the N points! N N

19 Factorization Approach ij = i it ( P j - T i ) ij = j it ( P j - T i ) Centroid at : N N Implication: N N N it N N N Note: this is the center of mass of coordinates in frame t

20 Factorization Approach Second Trick: subtract off the center of mass of the 2D points in each frame. (Centering) ij = i it ( P j - T i ) ij = j it ( P j - T i )

21 Factorization Approach ij = i it ( P j - T i ) centering ij = j it ( P j - T i ) What have we accomplished so far? ) Removed unknown camera locations from equations. 2) More importantl, we can now write everthing As a big matri equation

22 Factorization Approach Form a matri of centered image points. 2FN 2 3 N All N points in one frame F F2 F3 FN 2 3 N F F2 F3 FN

23 Factorization Approach Form a matri of centered image points. 2FN 2 3 N Tracking one point through all F frames F F2 F3 FN 2 3 N F F2 F3 FN

24 CSE486, Penn Factorization State Approach matri of centered image points: it it it it 2FN 2 3 N 2F3 i T 3N F F2 F3 FN 2 3 N = i T F j T P P 2 P N F F2 F3 FN j F T

25 Factorization Approach 2F N 2F 3 3 N W = M S Centered measurement matri Motion (camera rotation) Structure (3D scene points)

26 Factorization Approach 2F N 2F 3 3 N W = M S Rank Theorem: The 2FN centered observation matri has at most rank 3. Proof: Trivial, using the properties: rank of mn matri is at most min(m,n) rank of A*B is at most min(rank(a),rank(b))

27 Rank of a Matri What is rank of a matri, anwas? Number of columns (rows) that are linearl independent. If matri A is treated as a linear map, it is the intrinsic dimension of the space that is mapped into. M-dimensional space MN matri A N-dimensional space This matri would have rank

28 Factorization Rank Theorem Importance of rank theorem: Shows that video data is highl redundant Precisel quantifies the redundanc Suggests an algorithm for solving SFM

29 Factorization Approach Form SVD of measurement matri W 2FN 2F2F 2FN NN W = U D V T Diagonal matri with eigenvalues sorted in decreasing order: d >= d 22 >= d 33 >=

30 Factorization Approach Form SVD of measurement matri W 2FN 2F2F 2FN NN W = U D V T Another useful rank propert: Rank of a matri is equal to the number of nonzero eigenvalues. d, d 22, d 33 are onl nonzero eigenvalues (the rest are ).

31 Factorization Approach 2FN 2F2F 2FN NN = * * Eigenvalues in decreasing order

32 Factorization Approach 2FN 2F2F 2FN NN = * * Rank theorem sas: These 3 are nonzero These should be zero In practice, due to noise, there ma be more than 3 nonzero eigenvalues, but rank theorem tells us to ignore all but the largest three.

33 Factorization Approach 2FN 2F2F 2FN NN 2F3 33 3N = * * W = U D V T

34 Factorization Approach Observed image points W = SVD U D V T W = U D /2 D /2 V T 2FN 2F3 3N W = M S Camera motion Scene structure

35 Annoing Details W = (U D /2 )(D /2 V T ) 2FN 2F3 3N W = M S Problems: ) This is not a unique decomposition. eg: (M Q) (Q - S) = M Q Q - S = M S 2) i T, j T pairs (rows of M) are not necessaril orthogonal

36 Solving the Annoing Details Solution to both problems: Solve for Q such that appropriate rows of M satisf unit vectors 3N equations in 9 unknowns orthogonal But these are nonlinear equations linearize and iterate (see Eercise 8.8 in book for Newton s method) (alternative approach is to use Cholesk decomposition outside our scope)

37 Factorization Summar Assumptions - orthographic camera - N non-coplanar points tracking in F>=3 frames Form the centered measurement matri W=[X ; Y] - where ij = ij m j - where ij = ij m j - m j and m j are mean of points in frame i - j ranges over set of points Rank theorem: The centered measurement matri has a rank of at most 3

38 Factorization Algorithm ) Form the centered measurement matri W from N points tracked over F frames. 2) Compute SVD of W = U D V T - U is 2F2F - D is 2FN - V T is NN 3) Take largest 3 eigenvalues, and form - D = 33 diagonal matri of largest eigenvalues - U = 2F3 matri of corresponding column vectors from U - V T = 3N matri of corresponding row vectors from V T 4) Define M = U D /2 and S = D /2 V T 5) Solve for Q that makes appropriate rows of M orthogonal 6) Final solution is M* = M Q and S* = Q - S

39 Sample Results QuickTime and a Cinepak decompressor are needed to see this picture.

40 Sample Results QuickTime and a Cinepak decompressor are needed to see this picture.