Introduction to motion correspondence 1 IPAM - UCLA July 24, 2013 Iasonas Kokkinos Center for Visual Computing Ecole Centrale Paris / INRIA Saclay
Why estimate visual motion? 2 Tracking Segmentation Structure from motion Action recognition
Action recognition and motion 3 http://astro.temple.edu/~tshipley/mocap/dotmovie.html G. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.
Action recognition and motion 4 http://astro.temple.edu/~tshipley/mocap/dotmovie.html G. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.
Action recognition and motion 5 http://astro.temple.edu/~tshipley/mocap/dotmovie.html G. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, 201-211, 1973.
Motion Field 6 P(t) is a moving 3D point Velocity of scene point: V = dp/dt P(t) V P(t+dt) p(t) = (x(t),y(t)) is the projection of P in the image. Apparent velocity v in the image: given by components u = dx/dt and v = dy/dt Considering dt =1: p(t+1) = p(t) + (u,v) p(t) v p(t+dt) Motion estimation task: Estimate (u,v) Slide credit: S. Lazebnik 6
Brightness constancy constraint 7 Image is projection of 3D environment Each pixel: projection of a surface patch Pixel intensity influenced by 3D surface, incident light, camera Assumption: intensity of surface patch remains constant Slide credit: Georg Langs 7
Brightness constancy constraint 8 Optical flow estimation: (x + u(x, y),y+ v(x, y)) Constraint: I(x, y, t Taylor expansion of RHS: I(x, y, t t t 1 1) = I(x + u(x, y),y+ v(x, y),t) 1) ' I(x, y, t)+ @I @x Brightness Constancy Constraint: I x u + I y v + I t =0 (x, y) u(x, y)+@iv(x, y) @y
Optical flow estimation - 2D to 1D 9 Two input images Slide credit: Georg Langs
Optical flow estimation - 1D case 10 How can we estimate the displacement? Slide credit: Georg Langs
Optical Flow Estimation 1D case 11 Known: Gradient, Difference of intensities at x-d Unknown: d Slide credit: Georg Langs How about 2D?
Brightness constraint: not enough for 2D! I x u + I y v + I t = 0 12 I I x u + I y v + I t =0 ( u, v(u ) + 0,v I 0 )=(u, = 0 v)+c( I y,i x ) t unknown flow parallel to the edge ri (u, v)+i t =0 I ( u', v') = 0
The Aperture Problem 13 Perceived motion 13
The Aperture Problem 14 Actual motion 14
Optical flow uncertaintly 15
Overcoming the aperture effect 16 Brightness constancy constraint I x (p i )u i + I x (p i )v i + I t (p i )=0
Overcoming the aperture effect 17 united we move Brightness constancy constraint I x (p i )u + I x (p i )v + I t (p i )=0
Lucas-Kanade 2 6 4 I x (p i )u + I x (p i )v + I t (p i )=0 I x,1 I y,1. I x,25 I y,25 3 7 5 apple u v = 2 6 4 I t,1. I t,25 3 7 5 18 Rewrite: Residuals: Cost: Minimization: Au = b = Au T = b T b b A T Au = A T b 25 equations, 2 unknows 2u T A T b + u T A T Au B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. IJCAI, 1981.
Lucas-Kanade, continued 19 A T Au = A T b Is it invertible? A = u = A T A 2 6 4 I x,1 I y,1... 1 A T b 3 7 5 A T A = apple I x,25 I P x,25 P i I2 x,i Pi I x,ii y,i P i I x,ii y,i i I2 y,i B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. IJCAI, 1981.
Second Moment Matrix 20 Distribution of gradients: J = apple P x 0,y 0 Ix 2 P x 0,y 0 I x I y Px P 0,y 0 I x I y x 0,y 0 I 2 y
Second Moment Matrix 21 Distribution of gradients: J = X x 0,y 0 (I x,i y ) T (I x,i y )
Second Moment Matrix 22 Distribution of gradients: J = X x 0,y 0 (rg u) T (rg u)
Second Moment Matrix 23 Distribution of gradients: J = G h i (rg u) T (rg u)
Second Moment Matrix 24 J = G h i (rg u) T (rg u) Eigenvectors : directions of maximal and minimal variation of u Eigenvalues: amounts of minimal and maximal variation u
Local structure and motion estimation 25 1 ' 0, 2 ' 0 1 large, 2 ' 0 1, 2 : large
Interpreting the eigenvalues 26 Classification of image points using eigenvalues of the second moment matrix: λ 2 Edge λ 2 >> λ 1 Corner λ 1 and λ 2 are large, λ 1 ~ λ 2 λ 1 and λ 2 are small Flat region Edge λ 1 >> λ 2 Slide credit: K. Grauman λ 1
SSD measure 27 Sum of squared differences
High-Texture Region 28 Gradients are different, large magnitude Large λ 1, large λ 2 28
Edge 29 Gradients very large or very small Large λ 1, small λ 2 29
Low-Texture Region 30 Gradients have small magnitude Small λ 1, small λ 2 30
Shi-Tomasi feature tracker 31 1. Find good features (min eigenvalue of 2 2 Hessian) 2. Use Lucas-Kanade to track with pure translation 3. Use affine registration with first feature patch 4. Terminate tracks whose dissimilarity gets too large 5. Start new tracks when needed
Tracking example 32 J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.
Lucas-Kanade and the aperture effect 33 Brightness constancy + neighboring pixels have same (u,v) I x (p i )u + I x (p i )v + I t (p i )=0 5x5 window: 25 equations, 2 unknowns Solve the system around each pixel separately
Dense Lukas Kanade 34 34 Lucas Kanade Horn Schunk Anisotropic
Horn-Schunk and the aperture effect 35 Brightness constancy + flow smoothness: J(u, v) = 1 2 ZZ Minimize with respect to u(x, y),v(x, y) Euler-Lagrange derivative: [I t + ri (u, v)] 2 + (ru) 2 +(rv) 2 dxdy Minimum condition B.K.P. Horn and B.G. Schunck, "Determining optical flow." Artificial Intelligence, 1981.
Horn-Schunk results 36
Lucas-Kanade revisited 37 General expression for LK criterion: LK flow How does this compare with Horn-Schunk criterion?
Lucas-Kanade meets 2 3Horn-Schunk Introduce: w. u = 4 v 5 r 3 f =. 1 rw 2. = ru 2 + rv 2 Lucas-Kanade Horn-Schunk Bruhn-Weickert-Schnoerr Z E LK (w) = Z E HS (w) = Z E BWS (w) = J 2 4 f x f y f t 3 5 38. = G r 3 fr 3 f T w T J w dxdy w T J 0 w + a rw dxdy w T J w + a rw dxdy A. Bruhn, J. Weickert, C. Schnörr: 'Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods.' IJCV 2005
Horn-Schunk 39
Bruhn-Weickert-Schnoerr 40
Bruhn-Weickert-Schnoerr, continued Z E BWS (w) = w T J w + a rw dxdy 41 E 0 BWS(w) = Z Spatio-temporal regularization [0,T ] w T J w + a r 3 w dxdy Robust norms: E 00 BWS(w) = Z [0,T ] 1 w T J w + a 2 ( r 3 w )dxdy
Bruhn-Weickert-Schnoerr 42
Bruhn-Weickert-Schnoerr - anisotropic 43
Bruhn-Weickert-Schnoerr flow regularization 44
Numerical solutions 45 Euler-Lagrange: Numerical approximation to Laplacian Sparse linear system in u,v Gauss-Seidel, Successive Over Relaxation (SOR) Multigrid 12 fps, 2006 A. Brandt, Multi-level adaptive solutions to boundary-value problems, Math. Comput., 1977. D. Terzopoulos, Image Analysis Using Multigrid Relaxation Methods, PAMI, 1986 A. Kenigsberg, R. Kimmel, and I. Yavneh, A Multigrid Approach for Fast Geodesic Active Contours, 2001 G. Papandreou and P. Maragos, "Multigrid Geometric Active Contour Models", TIP 2007 A. Bruhn, J. Weickert, T. Kohlberger, C. Schnörr: A multigrid platform for real-time motion computation with discontinuity-preserving variational methods. IJCV 2006
Optical Flow: Iterative Estimation 46 estimate update Initial guess: Estimate: x 0 x (using d for displacement here instead of u)
Optical Flow: Iterative Estimation 47 estimate update Initial guess: Estimate: x 0 x
Optical Flow: Iterative Estimation 48 estimate update Initial guess: Estimate: x 0 x
Optical Flow: Iterative Estimation 49 x 0 x
Large Displacements: Reduce Resolution! 50
Coarse-to-fine Optical Flow Estimation 51 u=1.25 pixels u=2.5 pixels u=5 pixels Image 1 u=10 pixels Image 2 Gaussian pyramid of image 1 Gaussian pyramid of image 2 51
Coarse-to-fine Optical Flow Estimation 52 Run iterative OF Run iterative OF Warp & upsample... Image 1 Image 2 Gaussian pyramid of image 1 Gaussian pyramid of image 2 52
Beyond the brightness constancy constraint 53 SIFT flow: dense correspondence across different scenes, C Liu, J Yuen, A Torralba, J. Sivic, W. Freeman, ECCV 2008 T. Brox, J. Malik, Large displacement optical flow: descriptor matching in variational motion estimation, PAMI 2011
Beyond the brightness constancy constraint 54 E. Trulls, I. Kokkinos, A. Sanfeliu, and F. Moreno, Dense Segmentation-Aware Descriptors, CVPR 2013
Motion-based action recognition Discovering Discriminative Action Parts from Mid-Level Video Representations, M. Raptis, I. Kokkinos and S. Soatto. CVPR, 2012. 55
Motion-based action recognition Discovering Discriminative Action Parts from Mid-Level Video Representations, M. Raptis, I. Kokkinos and S. Soatto. CVPR 12. 56
Motion-based action recognition Discovering Discriminative Action Parts from Mid-Level Video Representations, M. Raptis, I. Kokkinos and S. Soatto. CVPR, 2012. 57
Motion-based action recognition Discovering Discriminative Action Parts from Mid-Level Video Representations, M. Raptis, I. Kokkinos and S. Soatto. CVPR, 2012. 58
Motion-based segmentation 59 Unsupervised segmentation incorporating colour, texture, and motion. T Brox, M Rousson, R Deriche, J Weickert, IVC 2010
Further Literature Pointers 60 Layered motion, parametric models, motion segmentation: Layered Representation for Motion Analysis. CVPR 1993. J. Wang and E. Adelson Black, M. J. and Anandan, P., The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields, CVIU 1996 D. Cremers, S. Soatto, Motion Competition: A Variational Approach to Piecewise Parametric. Motion Segmentation, IJCV 2004 Unsupervised segmentation incorporating colour, texture, and motion. T Brox, M Rousson, R Deriche, J Weickert, IVC 2010 (wait for R. Szeliski s talk) Learning: Roth, S., Black, M.J.: On the spatial statistics of optical flow. IJCV 74, 33 50 (2007) Discrete optimization: (W. Freeman s talk today) Fusion Moves for Markov Random Field Optimization. V. Lempitsky, C. Rother, S. Roth, and A. Blake, PAMI 2010. B. Glocker, N. Komodakis, G. Tziritas, N. Navab, N. Paragios Dense Image Registration through MRFs and Efficient Linear Programming MIA, 2008
Code 61 Secrets of optical flow estimation and their principles, Sun., Roth., and Black, CVPR 10 T. Brox, J. Malik, Large displacement optical flow: descriptor matching in variational motion estimation, PAMI 2011 Action recognition & descriptor works: http://vision.mas.ecp.fr/personnel/iasonas/code.html Registration service: http://cvn.ecp.fr/ www.ipol.im