Introduction to motion correspondence

Transcription

1 Introduction to motion correspondence 1 IPAM - UCLA July 24, 2013 Iasonas Kokkinos Center for Visual Computing Ecole Centrale Paris / INRIA Saclay

2 Why estimate visual motion? 2 Tracking Segmentation Structure from motion Action recognition

3 Action recognition and motion 3 G. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, , 1973.

4 Action recognition and motion 4 G. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, , 1973.

5 Action recognition and motion 5 G. Johansson, Visual Perception of Biological Motion and a Model For Its Analysis", Perception and Psychophysics 14, , 1973.

6 Motion Field 6 P(t) is a moving 3D point Velocity of scene point: V = dp/dt P(t) V P(t+dt) p(t) = (x(t),y(t)) is the projection of P in the image. Apparent velocity v in the image: given by components u = dx/dt and v = dy/dt Considering dt =1: p(t+1) = p(t) + (u,v) p(t) v p(t+dt) Motion estimation task: Estimate (u,v) Slide credit: S. Lazebnik 6

7 Brightness constancy constraint 7 Image is projection of 3D environment Each pixel: projection of a surface patch Pixel intensity influenced by 3D surface, incident light, camera Assumption: intensity of surface patch remains constant Slide credit: Georg Langs 7

8 Brightness constancy constraint 8 Optical flow estimation: (x + u(x, y),y+ v(x, y)) Constraint: I(x, y, t Taylor expansion of RHS: I(x, y, t t t 1 1) = I(x + u(x, y),y+ v(x, y),t) 1) ' I(x, Brightness Constancy Constraint: I x u + I y v + I t =0 (x, y) u(x, y)+@iv(x,

9 Optical flow estimation - 2D to 1D 9 Two input images Slide credit: Georg Langs

10 Optical flow estimation - 1D case 10 How can we estimate the displacement? Slide credit: Georg Langs

11 Optical Flow Estimation 1D case 11 Known: Gradient, Difference of intensities at x-d Unknown: d Slide credit: Georg Langs How about 2D?

12 Brightness constraint: not enough for 2D! I x u + I y v + I t = 0 12 I I x u + I y v + I t =0 ( u, v(u ) + 0,v I 0 )=(u, = 0 v)+c( I y,i x ) t unknown flow parallel to the edge ri (u, v)+i t =0 I ( u', v') = 0

13 The Aperture Problem 13 Perceived motion 13

14 The Aperture Problem 14 Actual motion 14

15 Optical flow uncertaintly 15

16 Overcoming the aperture effect 16 Brightness constancy constraint I x (p i )u i + I x (p i )v i + I t (p i )=0

17 Overcoming the aperture effect 17 united we move Brightness constancy constraint I x (p i )u + I x (p i )v + I t (p i )=0

18 Lucas-Kanade I x (p i )u + I x (p i )v + I t (p i )=0 I x,1 I y,1. I x,25 I y, apple u v = I t,1. I t, Rewrite: Residuals: Cost: Minimization: Au = b = Au T = b T b b A T Au = A T b 25 equations, 2 unknows 2u T A T b + u T A T Au B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. IJCAI, 1981.

19 Lucas-Kanade, continued 19 A T Au = A T b Is it invertible? A = u = A T A I x,1 I y, A T b A T A = apple I x,25 I P x,25 P i I2 x,i Pi I x,ii y,i P i I x,ii y,i i I2 y,i B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. IJCAI, 1981.

20 Second Moment Matrix 20 Distribution of gradients: J = apple P x 0,y 0 Ix 2 P x 0,y 0 I x I y Px P 0,y 0 I x I y x 0,y 0 I 2 y

21 Second Moment Matrix 21 Distribution of gradients: J = X x 0,y 0 (I x,i y ) T (I x,i y )

22 Second Moment Matrix 22 Distribution of gradients: J = X x 0,y 0 (rg u) T (rg u)

23 Second Moment Matrix 23 Distribution of gradients: J = G h i (rg u) T (rg u)

24 Second Moment Matrix 24 J = G h i (rg u) T (rg u) Eigenvectors : directions of maximal and minimal variation of u Eigenvalues: amounts of minimal and maximal variation u

25 Local structure and motion estimation 25 1 ' 0, 2 ' 0 1 large, 2 ' 0 1, 2 : large

26 Interpreting the eigenvalues 26 Classification of image points using eigenvalues of the second moment matrix: λ 2 Edge λ 2 >> λ 1 Corner λ 1 and λ 2 are large, λ 1 ~ λ 2 λ 1 and λ 2 are small Flat region Edge λ 1 >> λ 2 Slide credit: K. Grauman λ 1

27 SSD measure 27 Sum of squared differences

28 High-Texture Region 28 Gradients are different, large magnitude Large λ 1, large λ 2 28

29 Edge 29 Gradients very large or very small Large λ 1, small λ 2 29

30 Low-Texture Region 30 Gradients have small magnitude Small λ 1, small λ 2 30

31 Shi-Tomasi feature tracker Find good features (min eigenvalue of 2 2 Hessian) 2. Use Lucas-Kanade to track with pure translation 3. Use affine registration with first feature patch 4. Terminate tracks whose dissimilarity gets too large 5. Start new tracks when needed

32 Tracking example 32 J. Shi and C. Tomasi. Good Features to Track. CVPR 1994.

33 Lucas-Kanade and the aperture effect 33 Brightness constancy + neighboring pixels have same (u,v) I x (p i )u + I x (p i )v + I t (p i )=0 5x5 window: 25 equations, 2 unknowns Solve the system around each pixel separately

34 Dense Lukas Kanade Lucas Kanade Horn Schunk Anisotropic

35 Horn-Schunk and the aperture effect 35 Brightness constancy + flow smoothness: J(u, v) = 1 2 ZZ Minimize with respect to u(x, y),v(x, y) Euler-Lagrange derivative: [I t + ri (u, v)] 2 + (ru) 2 +(rv) 2 dxdy Minimum condition B.K.P. Horn and B.G. Schunck, "Determining optical flow." Artificial Intelligence, 1981.

36 Horn-Schunk results 36

37 Lucas-Kanade revisited 37 General expression for LK criterion: LK flow How does this compare with Horn-Schunk criterion?

38 Lucas-Kanade meets 2 3Horn-Schunk Introduce: w. u = 4 v 5 r 3 f =. 1 rw 2. = ru 2 + rv 2 Lucas-Kanade Horn-Schunk Bruhn-Weickert-Schnoerr Z E LK (w) = Z E HS (w) = Z E BWS (w) = J 2 4 f x f y f t = G r 3 fr 3 f T w T J w dxdy w T J 0 w + a rw dxdy w T J w + a rw dxdy A. Bruhn, J. Weickert, C. Schnörr: 'Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods.' IJCV 2005

39 Horn-Schunk 39

40 Bruhn-Weickert-Schnoerr 40

41 Bruhn-Weickert-Schnoerr, continued Z E BWS (w) = w T J w + a rw dxdy 41 E 0 BWS(w) = Z Spatio-temporal regularization [0,T ] w T J w + a r 3 w dxdy Robust norms: E 00 BWS(w) = Z [0,T ] 1 w T J w + a 2 ( r 3 w )dxdy

42 Bruhn-Weickert-Schnoerr 42

43 Bruhn-Weickert-Schnoerr - anisotropic 43

44 Bruhn-Weickert-Schnoerr flow regularization 44

45 Numerical solutions 45 Euler-Lagrange: Numerical approximation to Laplacian Sparse linear system in u,v Gauss-Seidel, Successive Over Relaxation (SOR) Multigrid 12 fps, 2006 A. Brandt, Multi-level adaptive solutions to boundary-value problems, Math. Comput., D. Terzopoulos, Image Analysis Using Multigrid Relaxation Methods, PAMI, 1986 A. Kenigsberg, R. Kimmel, and I. Yavneh, A Multigrid Approach for Fast Geodesic Active Contours, 2001 G. Papandreou and P. Maragos, "Multigrid Geometric Active Contour Models", TIP 2007 A. Bruhn, J. Weickert, T. Kohlberger, C. Schnörr: A multigrid platform for real-time motion computation with discontinuity-preserving variational methods. IJCV 2006

46 Optical Flow: Iterative Estimation 46 estimate update Initial guess: Estimate: x 0 x (using d for displacement here instead of u)

47 Optical Flow: Iterative Estimation 47 estimate update Initial guess: Estimate: x 0 x

48 Optical Flow: Iterative Estimation 48 estimate update Initial guess: Estimate: x 0 x

49 Optical Flow: Iterative Estimation 49 x 0 x

50 Large Displacements: Reduce Resolution! 50

51 Coarse-to-fine Optical Flow Estimation 51 u=1.25 pixels u=2.5 pixels u=5 pixels Image 1 u=10 pixels Image 2 Gaussian pyramid of image 1 Gaussian pyramid of image 2 51

52 Coarse-to-fine Optical Flow Estimation 52 Run iterative OF Run iterative OF Warp & upsample... Image 1 Image 2 Gaussian pyramid of image 1 Gaussian pyramid of image 2 52

53 Beyond the brightness constancy constraint 53 SIFT flow: dense correspondence across different scenes, C Liu, J Yuen, A Torralba, J. Sivic, W. Freeman, ECCV 2008 T. Brox, J. Malik, Large displacement optical flow: descriptor matching in variational motion estimation, PAMI 2011

54 Beyond the brightness constancy constraint 54 E. Trulls, I. Kokkinos, A. Sanfeliu, and F. Moreno, Dense Segmentation-Aware Descriptors, CVPR 2013

55 Motion-based action recognition Discovering Discriminative Action Parts from Mid-Level Video Representations, M. Raptis, I. Kokkinos and S. Soatto. CVPR,

56 Motion-based action recognition Discovering Discriminative Action Parts from Mid-Level Video Representations, M. Raptis, I. Kokkinos and S. Soatto. CVPR

57 Motion-based action recognition Discovering Discriminative Action Parts from Mid-Level Video Representations, M. Raptis, I. Kokkinos and S. Soatto. CVPR,

58 Motion-based action recognition Discovering Discriminative Action Parts from Mid-Level Video Representations, M. Raptis, I. Kokkinos and S. Soatto. CVPR,

59 Motion-based segmentation 59 Unsupervised segmentation incorporating colour, texture, and motion. T Brox, M Rousson, R Deriche, J Weickert, IVC 2010

60 Further Literature Pointers 60 Layered motion, parametric models, motion segmentation: Layered Representation for Motion Analysis. CVPR J. Wang and E. Adelson Black, M. J. and Anandan, P., The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields, CVIU 1996 D. Cremers, S. Soatto, Motion Competition: A Variational Approach to Piecewise Parametric. Motion Segmentation, IJCV 2004 Unsupervised segmentation incorporating colour, texture, and motion. T Brox, M Rousson, R Deriche, J Weickert, IVC 2010 (wait for R. Szeliski s talk) Learning: Roth, S., Black, M.J.: On the spatial statistics of optical flow. IJCV 74, (2007) Discrete optimization: (W. Freeman s talk today) Fusion Moves for Markov Random Field Optimization. V. Lempitsky, C. Rother, S. Roth, and A. Blake, PAMI B. Glocker, N. Komodakis, G. Tziritas, N. Navab, N. Paragios Dense Image Registration through MRFs and Efficient Linear Programming MIA, 2008

61 Code 61 Secrets of optical flow estimation and their principles, Sun., Roth., and Black, CVPR 10 T. Brox, J. Malik, Large displacement optical flow: descriptor matching in variational motion estimation, PAMI 2011 Action recognition & descriptor works: Registration service: