Optical flow. Subhransu Maji. CMPSCI 670: Computer Vision. October 20, 2016

Transcription

1 Optical flow Subhransu Maji CMPSC 670: Computer Vision October 20, 2016

2 Visual motion Man slides adapted from S. Seitz, R. Szeliski, M. Pollefes CMPSC 670 2

3 Motion and perceptual organization Sometimes, motion is the onl cue CMPSC 670 3

4 Motion and perceptual organization Sometimes, motion is the onl cue CMPSC 670 4

5 Motion and perceptual organization Even impoverished motion data can evoke a strong percept G. Johansson, Visual Perception of Biological Motion and a Model For ts Analsis", Perception and Pschophsics 14, , CMPSC 670 5

6 Uses of motion Segmenting objects based on motion cues Estimating the 3D structure Learning and tracking dnamical models Recognizing events and activities CMPSC 670 6

7 Motion field The motion field is the projection of the 3D scene motion into the image CMPSC 670 7

8 Optical flow Definition: optical flow is the apparent motion of brightness patterns in the image deall, optical flow would be the same as the motion field Have to be careful: apparent motion can be caused b lighting changes without an actual motion Think of a uniform rotating sphere under fied lighting vs. a stationar sphere under moving illumination CMPSC 670 8

9 Estimating optical flow Given two subsequent frames, estimate the apparent motion field u, and v, between them,,t 1,,t Ke assumptions Brightness constanc: projection of the same point looks the same in ever frame Small motion: points do not move ver far Spatial coherence: points move like their neighbors CMPSC 670 9

10 The brightness constanc constraint,,t 1,,t Brightness Constanc Equation:,, t 1 = + u,, + v,, t Linearizing the right side using Talor epansion:,, t 1,, t + u, v, + Hence, u + v + t 0 CMPSC

11 The brightness constanc constraint - How man equations and unknowns per piel? One equation, two unknowns u + v + t = 0 What does this constraint mean? u, v + = The component of the flow perpendicular to the gradient i.e., parallel to the edge is unknown t 0 CMPSC

12 The brightness constanc constraint - How man equations and unknowns per piel? One equation, two unknowns u + v + t = 0 What does this constraint mean? u, v + = The component of the flow perpendicular to the gradient i.e., parallel to the edge is unknown t 0 f u, v satisfies the equation, so does u+u, v+v if u', v' = 0 gradient u,v u,v u+u,v+v CMPSC 670 edge 12

13 The aperture problem Perceived motion

14 The aperture problem Actual motion

15 The aperture problem What direction is the motion?

16 The barber pole illusion

17 CMPSC 670 How to get more equations for a piel? Spatial coherence constraint: pretend the piel s neighbors have the same u,v E.g., if we use a 55 window, that gives us 25 equations per piel Solving the aperture problem 17 B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. n Proceedings of the nternational Joint Conference on Artificial ntelligence, pp , ], [ = + i i t v u " # % & = " # % & " # % & n t t t n n v u

18 CMPSC 670 Least squares problem: Solving the aperture problem 18 B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. n Proceedings of the nternational Joint Conference on Artificial ntelligence, pp , When is this sstem solvable? What if the window contains just a single straight edge? " # % & = " # % & " # % & n t t t n n v u

19 Conditions for solvabilit Bad case: single straight edge CMPSC

20 Conditions for solvabilit Good case: corner CMPSC

21 CMPSC 670 Linear least squares problem Lucas-Kanade flow 21 B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. n Proceedings of the nternational Joint Conference on Artificial ntelligence, pp , The summations are over all piels in the window Solution given b " # % & = " # % & " # % & n t t t n n v u = n n b A d b A Ad A T T = " # % & = " # % & " # % & t t v u

22 Lucas-Kanade flow & % #& u# "% v" & = % t t # " Recall the Harris corner detector: M = A T A is the second moment matri We can figure out whether the sstem is solvable b looking at the eigenvalues of the second moment matri The eigenvectors and eigenvalues of M relate to edge direction and magnitude The eigenvector associated with the larger eigenvalue points in the direction of fastest intensit change, and the other eigenvector is orthogonal to it CMPSC

23 Visualization of second moment matrices CMPSC

24 Visualization of second moment matrices CMPSC

25 nterpreting the eigenvalues Classification of image points using eigenvalues of the second moment matri: λ 2 Edge λ 2 >> λ 1 Corner λ 1 and λ 2 are large, λ 1 ~ λ 2 λ 1 and λ 2 are small Flat Edge region λ 1 >> λ 2 CMPSC 670 λ 1 25

26 Eample CMPSC 670 Subhransu Maji UMass, * From Fall Khurram 16 Hassan-Shafique CAP5415 Computer Vision

27 Uniform region gradients have small magnitude small λ 1, small λ 2 sstem is ill-conditioned CMPSC

28 Edge CMPSC 670 gradients have one dominant direction large λ 1, small λ 2 sstem is ill-conditioned 28

29 High-teture or corner region gradients have different directions, large magnitudes large λ 1, large λ 2 sstem is well-conditioned CMPSC

30 Optical Flow Results CMPSC 670 * From Khurram Hassan-Shafique CAP5415 Computer Vision

31 Errors in Lucas-Kanade The motion is large larger than a piel terative refinement Coarse-to-fine estimation Ehaustive neighborhood search feature matching A point does not move like its neighbors Motion segmentation Brightness constanc does not hold Ehaustive neighborhood search with normalized correlation CMPSC

32 Multi-resolution registration CMPSC 670 * From Khurram Hassan-Shafique CAP5415 Computer Vision

33 Optical flow results CMPSC 670 * From Khurram Hassan-Shafique CAP5415 Computer Vision

34 Optical flow results CMPSC 670 * From Khurram Hassan-Shafique CAP5415 Computer Vision

35 State-of-the-art optical flow Start with something similar to Lucas-Kanade + gradient constanc + energ minimization with smoothing term + region matching + kepoint matching long-range Region-based +Piel-based +Kepoint-based Large displacement optical flow, Bro et al., CVPR 2009 CMPSC 670 Source: J. Has35

36 State-of-the-art optical flow Epic Flow: Feature matching + edge preserving flow interpolation EpicFlow: Edge-Preserving nterpolation of Correspondences for Optical Flow, Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui and Cordelia Schmid, CVPR CMPSC

37 Feature tracking So far, we have onl considered optical flow estimation in a pair of images f we have more than two images, we can compute the optical flow from each frame to the net Given a point in the first image, we can in principle reconstruct its path b simpl following the arrows t t+1 t+2 CMPSC

38 Tracking challenges Ambiguit of optical flow Need to find good features to track Large motions, changes in appearance, occlusions, disocclusions Need mechanism for deleting, adding new features Drift errors ma accumulate over time Need to know when to terminate a track CMPSC

39 Shi-Tomasi feature tracker Find good features using eigenvalues of second-moment matri Ke idea: good features to track are the ones whose motion can be estimated reliabl From frame to frame, track with Lucas-Kanade This amounts to assuming a translation model for frame-to-frame feature movement Check consistenc of tracks b affine registration to the first observed instance of the feature Affine model is more accurate for larger displacements Comparing to the first frame helps to minimize drift J. Shi and C. Tomasi. Good Features to Track. CVPR CMPSC

40 Tracking eample CMPSC 670 J. Shi and C. Tomasi. Good Features to Track. CVPR

41 Tracking b matching CMPSC

42 Tracking eample CMPSC