Optical flow Subhransu Maji CMPSC 670: Computer Vision October 20, 2016
Visual motion Man slides adapted from S. Seitz, R. Szeliski, M. Pollefes CMPSC 670 2
Motion and perceptual organization Sometimes, motion is the onl cue CMPSC 670 3
Motion and perceptual organization Sometimes, motion is the onl cue CMPSC 670 4
Motion and perceptual organization Even impoverished motion data can evoke a strong percept G. Johansson, Visual Perception of Biological Motion and a Model For ts Analsis", Perception and Pschophsics 14, 201-211, 1973. CMPSC 670 5
Uses of motion Segmenting objects based on motion cues Estimating the 3D structure Learning and tracking dnamical models Recognizing events and activities CMPSC 670 6
Motion field The motion field is the projection of the 3D scene motion into the image CMPSC 670 7
Optical flow Definition: optical flow is the apparent motion of brightness patterns in the image deall, optical flow would be the same as the motion field Have to be careful: apparent motion can be caused b lighting changes without an actual motion Think of a uniform rotating sphere under fied lighting vs. a stationar sphere under moving illumination CMPSC 670 8
Estimating optical flow Given two subsequent frames, estimate the apparent motion field u, and v, between them,,t 1,,t Ke assumptions Brightness constanc: projection of the same point looks the same in ever frame Small motion: points do not move ver far Spatial coherence: points move like their neighbors CMPSC 670 9
The brightness constanc constraint,,t 1,,t Brightness Constanc Equation:,, t 1 = + u,, + v,, t Linearizing the right side using Talor epansion:,, t 1,, t + u, v, + Hence, u + v + t 0 CMPSC 670 10
The brightness constanc constraint - How man equations and unknowns per piel? One equation, two unknowns u + v + t = 0 What does this constraint mean? u, v + = The component of the flow perpendicular to the gradient i.e., parallel to the edge is unknown t 0 CMPSC 670 11
The brightness constanc constraint - How man equations and unknowns per piel? One equation, two unknowns u + v + t = 0 What does this constraint mean? u, v + = The component of the flow perpendicular to the gradient i.e., parallel to the edge is unknown t 0 f u, v satisfies the equation, so does u+u, v+v if u', v' = 0 gradient u,v u,v u+u,v+v CMPSC 670 edge 12
The aperture problem Perceived motion
The aperture problem Actual motion
The aperture problem What direction is the motion?
The barber pole illusion http://en.wikipedia.org/wiki/barberpole_illusion
CMPSC 670 How to get more equations for a piel? Spatial coherence constraint: pretend the piel s neighbors have the same u,v E.g., if we use a 55 window, that gives us 25 equations per piel Solving the aperture problem 17 B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. n Proceedings of the nternational Joint Conference on Artificial ntelligence, pp. 674 679, 1981. 0 ], [ = + i i t v u " # % & = " # % & " # % & 2 1 2 2 1 1 n t t t n n v u
CMPSC 670 Least squares problem: Solving the aperture problem 18 B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. n Proceedings of the nternational Joint Conference on Artificial ntelligence, pp. 674 679, 1981. When is this sstem solvable? What if the window contains just a single straight edge? " # % & = " # % & " # % & 2 1 2 2 1 1 n t t t n n v u
Conditions for solvabilit Bad case: single straight edge CMPSC 670 19
Conditions for solvabilit Good case: corner CMPSC 670 20
CMPSC 670 Linear least squares problem Lucas-Kanade flow 21 B. Lucas and T. Kanade. An iterative image registration technique with an application to stereo vision. n Proceedings of the nternational Joint Conference on Artificial ntelligence, pp. 674 679, 1981. The summations are over all piels in the window Solution given b " # % & = " # % & " # % & 2 1 2 2 1 1 n t t t n n v u 1 1 2 2 = n n b A d b A Ad A T T = " # % & = " # % & " # % & t t v u
Lucas-Kanade flow & % #& u# "% v" & = % t t # " Recall the Harris corner detector: M = A T A is the second moment matri We can figure out whether the sstem is solvable b looking at the eigenvalues of the second moment matri The eigenvectors and eigenvalues of M relate to edge direction and magnitude The eigenvector associated with the larger eigenvalue points in the direction of fastest intensit change, and the other eigenvector is orthogonal to it CMPSC 670 22
Visualization of second moment matrices CMPSC 670 23
Visualization of second moment matrices CMPSC 670 24
nterpreting the eigenvalues Classification of image points using eigenvalues of the second moment matri: λ 2 Edge λ 2 >> λ 1 Corner λ 1 and λ 2 are large, λ 1 ~ λ 2 λ 1 and λ 2 are small Flat Edge region λ 1 >> λ 2 CMPSC 670 λ 1 25
Eample CMPSC 670 Subhransu Maji UMass, * From Fall Khurram 16 Hassan-Shafique CAP5415 Computer Vision 2003 26
Uniform region gradients have small magnitude small λ 1, small λ 2 sstem is ill-conditioned CMPSC 670 27
Edge CMPSC 670 gradients have one dominant direction large λ 1, small λ 2 sstem is ill-conditioned 28
High-teture or corner region gradients have different directions, large magnitudes large λ 1, large λ 2 sstem is well-conditioned CMPSC 670 29
Optical Flow Results CMPSC 670 * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003 30
Errors in Lucas-Kanade The motion is large larger than a piel terative refinement Coarse-to-fine estimation Ehaustive neighborhood search feature matching A point does not move like its neighbors Motion segmentation Brightness constanc does not hold Ehaustive neighborhood search with normalized correlation CMPSC 670 31
Multi-resolution registration CMPSC 670 * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003 32
Optical flow results CMPSC 670 * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003 33
Optical flow results CMPSC 670 * From Khurram Hassan-Shafique CAP5415 Computer Vision 2003 34
State-of-the-art optical flow Start with something similar to Lucas-Kanade + gradient constanc + energ minimization with smoothing term + region matching + kepoint matching long-range Region-based +Piel-based +Kepoint-based Large displacement optical flow, Bro et al., CVPR 2009 CMPSC 670 Source: J. Has35
State-of-the-art optical flow Epic Flow: Feature matching + edge preserving flow interpolation EpicFlow: Edge-Preserving nterpolation of Correspondences for Optical Flow, Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui and Cordelia Schmid, CVPR 2015. CMPSC 670 36
Feature tracking So far, we have onl considered optical flow estimation in a pair of images f we have more than two images, we can compute the optical flow from each frame to the net Given a point in the first image, we can in principle reconstruct its path b simpl following the arrows t t+1 t+2 CMPSC 670 37
Tracking challenges Ambiguit of optical flow Need to find good features to track Large motions, changes in appearance, occlusions, disocclusions Need mechanism for deleting, adding new features Drift errors ma accumulate over time Need to know when to terminate a track CMPSC 670 38
Shi-Tomasi feature tracker Find good features using eigenvalues of second-moment matri Ke idea: good features to track are the ones whose motion can be estimated reliabl From frame to frame, track with Lucas-Kanade This amounts to assuming a translation model for frame-to-frame feature movement Check consistenc of tracks b affine registration to the first observed instance of the feature Affine model is more accurate for larger displacements Comparing to the first frame helps to minimize drift J. Shi and C. Tomasi. Good Features to Track. CVPR 1994. CMPSC 670 39
Tracking eample CMPSC 670 J. Shi and C. Tomasi. Good Features to Track. CVPR 1994. 40
Tracking b matching CMPSC 670 41
Tracking eample CMPSC 670 https://www.outube.com/watch?v=rg5uv_h50b0 42