Lucas-Kanade Optical Flow Computer Vision 16-385 Carnegie Mellon University (Kris Kitani)
I x u + I y v + I t =0 I x = @I @x I y = @I u = dx v = dy I @y t = @I dt dt @t spatial derivative optical flow temporal derivative How can we use the brightness constancy equation to estimate the optical flow?
unknown I x u + I y v + I t =0 known We need at least equations to solve for 2 unknowns.
unknown I x u + I y v + I t =0 known Where do we get more equations (constraints)?
Where do we get more equations (constraints)? I x u + I y v + I t =0 Assume that the surrounding patch (say 5x5) has constant flow
Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations
Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations I x (p 1 )u + I y (p 1 )v = I t (p 1 ) I x (p 2 )u + I y (p 2 )v = I t (p 2 )... I x (p 25 )u + I y (p 25 )v = I t (p 25 )
Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations 2 6 4 I x (p 1 ) I y (p 1 ) I x (p 2 ) I y (p 2 )...... I x (p 25 ) I y (p 25 ) 3 7 5 apple u v Matrix form = 2 6 4 I t (p 1 ) I t (p 2 )... I t (p 25 ) 3 7 5
Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations 2 6 4 I x (p 1 ) I y (p 1 ) I x (p 2 ) I y (p 2 )...... I x (p 25 ) I y (p 25 ) 3 7 5 apple u v = 2 6 4 I t (p 1 ) I t (p 2 )... I t (p 25 ) 3 7 5 A x b 25 2 2 1 25 1 How many equations? How many unknowns? How do we solve this?
Least squares approximation ˆx = arg min x Ax b 2 is equivalent to solving A > Aˆx = A > b
Least squares approximation ˆx = arg min x Ax b 2 is equivalent to solving A > Aˆx = A > b To obtain the least squares solution solve: A > A ˆx A > b 2 4 P p2p P p2p I x I x I y I x P p2p P p2p I x I y I y I y 3 5 apple u v = 2 4 P p2p P p2p I x I t I y I t 3 5 where the summation is over each pixel p in patch P x =(A > A) 1 A > b
Least squares approximation ˆx = arg min x Ax b 2 is equivalent to solving A > Aˆx = A > b To obtain the least squares solution solve: A > A ˆx A > b 2 4 P p2p P p2p I x I x I y I x P p2p P p2p I x I y I y I y 3 5 apple u v = 2 4 P p2p P p2p I x I t I y I t 3 5 where the summation is over each pixel p in patch P Sometimes called Lucas-Kanade Optical Flow (special case of the LK method with a translational warp model)
When is this solvable? A > Aˆx = A > b A T A should be invertible A T A should not be too small λ 1 and λ 2 should not be too small A T A should be well conditioned λ 1 /λ 2 should not be too large (λ 1 =larger eigenvalue)
Where have you seen this before? A > A = 2 4 P p2p P p2p I x I x I y I x P p2p P p2p I x I y I y I y 3 5
Where have you seen this before? 2 4 P p2p P p2p I x I x I y I x P p2p P p2p I x I y I y I y 3 5 Harris Corner Detector!
Implications Corners are when λ1, λ2 are big; this is also when Lucas-Kanade optical flow works best Corners are regions with two different directions of gradient (at least) Corners are good places to compute flow! What happens when you have no corners?
You want to compute optical flow. What happens if the image patch contains only a line?
small visible image patch In which direction is the line moving?
small visible image patch In which direction is the line moving?
Want patches with different gradients to the avoid aperture problem
Want patches with different gradients to the avoid aperture problem
H(x,y) = y I(x,y) 1 1 1 1 2 2 2 2 x optical flow: (1,1) - - - - - 1 1 1 x 3 3 3 3-2 2 2 4 4 4 4-3 3 3 y 5 5 5 5 y - 4 4 4 I x u + I y v + I t =0 Compute gradients I x (3, 3) = 0 I y (3, 3) = 1 I t (3, 3) = I(3, 3) H(3, 3) = 1 Solution: v =1 We recover the v of the optical flow but not the u. This is the aperture problem.
Note: The Lucas-Kanade method is a general framework for image matching (not only optical flow) Applied to optical flow: Assume that the flow over a small image patch is constant
Horn-Shunck! Optical Flow (1981) Lucas-Kanade! Optical Flow! (1981) smooth flow constant flow Constraint: Flow of neighboring pixels should be smooth Contraint: Flow over a small patch should be the same
Horn-Schunck 16-385 Computer Vision Carnegie Mellon University (Kris Kitani)
Horn-Schunck! Optical Flow (1981) Lucas-Kanade! Optical Flow (1981) brightness constancy small motion smooth flow! (flow can vary from pixel to pixel) method of differences constant flow! (flow is constant for all pixels) global method (dense) local method (sparse)
Optical Flow (Problem definition) I(x, y, t) I(x, y, t 0 ) Estimate the motion (flow) between these two consecutive images How is this different from estimating a 2D transform?
Key Assumptions (unique to optical flow) Color Constancy! (Brightness constancy for intensity images) Implication: allows for pixel to pixel comparison (not image features) Small Motion! (pixels only move a little bit) Implication: linearization of the brightness constancy constraint
Approach I(x, y, t) I(x, y, t 0 ) Look for nearby pixels with the same color (small motion) (color constancy)
Brightness constancy (x + u t, y + v t) (x, y) I(x, y, t) (x, y) I(x, y, t + t) Optical flow (velocities): (u, v) Displacement: ( x, y) =(u t, v t) I(x + u t, y + v t, t + t) =I(x, y, t) For a really small time step
Optical Flow Constraint equation I(x + u t, y + v t, t + t) =I(x, y, t) I(x, y, t)+ @I @x x + @I @y y + @I @t t = I(x, y, t) @I @x dx dt + @I @y dy dt + @I @t =0 I x u + I y v + I t =0
Solution lies on a straight line u I x u + I y v + I t =0 v The solution cannot be determined uniquely with a single constraint (a single pixel)
Where can we get an additional constraint?
Horn-Schunck! Optical Flow (1981) Lucas-Kanade! Optical Flow (1981) brightness constancy small motion smooth flow! (flow can vary from pixel to pixel) global method method of differences constant flow! (flow is constant for all pixels) local method
Smoothness most objects in the world are rigid or deform elastically! moving together coherently we expect optical flow fields to be smooth
Key idea (of Horn-Schunck optical flow) Enforce brightness constancy Enforce smooth flow field to compute optical flow
Enforce brightness constancy I x u + I y v + I t =0 min u,v For every pixel, applei 2 x u ij + I y v ij + I t
Enforce brightness constancy I x u + I y v + I t =0 min u,v For every pixel, applei 2 x u ij + I y v ij + I t lazy notation for I x (i, j)
displacement of object Find the optical flow such that it satisfies: I x I y I t + = u changes these v changes these
Enforce smooth flow field u i,j+1 u i 1,j u ij u i+1,j min u (u i,j u i+1,j ) 2 u i,j 1 u-component of flow
Which flow field is more likely? X (u ij u i+1,j ) 2 X (u ij u i+1,j ) 2? ij ij
Which flow field is more likely? less likely more likely
optical flow objective function Optical flow constraint E d (i, j) = applei x u ij + I y v ij + I t 2 E s (i, j) = 1 4 Smoothness constraint apple (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) 2 i, j +1 i, j +1 i, j +1 i, j +1 (u ij u i+1,j ) (u ij u i,j+1 ) (v ij v i+1,j ) (v ij v i,j+1 ) i 1,j ij i +1,j i 1,j ij i +1,j i 1,j ij i +1,j i 1,j ij i +1,j i, j 1 i, j 1 i, j 1 i, j 1
Horn-Schunck optical flow X min u, v E d (i, j)+ E s (i, j) ij where E d (i, j) = applei x u ij + I y v ij + I t 2 E s (i, j) = 1 4 and apple (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) 2
How do we solve this minimization problem? X min u, v E d (i, j)+ E s (i, j) ij
How do we solve this minimization problem? X min u, v E d (i, j)+ E s (i, j) ij Compute partial derivative, derive update equations (gradient decent)
ij Compute the partial derivatives of this huge sum! ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) 2 + 4 applei x u ij + I y v ij + I t 2 )
ij Compute the partial derivatives of this huge sum! ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) 2 + 4 applei x u ij + I y v ij + I t 2 ) @E @u kl = it s not so bad how many terms depend on k and l?
ij Compute the partial derivatives of this huge sum! ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) 2 + 4 applei x u ij + I y v ij + I t 2 ) it s not so bad @E @u kl = 2(u kl ū kl )+2 (I x u kl + I y v kl + I t )I x how many terms depend on k and l?
ij Compute the partial derivatives of this huge sum! ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) 2 + 4 applei x u ij + I y v ij + I t 2 ) i, j +1 (u ij u i+1,j ) (u 2 ij 2u ij u i+1,j + u 2 i+1,j) (u 2 ij 2u ij u i,j+1 + u 2 i,j+1) i 1,j ij i +1,j (variable will appear four times in sum) i, j 1 @E = 2(u kl @u kl @E = 2(v kl @v kl local average ū kl )+2 (I x u kl + I y v kl + I t )I x v kl )+2 (I x u kl + I y v kl + I t )I y ū ij = 1 4 u i+1,j + u i 1,j + u i,j+1 + u i,j 1
@E = 2(u kl @u kl @E = 2(v kl @v kl ū kl )+2 (I x u kl + I y v kl + I t )I x v kl )+2 (I x u kl + I y v kl + I t )I y Where are the extrema of E?
@E = 2(u kl @u kl @E = 2(v kl @v kl ū kl )+2 (I x u kl + I y v kl + I t )I x v kl )+2 (I x u kl + I y v kl + I t )I y Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) (1 + I 2 x)u kl + I x I y v kl =ū kl I x I t I x I y u kl +(1+ I 2 y)v kl = v kl I y I t
@E = 2(u kl @u kl @E = 2(v kl @v kl ū kl )+2 (I x u kl + I y v kl + I t )I x v kl )+2 (I x u kl + I y v kl + I t )I y Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) (1 + I 2 x)u kl + I x I y v kl =ū kl I x I t I x I y u kl +(1+ I 2 y)v kl = v kl I y I t this is a linear system Ax = b how do you solve this?
ok, take a step back, why are we doing all this math?
We are solving for the optical flow (u,v) given two constraints ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) 2 + 4 ij applei x u ij + I y v ij + I t 2 ) smoothness brightness constancy We need the math to minimize this (back to the math)
Partial derivatives of Horn-Schunck objective function E: @E = 2(u kl @u kl @E = 2(v kl @v kl ū kl )+2 (I x u kl + I y v kl + I t )I x v kl )+2 (I x u kl + I y v kl + I t )I y Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) (1 + I 2 x)u kl + I x I y v kl =ū kl I x I t I x I y u kl +(1+ I 2 y)v kl = v kl I y I t Ax = b how do you solve this?
(1 + I 2 x)u kl + I x I y v kl =ū kl I x I t I x I y u kl +(1+ I 2 y)v kl = v kl I y I t Recall x = A 1 b = adja det A b
(1 + I 2 x)u kl + I x I y v kl =ū kl I x I t I x I y u kl +(1+ I 2 y)v kl = v kl I y I t Recall x = A 1 b = adja det A b Same as the linear system: {1+ (I 2 x + I 2 y)}u kl =(1+ I 2 x)ū kl I x I y v kl I x I t (determinant of A) {1+ (I 2 x + I 2 y)}v kl =(1+ I 2 y) v kl I x I y ū kl I y I t (determinant of A)
{1+ (I 2 x + I 2 y)}u kl =(1+ I 2 x)ū kl I x I y v kl I x I t (determinant of A) {1+ (I 2 x + I 2 y)}v kl =(1+ I 2 y) v kl I x I y ū kl I y I t (determinant of A) Rearrange to get update equations: û kl =ū kl new value old average ˆv kl = v kl I x ū kl + I y v kl + I t 1+ (Ix 2 + Iy) 2 I x ū kl + I y v kl + I t 1+ (Ix 2 + Iy) 2 I x I y check for missing lambda
ok, take a step back, why did we do all this math?
We are solving for the optical flow (u,v) given two constraints ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) 2 + 4 ij applei x u ij + I y v ij + I t 2 ) smoothness brightness constancy We needed the math to minimize this (now to the algorithm)
Horn-Schunck Optical Flow Algorithm 1. Precompute image gradients! 2. Precompute temporal gradients! I y I t I x 3. Initialize flow field! u = 0 v = 0 4. While not converged Compute flow field updates for each pixel: û kl =ū kl I x ū kl + I y v kl + I t 1+ (I 2 x + I 2 y) I x ˆv kl = v kl I x ū kl + I y v kl + I t 1+ (I 2 x + I 2 y) I y