Lucas-Kanade Optical Flow. Computer Vision Carnegie Mellon University (Kris Kitani)

Transcription

1 Lucas-Kanade Optical Flow Computer Vision Carnegie Mellon University (Kris Kitani)

2 I x u + I y v + I t =0 I I y u = dx v = dy t dt spatial derivative optical flow temporal derivative How can we use the brightness constancy equation to estimate the optical flow?

3 unknown I x u + I y v + I t =0 known We need at least equations to solve for 2 unknowns.

4 unknown I x u + I y v + I t =0 known Where do we get more equations (constraints)?

5 Where do we get more equations (constraints)? I x u + I y v + I t =0 Assume that the surrounding patch (say 5x5) has constant flow

6 Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations

7 Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations I x (p 1 )u + I y (p 1 )v = I t (p 1 ) I x (p 2 )u + I y (p 2 )v = I t (p 2 )... I x (p 25 )u + I y (p 25 )v = I t (p 25 )

8 Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations I x (p 1 ) I y (p 1 ) I x (p 2 ) I y (p 2 ) I x (p 25 ) I y (p 25 ) apple u v Matrix form = I t (p 1 ) I t (p 2 )... I t (p 25 ) 3 7 5

9 Assumptions: Flow is locally smooth Neighboring pixels have same displacement Using a 5 x 5 image patch, gives us 25 equations I x (p 1 ) I y (p 1 ) I x (p 2 ) I y (p 2 ) I x (p 25 ) I y (p 25 ) apple u v = I t (p 1 ) I t (p 2 )... I t (p 25 ) A x b How many equations? How many unknowns? How do we solve this?

10 Least squares approximation ˆx = arg min x Ax b 2 is equivalent to solving A > Aˆx = A > b

11 Least squares approximation ˆx = arg min x Ax b 2 is equivalent to solving A > Aˆx = A > b To obtain the least squares solution solve: A > A ˆx A > b 2 4 P p2p P p2p I x I x I y I x P p2p P p2p I x I y I y I y 3 5 apple u v = 2 4 P p2p P p2p I x I t I y I t 3 5 where the summation is over each pixel p in patch P x =(A > A) 1 A > b

12 Least squares approximation ˆx = arg min x Ax b 2 is equivalent to solving A > Aˆx = A > b To obtain the least squares solution solve: A > A ˆx A > b 2 4 P p2p P p2p I x I x I y I x P p2p P p2p I x I y I y I y 3 5 apple u v = 2 4 P p2p P p2p I x I t I y I t 3 5 where the summation is over each pixel p in patch P Sometimes called Lucas-Kanade Optical Flow (special case of the LK method with a translational warp model)

13 When is this solvable? A > Aˆx = A > b A T A should be invertible A T A should not be too small λ 1 and λ 2 should not be too small A T A should be well conditioned λ 1 /λ 2 should not be too large (λ 1 =larger eigenvalue)

14 Where have you seen this before? A > A = 2 4 P p2p P p2p I x I x I y I x P p2p P p2p I x I y I y I y 3 5

15 Where have you seen this before? 2 4 P p2p P p2p I x I x I y I x P p2p P p2p I x I y I y I y 3 5 Harris Corner Detector!

16 Implications Corners are when λ1, λ2 are big; this is also when Lucas-Kanade optical flow works best Corners are regions with two different directions of gradient (at least) Corners are good places to compute flow! What happens when you have no corners?

17 You want to compute optical flow. What happens if the image patch contains only a line?

18 small visible image patch In which direction is the line moving?

19 small visible image patch In which direction is the line moving?

20

21

22

23

24 Want patches with different gradients to the avoid aperture problem

25 Want patches with different gradients to the avoid aperture problem

26 H(x,y) = y I(x,y) x optical flow: (1,1) x y y I x u + I y v + I t =0 Compute gradients I x (3, 3) = 0 I y (3, 3) = 1 I t (3, 3) = I(3, 3) H(3, 3) = 1 Solution: v =1 We recover the v of the optical flow but not the u. This is the aperture problem.

27 Note: The Lucas-Kanade method is a general framework for image matching (not only optical flow) Applied to optical flow: Assume that the flow over a small image patch is constant

28 Horn-Shunck! Optical Flow (1981) Lucas-Kanade! Optical Flow! (1981) smooth flow constant flow Constraint: Flow of neighboring pixels should be smooth Contraint: Flow over a small patch should be the same

29 Horn-Schunck Computer Vision Carnegie Mellon University (Kris Kitani)

30 Horn-Schunck! Optical Flow (1981) Lucas-Kanade! Optical Flow (1981) brightness constancy small motion smooth flow! (flow can vary from pixel to pixel) method of differences constant flow! (flow is constant for all pixels) global method (dense) local method (sparse)

31 Optical Flow (Problem definition) I(x, y, t) I(x, y, t 0 ) Estimate the motion (flow) between these two consecutive images How is this different from estimating a 2D transform?

32 Key Assumptions (unique to optical flow) Color Constancy! (Brightness constancy for intensity images) Implication: allows for pixel to pixel comparison (not image features) Small Motion! (pixels only move a little bit) Implication: linearization of the brightness constancy constraint

33 Approach I(x, y, t) I(x, y, t 0 ) Look for nearby pixels with the same color (small motion) (color constancy)

34 Brightness constancy (x + u t, y + v t) (x, y) I(x, y, t) (x, y) I(x, y, t + t) Optical flow (velocities): (u, v) Displacement: ( x, y) =(u t, v t) I(x + u t, y + v t, t + t) =I(x, y, t) For a really small time step

35 Optical Flow Constraint equation I(x + u t, y + v t, t + t) =I(x, y, t) I(x, t = I(x, dx dy =0 I x u + I y v + I t =0

36 Solution lies on a straight line u I x u + I y v + I t =0 v The solution cannot be determined uniquely with a single constraint (a single pixel)

37 Where can we get an additional constraint?

38 Horn-Schunck! Optical Flow (1981) Lucas-Kanade! Optical Flow (1981) brightness constancy small motion smooth flow! (flow can vary from pixel to pixel) global method method of differences constant flow! (flow is constant for all pixels) local method

39 Smoothness most objects in the world are rigid or deform elastically! moving together coherently we expect optical flow fields to be smooth

40 Key idea (of Horn-Schunck optical flow) Enforce brightness constancy Enforce smooth flow field to compute optical flow

41 Enforce brightness constancy I x u + I y v + I t =0 min u,v For every pixel, applei 2 x u ij + I y v ij + I t

42 Enforce brightness constancy I x u + I y v + I t =0 min u,v For every pixel, applei 2 x u ij + I y v ij + I t lazy notation for I x (i, j)

43 displacement of object Find the optical flow such that it satisfies: I x I y I t + = u changes these v changes these

44 Enforce smooth flow field u i,j+1 u i 1,j u ij u i+1,j min u (u i,j u i+1,j ) 2 u i,j 1 u-component of flow

45 Which flow field is more likely? X (u ij u i+1,j ) 2 X (u ij u i+1,j ) 2? ij ij

46 Which flow field is more likely? less likely more likely

47 optical flow objective function Optical flow constraint E d (i, j) = applei x u ij + I y v ij + I t 2 E s (i, j) = 1 4 Smoothness constraint apple (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) 2 i, j +1 i, j +1 i, j +1 i, j +1 (u ij u i+1,j ) (u ij u i,j+1 ) (v ij v i+1,j ) (v ij v i,j+1 ) i 1,j ij i +1,j i 1,j ij i +1,j i 1,j ij i +1,j i 1,j ij i +1,j i, j 1 i, j 1 i, j 1 i, j 1

48 Horn-Schunck optical flow X min u, v E d (i, j)+ E s (i, j) ij where E d (i, j) = applei x u ij + I y v ij + I t 2 E s (i, j) = 1 4 and apple (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) 2

49 How do we solve this minimization problem? X min u, v E d (i, j)+ E s (i, j) ij

50 How do we solve this minimization problem? X min u, v E d (i, j)+ E s (i, j) ij Compute partial derivative, derive update equations (gradient decent)

51 ij Compute the partial derivatives of this huge sum! ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) applei x u ij + I y v ij + I t 2 )

52 ij Compute the partial derivatives of this huge sum! ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) applei x u ij + I y v ij + I t kl = it s not so bad how many terms depend on k and l?

53 ij Compute the partial derivatives of this huge sum! ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) applei x u ij + I y v ij + I t 2 ) it s not kl = 2(u kl ū kl )+2 (I x u kl + I y v kl + I t )I x how many terms depend on k and l?

54 ij Compute the partial derivatives of this huge sum! ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) applei x u ij + I y v ij + I t 2 ) i, j +1 (u ij u i+1,j ) (u 2 ij 2u ij u i+1,j + u 2 i+1,j) (u 2 ij 2u ij u i,j+1 + u 2 i,j+1) i 1,j ij i +1,j (variable will appear four times in sum) i, j = 2(u = 2(v kl local average ū kl )+2 (I x u kl + I y v kl + I t )I x v kl )+2 (I x u kl + I y v kl + I t )I y ū ij = 1 4 u i+1,j + u i 1,j + u i,j+1 + u i,j 1

55 @E = 2(u = 2(v kl ū kl )+2 (I x u kl + I y v kl + I t )I x v kl )+2 (I x u kl + I y v kl + I t )I y Where are the extrema of E?

56 @E = 2(u = 2(v kl ū kl )+2 (I x u kl + I y v kl + I t )I x v kl )+2 (I x u kl + I y v kl + I t )I y Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) (1 + I 2 x)u kl + I x I y v kl =ū kl I x I t I x I y u kl +(1+ I 2 y)v kl = v kl I y I t

57 @E = 2(u = 2(v kl ū kl )+2 (I x u kl + I y v kl + I t )I x v kl )+2 (I x u kl + I y v kl + I t )I y Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) (1 + I 2 x)u kl + I x I y v kl =ū kl I x I t I x I y u kl +(1+ I 2 y)v kl = v kl I y I t this is a linear system Ax = b how do you solve this?

58 ok, take a step back, why are we doing all this math?

59 We are solving for the optical flow (u,v) given two constraints ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) ij applei x u ij + I y v ij + I t 2 ) smoothness brightness constancy We need the math to minimize this (back to the math)

60 Partial derivatives of Horn-Schunck objective function = 2(u = 2(v kl ū kl )+2 (I x u kl + I y v kl + I t )I x v kl )+2 (I x u kl + I y v kl + I t )I y Where are the extrema of E? (set derivatives to zero and solve for unknowns u and v) (1 + I 2 x)u kl + I x I y v kl =ū kl I x I t I x I y u kl +(1+ I 2 y)v kl = v kl I y I t Ax = b how do you solve this?

61 (1 + I 2 x)u kl + I x I y v kl =ū kl I x I t I x I y u kl +(1+ I 2 y)v kl = v kl I y I t Recall x = A 1 b = adja det A b

62 (1 + I 2 x)u kl + I x I y v kl =ū kl I x I t I x I y u kl +(1+ I 2 y)v kl = v kl I y I t Recall x = A 1 b = adja det A b Same as the linear system: {1+ (I 2 x + I 2 y)}u kl =(1+ I 2 x)ū kl I x I y v kl I x I t (determinant of A) {1+ (I 2 x + I 2 y)}v kl =(1+ I 2 y) v kl I x I y ū kl I y I t (determinant of A)

63 {1+ (I 2 x + I 2 y)}u kl =(1+ I 2 x)ū kl I x I y v kl I x I t (determinant of A) {1+ (I 2 x + I 2 y)}v kl =(1+ I 2 y) v kl I x I y ū kl I y I t (determinant of A) Rearrange to get update equations: û kl =ū kl new value old average ˆv kl = v kl I x ū kl + I y v kl + I t 1+ (Ix 2 + Iy) 2 I x ū kl + I y v kl + I t 1+ (Ix 2 + Iy) 2 I x I y check for missing lambda

64 ok, take a step back, why did we do all this math?

65 We are solving for the optical flow (u,v) given two constraints ( X apple 1 (u ij u i+1,j ) 2 +(u ij u i,j+1 ) 2 +(v ij v i+1,j ) 2 +(v ij v i,j+1 ) ij applei x u ij + I y v ij + I t 2 ) smoothness brightness constancy We needed the math to minimize this (now to the algorithm)

66 Horn-Schunck Optical Flow Algorithm 1. Precompute image gradients! 2. Precompute temporal gradients! I y I t I x 3. Initialize flow field! u = 0 v = 0 4. While not converged Compute flow field updates for each pixel: û kl =ū kl I x ū kl + I y v kl + I t 1+ (I 2 x + I 2 y) I x ˆv kl = v kl I x ū kl + I y v kl + I t 1+ (I 2 x + I 2 y) I y