Video and Motion Analysis Computer Vision Carnegie Mellon University (Kris Kitani)

Transcription

1 Video and Motion Analysis Computer Vision Carnegie Mellon University (Kris Kitani)

2 Optical flow used for feature tracking on a drone

3 Interpolated optical flow used for super slow-mo

4 optical flow used for motion estimation in visual odometry

5 Roadmap (Where we have been and where we are going)

6 Image filtering Frequency domain image pyramids Image gradients Boundaries Hough Transform Image Manipulation (January)

7 Corner detection Multi-scale detection Haar-like HOG SURF SIFT Image Features (February)

8 Bag-of-words K-means Nearest Neighbor Naive Bayes SVM Object Recognition (February)

9 2D Transforms DLT RANSAC 2D Alignment (March)

10 x = PX P X camera matrix pose estimation triangulation F fundamental matrix epipolar geometry Reconstruction 2 view geometry (March)

11 Block matching Energy minimization Stereo (March)

12 What you can do now Detect lines (circles, shapes) in an image Recognize objects using a bag-of-words model Automatic image warping and basic AR Reconstruct 3D scene structure from two images

13 What you will learn next Object tracking in video

14 2 6 4 I x (p 1 ) I y (p 1 ) I x (p 2 ) I y (p 2 ).. I x (p 25 ) I y (p 25 ) apple u v = I t (p 1 ) I t (p 2 ). I t (p 25 ) min u, v X ij E d (i, j)+ E s (i, j) Constant Flow Horn Schunck Optical Flow (April)

15 Lucas Kanade (Forward additive) Baker Matthews (Inverse Compositional) Image Alignment (April)

16 KLT Mean shift Kalman Filtering SLAM Tracking in Video (April-May)

17 Brightness Constancy Computer Vision Carnegie Mellon University (Kris Kitani)

18 Optical Flow Problem Definition Given two consecutive image frames, estimate the motion of each pixel Assumptions Brightness constancy Small motion

19 Assumption 1 Brightness constancy Scene point moving through image sequence (x(2),y(2)) (x(k),y(k)) (x(1),y(1))

20 Assumption 1 Brightness constancy Scene point moving through image sequence (x(2),y(2)) (x(k),y(k)) (x(1),y(1)) I(x, y, 1) I(x, y, 2) I(x, y, k)

21 Assumption 1 Brightness constancy Scene point moving through image sequence (x(2),y(2)) (x(k),y(k)) (x(1),y(1)) I(x, y, 1) I(x, y, 2) I(x, y, k) Assumption:Brightness of the point will remain the same

22 Assumption 1 Brightness constancy Scene point moving through image sequence (x(2),y(2)) (x(k),y(k)) (x(1),y(1)) I(x, y, 1) I(x, y, 2) I(x, y, k) Assumption:Brightness of the point will remain the same I(x(t),y(t),t)=C constant

23 Assumption 2 Small motion I(x, y, t) I(x, y, t + t)

24 Assumption 2 Small motion (x + u t, y + v t) (x, y) I(x, y, t) (x, y) I(x, y, t + t)

25 Assumption 2 Small motion (x + u t, y + v t) (x, y) I(x, y, t) (x, y) I(x, y, t + t) Optical flow (velocities): (u, v) Displacement: ( x, y) =(u t, v t)

26 Assumption 2 Small motion (x + u t, y + v t) (x, y) I(x, y, t) (x, y) I(x, y, t + t) Optical flow (velocities): (u, v) Displacement: ( x, y) =(u t, v t) For a really small space-time step I(x + u t, y + v t, t + t) =I(x, y, t) the brightness between two consecutive image frames is the same

27 These assumptions yield the Brightness Constancy Equation di dx dy =0 total derivative partial derivative Where does this come from?

28 I(x + u t, y + v t, t + t) =I(x, y, t) For small space-time step, brightness of a point is the same If the time step is really small, we can linearize the intensity function

29 I(x + u t, y + v t, t + t) =I(x, y, t) Multivariable Taylor Series Expansion (First order approximation, two variables) f(x, y) f(a, b)+f x (a, b)(x a) f y (a, b)(y b)

30 I(x + u t, y + v t, t + t) =I(x, y, t) Multivariable Taylor Series Expansion (First order approximation, two variables) f(x, y) f(a, b)+f x (a, b)(x a) f y (a, b)(y b) I(x, t = I(x, y, t) assuming small motion

31 I(x + u t, y + v t, t + t) =I(x, y, t) Multivariable Taylor Series Expansion (First order approximation, two variables) f(x, y) f(a, b)+f x (a, b)(x a) f y (a, b)(y b) I(x, t = I(x, y, t) assuming small motion cancel terms

32 I(x + u t, y + v t, t + t) =I(x, y, t) Multivariable Taylor Series Expansion (First order approximation, two variables) f(x, y) f(a, b)+f x (a, b)(x a) f y (a, b)(y b) I(x, t = I(x, y, t) assuming t =0 cancel terms

33 I(x + u t, y + v t, t + t) =I(x, y, t) Multivariable Taylor Series Expansion (First order approximation, two variables) f(x, y) f(a, b)+f x (a, b)(x a) f y (a, b)(y b) I(x, t = I(x, y, t) assuming t =0 divide by take limit t t! 0

34 I(x + u t, y + v t, t + t) =I(x, y, t) Multivariable Taylor Series Expansion (First order approximation, two variables) f(x, y) f(a, b)+f x (a, b)(x a) f y (a, b)(y b) I(x, t = I(x, y, t) assuming t =0 divide by take limit t dx dy =0

35 I(x + u t, y + v t, t + t) =I(x, y, t) Multivariable Taylor Series Expansion (First order approximation, two variables) f(x, y) f(a, b)+f x (a, b)(x a) f y (a, b)(y b) I(x, t = I(x, y, t) assuming t =0 divide by take limit t dx dy =0 Brightness Constancy Equation

36 dx dy =0 Brightness Constancy Equation I x u + I y v + I t =0 shorthand notation (displacement) ri > v + I t =0 vector form

37 (putting the math aside for a second ) What do the term of the brightness constancy equation represent?! I x u + I y v + I t =0

38 (putting the math aside for a second ) What do the term of the brightness constancy equation represent?! I x u + I y v + I t =0 Image gradients

39 (putting the math aside for a second ) What do the term of the brightness constancy equation represent?! flow velocities I x u + I y v + I t =0 Image gradients

40 (putting the math aside for a second ) What do the term of the brightness constancy equation represent?! flow velocities I x u + I y v + I t =0 Image gradients temporal gradient How do you compute these terms?

41 I x u + I y v + I t =0 How do you compute I I spatial derivative

42 I x u + I y v + I t =0 How do you compute I I spatial derivative Forward difference Sobel filter Scharr filter

43 I x u + I y v + I t =0 How do you compute I I I spatial derivative temporal derivative Forward difference Sobel filter Scharr filter

44 I x u + I y v + I t =0 How do you compute I I I spatial derivative temporal derivative Forward difference Sobel filter Scharr filter frame differencing

45 Frame differencing t t +1 I = (example of a forward difference)

46 Example: t t +1 x x x 3 patch y y I x I x I x y y y

47 I x u + I y v + I t =0 How do you compute I I spatial derivative u = dx v = dy dt dt optical flow I temporal derivative Forward difference Sobel filter Scharr filter How do you compute this? frame differencing

48 I x u + I y v + I t =0 How do you compute I I spatial derivative u = dx v = dy dt dt optical flow I temporal derivative Forward difference Sobel filter Scharr filter We need to solve for this!! (this is the unknown in the optical flow problem) frame differencing

49 I x u + I y v + I t =0 How do you compute I I spatial derivative u = dx v = dy dt dt optical flow I temporal derivative Forward difference Sobel filter Scharr filter (u, v) Solution lies on a line Cannot be found uniquely with a single constraint frame differencing

50 I x u + I y v + I t =0 I I y u = dx v = dy t dt spatial derivative optical flow temporal derivative How can we use the brightness constancy equation to estimate the optical flow?

51 unknown I x u + I y v + I t =0 known We need at least equations to solve for 2 unknowns.

52 unknown I x u + I y v + I t =0 known Where do we get more equations (constraints)?