CSCI 250: Intro to Robotics. Spring Term 2017 Prof. Levy. Computer Vision: A Brief Survey

Transcription

1 CSCI 25: Intro to Robotics Spring Term 27 Prof. Levy Computer Vision: A Brief Survey

2 What Is Computer Vision? Higher-order tasks Face recognition Object recognition (Deep Learning)

3 What Is Computer Vision? Lower-order tasks Edge detection Optical flow

4 Image Convolution

5

6

7 Convolution: Convolution is a mathematical way of combining two signals to form a third signal. It is the single most important technique in Digital Signal Processing.* *

8 Convolution: The Dot Product on Steroids Let's look dot product in terms of vectors (arrays): n xi w i i= x x2 x3... xn * * * w w2 w3... * wn +

9 Convolution: The Dot Product on Steroids To get convolution, we slide w across x: x x2 x3... xn * * * w w2 w3... * wn x x2 x3... * * * w2 w3 w y Do you see a problem? + y2 xn * wn+ x x2 x3... * * * w3 w4 w y3 xn * wn+2

10 Convolution: The Dot Product on Steroids Usual definition of convolution assumes infinite vectors: yn =... x 2 x x x 3 k= x2 x n k w k x x 4 x3 x2... * * *... w w w... - * * *... w w w... - * * *... w w w y + y2 + y3 Of course, this is unrealistic, so...

11 Convolution: The Dot Product on Steroids We treat w as a finite convolution kernel, and switch to nonnegative indices: e.g.,: 2 y n = x n k w k... x 2 x x... k=... x x x x 4 x3 x2... * * * w w w2 * * * w w w2 * * * w w w 2 + y2 + y3 + y4 Do you notice another (small) problem?

12 w x y

13 w x y

14 w x y

15 w x y

16 w x y

17 w x y c c

18 w x y c c c

19 w x y c c c c

20 w x y c c c c c

21 Convolution: What's It Goodfer? Many useful operations can be expressed as a convolution E.g., smoothing, a.ka. moving average, a.k.a. low-pass filtering.

22 w x y

23 w x y

24 w x y

25 w x y

26 w x y

27 w x y

28 w x y

29 w x y

30 w x y

31

32 Convolution: What's It Goodfer? Many useful operations can be expressed as a convolution E.g., edge detection, a.k.a. high-pass filtering

33 w x y

34 w x y -2

35 w x y

36 w x y

37 w x y

38 w x y

39 w x y

40 w x y

41 w x y

42

43 Two-Dimensional Convolution x w

44 Two-Dimensional Convolution x y

45 Two-Dimensional Convolution x y

46 Two-Dimensional Convolution x y

47 Two-Dimensional Convolution x y

48 Common Kernels for Image Manipulation

49 Common Kernels for Image Manipulation

50 Homebrew Convolution with NumPy

51 Optical Flow

52 Optical Flow A basic question: how do we (and other sighted creatures) move around without bumping into things? Ecological insight (J.J. Gibson): It can t involve a complicated computational algorithm in the brain* As much as possible, an animal must be have its senses / nervous system tightly coupled with its environment. * But, because of the lack of analog optical-flow circuits, nearly everyone still does it with an algorithm / package (OpenCV)!

53 Optical Flow: the Computational Problem Given the current X,Y locations of a set of points of interest in a image, and their locations in the next image, simple subtraction gives us the dx, dy arrows indicating flow. But images do not come with pre-labeled points; they contain a uniform field of pixels.

54 Optical Flow: the Computational Problem So the problem becomes: given two images, how do we identify which points in the first correspond to which points in the second? You will sometimes hear the terms sparse optical flow and dense optical flow to distinguish the solutions, but it is more helpful to look at specific algorithms.

55 Algorithm #: Lucas-Kanade (98) On each iteration, identifies a set of good points (typically, corners of objects) to use for tracking movement. Each point becomes the center of a small patch ( window ) on which optical flow will be computed. Optical flow arrow is computed so as to minimize the difference (linear least squares) between the resulting patch and the actual patch.

56 patcht- patcht

57 patcht- patcht

58 patcht- patcht

59 To maximize accuracy, the flow is computed at several levels of resolution, using a pyramid approach:

60

61 Algorithm #2: Farnebäck (23) Models the neighborhood of each pixel as a polynomial function E.g., f(x) = ax2 + bx + c : Then use linear algebra to solve for the displacement between time t and t2 :

62

63

64

65

66

67 How to get optical flow on your robot? Option A: Buy it built-in Problems: Black Box Can t use it on another vehicle

68 How to get optical flow on your robot? Option B: Buy a special-purpose sensor

69 How to get optical flow on your robot? Option C: DIY Problems: RaspberryPi currently too slow to compute flow in real-time. Webcam suffers from rolling shutter effect

70 How to get optical flow on your robot? Option D: Custom Vision Chips