TRACKING and DETECTION in COMPUTER VISION Filtering and edge detection

Transcription

1 Technischen Universität München Winter Semester 0/0 TRACKING and DETECTION in COMPUTER VISION Filtering and edge detection Slobodan Ilić

2 Overview Image formation Convolution Non-liner filtering: Median and Bilateral filters Gaussian smoothing Image derivatives: Gradients and Laplacians Edge detection

3 Image formation Basics of image formation

4 Image formation Image formation occurs when a sensor registers radiation that has interacted with physical objects. It consists of several components: Basics of image formation

5 Image formation Image formation occurs when a sensor registers radiation that has interacted with physical objects. It consists of several components: An imaging function - a fundamental abstraction of an image Basics of image formation

6 Image formation Image formation occurs when a sensor registers radiation that has interacted with physical objects. It consists of several components: An imaging function - a fundamental abstraction of an image A geometric model - projection of the D world into D representation Basics of image formation

7 Image formation Image formation occurs when a sensor registers radiation that has interacted with physical objects. It consists of several components: An imaging function - a fundamental abstraction of an image A geometric model - projection of the D world into D representation A radiometric model - describes how light radiated from the object is measured by the imaging sensor. Basics of image formation

8 Image formation Image formation occurs when a sensor registers radiation that has interacted with physical objects. It consists of several components: An imaging function - a fundamental abstraction of an image A geometric model - projection of the D world into D representation A radiometric model - describes how light radiated from the object is measured by the imaging sensor. A color model - describes how different spectral measurements are related to image colors Basics of image formation

9 Imaging function I is an imaging function defined on the area positive real numbers: Because of the discretization we have: Or in general: I : Ω R R + ;(x, y) I(x, y). and takes values in the Ω = [, 640] [, 480] Z, R + = [0, 55] Z + I = I(x, y) ={I R (x, y),i G (x, y),i B (x, y)} Ω Basics of image formation 4

10 Geometric model The pinhole camera model (a) Projection geometry (b) Image plane in front. (c) Notations for pinhole camera model Basics of image formation 5

11 Perspective projection Basics of image formation 6

12 Perspective projection Wold coordinate system is identical to the camera coordinate system with the origin at the center of projection. Under perspective projection, the object point with world coordinates projects to the image point with ideal image coordinates: Ideal image coordinates are expressed in terms of the image coordinate system with the origin at the optical center. Basics of image formation 6

13 Perspective projection Wold coordinate system is identical to the camera coordinate system with the origin at the center of projection. Under perspective projection, the object point with world coordinates P =(X, Y, Z) projects to the image point with ideal image coordinates: x = f X Z y = f Y Z Ideal image coordinates are expressed in terms of the image coordinate system with the origin at the optical center. Basics of image formation 6

14 Lenses and Discrepancies from the Pinhole Camera d f D thin lens equation D + d = f n = f a aperture measure is f-number: D inf f = d 7

15 Aperture Aperture measure is f-number: n = f a Large(wide) aperture, small f-number a = f 4. Small aperture, large f-number a = f 9 Why use wide aperture if images can be made sharp with small aperture? Basics of image formation 8

16 Improving images Noise removal Contrast increase 9

17 Detecting edges Steps in edge detection 0

18 Image processing Image processing is in general not part of Computer Vision, but it is sometimes a very necessary preprocessing step. Image processing provides a number of methods to convert an image into something suitable for analysis. There are two main approaches: Processing in the spacial domain: Point processing (brightness, contrast, histogram equalization etc.) Filtering Processing in the frequency domain: Fourier transform

19 Image processing Image processing is in general not part of Computer Vision, but it is sometimes a very necessary preprocessing step. Image processing provides a number of methods to convert an image into something suitable for analysis. There are two main approaches: Processing in the spacial domain: Point processing (brightness, contrast, histogram equalization etc.) Filtering Processing in the frequency domain: Fourier transform

20 Filtering Original image Blurred image From the draft of Szeliski s book: Filtering is based on neighborhood operations.

21 Neighborhood filtering Correlation and Convolution From the draft of Szeliski s book: H(i, j), i, j convolution kernel J(x, y) I(x, y), x, y x dim,y dim J(x, y) =H I = J(x, y) =H I = i= H(i, j)i(x + i, y + j) j= i= j= H(i, j)i(x i, y j) - correlation - convolution

22 Convolution Example Rotate the convolution kernel H Rotate I - Apply correlation - - J(x, y) =H I = H(i, j)i(x + i, y + j) i= j= 4

23 Convolution Example Step H I H y - - I x J(x, y) =H I 5

24 Convolution Example Step H I H y - - I x J(x, y) =H I 6

25 Convolution Example Step H I H y - - I J(x, y) =H I x 7

26 Convolution Example Step 4 H I H y - - I J(x, y) =H I x 8

27 Convolution Example Step 5 H I H y I x J(x, y) =H I 9

28 Convolution Example Step 6 H I H y I x J(x, y) =H I 0

29 Example * =

30 Example * =

31 Expressing convolution Mathematically convolution can be expresses as: + + J(x, y) =H I = H(i, j)i(x i, y j) i= j= if we change the variables i x i and j y j we get: + + J(x, y) =H I = H(x i, y j)i(i, j) i= j= with the image and the kernel playing interchangeable roles.

32 Convolution by shifting, copying and multiplying the image For each element H(i,j) in the kernel matrix, image I is copied into a zero-padded image, P starting at (i,j). Each P is m u l t i p l i e d b y t h e corresponding weight H(i,j). All the P images are summed pixel-wise, then divided by the sum of the elements of h. The result is copped out of the center of the accumulated P s. J(x, y) =H I = + + i= j= H(x i, y j)i(i, j) 4

33 Shift invariance δ a,b (i, j) = {, for i = a and j = b 0, elsewhere H 5 δ a,b H(x, y) =H(x a, c b) The kernel(point spread function) H is shift invariant or space-invariant because the entries in H do not depend on the position (x, y) in the output image.

34 Convolution properties Commutativity Associativity Distributivity H I = I H H (G I) =(H G) I H (I + J) =(H I)+(H J) Associativity with scalar multiplication a(h I) =(ah) I = H (ai) 6

35 Padding (border effects) zero clamp mirror 7

36 Non-linear filtering Linear filters combine input pixels in a way that depends on where a pixel is in the image and not on its value Non-linear filters take into account input pixel values before deciding how to use them in the output. 8

37 Linear vs. non liner filters original image salt&pepper noise added average filter median filter 9

38 Median Filtering Sort: ( ) Example Original signal: Noisy signal: Filter by [ ]/: Filter by x median filter:

39 Median Filter Median filters are nonlinear Median filtering reduces noise without blurring edges and other sharp details Median filtering is particularly effective when the noise pattern consists of strong, spike-like components. (Salt-and-pepper noise.)

40 Bilateral filter It is nonlinear filtering technique based on: Domain(smoothing) filter - It is based on closeness function which accounts for spacial distance between the central pixel x and its neighbors ξ c(ξ, x) =e d(ξ,x) σ d, d(ξ, x) = ξ x Range filter - It is based on the similarity function between image intensities between the central pixel and its neighbors I(ξ) I(x) σ d s(ξ, x) =e ( δ(i(ξ),i(x)) σr ), δ(i(ξ),i(x)) = I(ξ) I(x) - desired amount of spacial smoothing σ r - desired amount of combining of pixel values

41 Bilateral filter Combining similarity and closeness functions we obtain: h(x) = c ξ Ω I(ξ)c(ξ, x)s((i(ξ),i(x)) c = ξ Ω c(ξ, x)s((i(ξ),i(x)) This filter is know to reduce noise and to preserve edges

42 Examples C.Tomasi, R. Manduchi, "Bilateral Filtering for gray and color images", Sixth International Conference on Computer Vision, pp 89-46, New Delhi, India,

43 Gaussian smoothing G(u, v) =e u +v σ Gaussian kernel Gaussian function 5

44 Gaussian smoothing example Original image Smoothed σ = Smoothed Smoothed σ = σ =4 6

45 Separability of Gaussians G(u, v) =g(u)g(v) G(u, v) =e u +v σ = G(u, v) =e ( u σ ) e ( v σ ) Gaussian separation speeds up computation, because with the separable kernel G the convolution can be separated in two onedimensional convolutions like: n n J(x, y) = g(u) g(v)i(x u, y v) J(x, y) = n u= n u= n v= n g(u)φ(x u, y) φ(x, y) = this requires m multiplications and (m-) additions compared to m^ multiplications and m^- additions with D kernel, where m=n+. n v= n g(v)i(x, y v) 7

46 Gaussian normalization G(x) = πσ e x σ G(x, y) = πσ e x +y σ Since we discretize in practice we normalize not by the integral, but by the sum G(x, y) = c e x +y σ c = n i= n n j= n G(i, j) 8

47 Image derivatives Remember that the image can be represented as a function of its pixel intensities I(x) I(x 0 ) x 0 consequently we should be able to compute derivatives of this function... x 9

48 Linear approximation I(x) I(x 0 ) I(x 0 ) x x x 0 x 0 + x x I(x 0 + x) I(x 0 )+ I(x 0) x x + h.o.t I(x 0 ) x = lim x 0 I(x 0 + x) I(x 0 ) x I(x 0 + x) I(x 0 ) 40

49 Image D profile example D image line profile D image line profile smoothed with Gaussian 4

50 Example images Discontinuity 4 Smoothed with Gaussian

51 Example images derivatives 0 0 Edges correspond to the fast changes in the intensities Magnitude of the derivatives is large -0 4

52 Derivatives as linear filters Image derivative: I(x 0 ) x = lim x 0 I(x 0 + x) I(x 0 ) x I(x 0 + x) I(x 0 ) can be implemented as linear filters: H I =[ ] [I(x 0 ) I(x 0 + x)] T or in its symmetric version: H I = [ 0 ] [I(x 0 x) I(x 0 ) I(x 0 + x)] T 44

53 Partial image derivatives If we consider an image I as a function of vector variables I(x,y) we write image gradient using partial derivatives as: I(x, y) = I x i x + I y i y This can be written using gradient filters: I(x, y) = (D x I) i x +(D y I) i y And we can define gradient intensity and direction: I(x, y) = (D x I) +(D y I) ψ( I) = arctan ( Dy I D x I ) 45

54 Gradient filters Basic derivative filters: D x = 0 0 D y = Prewitt gradient filter Sobel gradient filter D x = 0 0 D x = D y = D y =

55 Gradients smoothed with Gaussians Images smoothed with Gaussians and then filtered with derivatives filters: I x = D x (G I) I y = D y (G I) Because of the associativity: I x =(D x G) I = G x I I y =(D y G) I = G y I images can be directly convolved by the derivatives of the Gaussian filters. 47

56 Derivatives of Gaussians in D G(x) = πσ e x σ G x = G x = x σ πσ e x σ 48

57 Derivatives of Gaussians in D Images are from Carlo Tomasi Computer Vision lecture notes G(x, y) = πσ e x +y σ G x = G x = x πσ 4 e x +y σ G y = G y = y πσ 4 e x +y σ 49

58 Practical Aspects and normalization Derivatives without normalization and due to separability: G x (x, y) = x σ e G y (x, y) = y σ e in discreet case: I x (x, y) = = I y (x, y) = = n i= n j= n n i= n n n i= n n G x (x i) n i= n j= n G(x i) n j= n n j= n 50 x +y σ x +y σ I(i, j)g x (x i, y j) = I(i, j)g(y j) = I(i, j)g y (x i, y j) = I(i, j)g y (y j) = = G x (x)g(y) = G y (y)g(x) n j= n n j= n G(y j) G y (y j) n i= n n i= n I(i, j)g x (x i) I(i, j)g(x i)

59 With normalization I x (x, y) = I y (x, y) = n i= n n i= n G x (x i) G(x i) n j= n n j= n I(i, j)g(y j) I(i, j)g y (y j) Normalize by the sum of the filter samples: G u (u) =k g G u (u), where G u (u) = ue ( u σ ) and k d = n ug u (u) u= n G(u) =k G(u), where G(u) =e ( u σ ) and k = n G u (u) u= n 5

60 Image directional gradients Gradient in x direction Gx Gradient in y direction Gy 5

61 Gradient magnitude I(x, y) = (G x I) +(G y I) σ =0.5 σ =.0 σ =.5 σ =.0 5

62 Gradient orientation T h e u p p e r l e f t corner of the image of 50x50 pixels large. Gradient orientation computed as: ψ( I) = arctan ( G x I G y I ) 54

63 Second order derivatives Mathematically we have: I(x 0 ) x = lim x 0 I (x 0 + x) I (x 0 ) x I (x 0 + x) I (x 0 ) Further simplified: I(x 0 ) x = I = I(x 0 + x) I(x 0 + x)+i(x 0 ) In D: I(x,y) x = I(x, y) = I x + I y 55

64 Example images derivatives 0 Original image 0 first order derivative -0 5 Smoothed with Gaussian 0 second order derivative 56-5

65 Second order derivative filters Second order derivative or Laplacian: can be implemented as simple linear filter: H = I(x, y) = I x + I y I(x 0 ) x = I(x 0 + x) I(x 0 + x)+i(x 0 ) I = H [I(x 0 + x) I(x 0 + x) I(x 0 )] T or by separately in x and y direction: H xx = H T yy = [ ] I = H xx I + H T yy I 57

66 Edge detection To compute image edges we need: Image smoothing Image derivatives Non-maximal suppression Thresholding 58

67 D edge detection 59

68 D edge detection 60

69 Convolving with the first order derivative of Gaussian Smoothing removes noise and amplifies signal at the edge discontinuities. 6

70 Zero crossings and convolution with Laplacian Looking at minima or maxima of fist order image derivatives is the same as looking for zero-crossings of second order image derivatives. 6

71 Multi-scale edge detection The amount of smoothing controls the scale at which we analyze the image. Small smoothing brings edges at a fine scale Signal noise is not suppressed 6 Increased smoothing suppresses noise

72 D edge detection Original image Gradient image smoothed with Gaussian Gradient image smoothed with Gaussian 64

73 D edge detection Original image Gradient image smoothed with Gaussian Gradient image smoothed with Gaussian σ =.0 64

74 D edge detection Original image Gradient image smoothed with Gaussian σ =0.5 Gradient image smoothed with Gaussian σ =.0 64

75 Non-maximal suppression Non-maxima suppression does thinning of the edges. The gradient orientations are quantized into four bins original quantization of gradient orientations Pick neighboring pixels of the edge pixel in the direction of the gradient and take the one with the maximum gradient magnitude, while the others are set to zero. gradient magnitude gradients after non-maximum suppression 65

76 Canny edge detection Canny edge detection composes of all already mentioned steps which are: Image smoothing Image derivatives Non-maximal suppression Hysteresis thresholding 66

77 Hysteresis thresholding Apply two thresholds Thigh and Tlow to follow edges Algorithm:. Search for the pixel with the gradient magnitude value in the non-maximum suppressed image is higher then Thigh. Recursively search its neighbors and assign them to the edge if their gradient magnitude value in the non-maximum suppressed image is higher then the Tlow.. otherwise, stop if the gradient magnitude is bellow Tlow or the pixel is already visited and assigned to be on an edge and go to. 67

78 Canny examples Gradient image Th=50, Tl=0 Th=00, Tl=0 Th=50, Tl=40 68