Outline. Convolution. Filtering

Transcription

1 Filtering

2 Outline Convolution Filtering

3 Logistics HW1 HW2 - out tomorrow

4 Recall: what is a digital (grayscale) image? Matrix of integer values

5 Images as height fields Let s think of image as zero-padded functions

6 How do we characterize image processing operations? Properties of nice functional transformations Additivity Scaling Shift Invariance

7 How do we characterize image processing operations? Properties of nice transformations Additivity Scaling Shift invariance G[i] =T (F [i]) T ( F 1 + F 2 )= G 1 + G 2 G[i j] =T (x[i j])

8 Convolution = F [i] =F [] [i]+f [1] [i 1] +... F [i] = X u F [u] [i u] T (F [i]) = X u F [u]t ( [i u]) impulse response, filter, kernel G[i] = X u F [u]h[i u] where h[i] =T ( [i]),g[i] =T (F [i]) G = F H

9 Example G[i] =F [i] H[i] = X u F [u]h[i u] * 1 2 X

10 Example * G[] = 5x1 = 5 G[1] = 5x2+ 4x1 = 14 G[2] = 5x3 + 4x2 + 2x1 = 25

11 Properties of convolution F H = H F F (H G) =(F H) G (F G)+(H G) =(F + H) G Commutative Associative Distributive Implies that we can efficiently implement complex operations

12 Proof: commutativity H F = X u H[u]F [i u] = X u H[i u]f [u ] where u = i u = X u F [u]h[i u] =F H

13 Size Given F of length N and H of length M, what s size of G = F * H?

14 Size Given F of length N and H of length M, what s size of G = F * H? >>conv(f,h, full ) >>conv(f,h, valid ) >>conv(f,h, same ) N+M-1 N-M+1 N

15 A simpler approach F H Scan original F instead of flipped version. What s the math?

16 (Cross) correlation F H Scan original F instead of flipped version. What s the math? F [i] H[i] = u=k X u= k H[u]F [i + u]

17 Properties Associativity, Commutative poperties do not hold but correlation is easier to think about

18 Convolution vs correlation (1-d) X G[i] =F [i] H[i] = X u F [u]h[i u] (convolution) X = H[i] F [i] = X u H[u]F [i u] (commutative property) G[i] =F [i] H[i] = X H[u]F [i + u] u (cross-correlation) = F [i] H[ i] (exercise for reader!) X

19 2D correlation G[i, j] =F H = kx u= k kx v= k H[u, v]f [i + u, j + v]

20 Convolution vs correlation X X (2-d) Convolution: G[i, j] =F H = H F = X u X v H[u, v]f [i u, j v] h f Correlation: X X X X G[i, j] =F H = H[u, v]f [i + u, j + v] u v X X >> conv2(h,f) convolution >> filter2(h,f) correlation h f Can we compute correlation with convolution?

21 Border effects g full same valid g g g g g f f f g g g g g g

22 Border padding

23 Examples of correlation = Original Blur (with a mean filter)

24 Examples of correlation 1? Original

25 Examples of correlation 1 Original Filtered (no change)

26 Examples of correlation 1 1? Original

27 Examples of correlation 1 Original Shifted left By 1 pixel What would this look like for convolution?

28 Examples of correlation 1 1? Original

29 Examples of correlation 1 1? Original Shifted left By 1 pixel

30 Examples of correlation /16 Original What would this look like for convolution?

31 Examples of correlation ? Original Original

32 Examples of correlation Original 1 ( - /16 ) scaled impulse Unsharp filter

33 Examples????????? Can rotations be represented with a convolution? Are they linear operations G[i,j] = T(F[i,j])?

34 Derivative filters (correlation) apple

35 Derivative filters: motivation rf = rf = apple fx f y q fx 2 + fy 2 s = tan 1 f y f x

36 Gradient magnitude Common approach for discrete filter design: start with continuous operation (like derivative), and discretize to make a filter

37 Question: what happens as we repeatedly convolve an image F with filter H? F F*H

38

39 Gaussian

40 = 1 pixel = 5 pixels = 1 pixels = 3 pixels

41 Implementation Matlab: >> G = FSPECIAL('gaussian',HSIZE,SIGMA) We usually do not need to use larger Gaussians - why? * = 2 a b = a 2 + b 2

42 Finite-support filters What should HSIZE be?

43 Rule-of-thumb Set radius of filter to be 3 sigma

44 Useful representation: Gaussian pyramid Filter + subsample (to exploit redundancy in output) Burt & Adelson 83

45 Smoothing vs edge filters How should filters behave on a flat region with value v?

46 Smoothing vs edge filters How should filters behave on a flat region with value v? Output v Output X X H[i, j] =1 H[i, j] = ij ij

47 Revisiting gradients (let s plot a single row of image as a function)

48 Solution: smooth first Where is the edge? Look for peaks in

49 Combine smoothing and derivative operation into a single filter Derivative of Gaussian We can think of this as an improved gradient filter (Lots of other edge filters) 49

50 Template matching with filters Can we use filtering to build detectors? 5

51 Attempt 1: correlate with eye patch G[i, j] = kx u= k kx v= k H[u, v]f [i + u, j + v] = H T F ij = H F ij cos, H,F ij 2 R (2K+1)2 Useful to think about correlation and convolution Input Filtered Image If we look for high responses, we d find the shirt. What s wrong?

52 Attempt 2: correlate with zero-mean eye patch G[i, j] =(H H) T F ij k,l True detections False detections Input Filtered Image (scaled) Thresholded Image 52

53 Attempt 3: SSD SSD[i, j] = H F ij 2 =(H F ij ) T (H F ij ) Can this be implemented with filtering? k,l True detections Input 1- sqrt(ssd) Thresholded Image 53

54 What will SSD find here? k,l Input 1- sqrt(ssd) 54

55 Normalized cross correlation NCC[i, j] = = HT F H F H T F p HT H p F T F = cos True detections Input Normalized X-Correlation Thresholded Image 55

56 Modern filter banks Convolutional Neural Nets (CNNs) Lecun et al 98 Learn filters from training data to look for low, mid, and high-level features 56

57 Any linear shift-invariant operation can be characterized by a convolution (Convolution) correlation intuitively corresponds to (flipped) matched-filters Derive filters by continuous operations (derivative, Gaussian, ) Contemporary application: convolutional neural networks 57