ECE Digital Image Processing and Introduction to Computer Vision

Transcription

1 ECE Digital Image Processing and Introduction to Computer Vision Depart. of ECE, NC State University Instructor: Tianfu (Matt) Wu Spring 2017 Outline Recap, image degradation / restoration Template Matching Image Pyramid Gaussian pyramid Laplacian pyramid Application: Image blending 1

2 Recap, A Model of the Image Degradation / Restoration Process Optimization Ad-hoc: g x, y = f x, y h x, y + η x, y G u, v = F u, v H u, v + N u, v Estimation: interactive inspection or mathematical modeling Removal: inverse filtering Type: uncorrelated or not? Parameters: interactive inspection in spatial / freq. domain Removal: filtering in spatial / freq. domain Goal: find in image Template Matching Main challenge: What is a good similarity or distance measure between two patches? Correlation Zero-mean correlation Sum Square Difference Normalized Cross Correlation Slides: James, Hoiem and others 2

3 Matching with filters Goal: find in image Method 0: filter the image with eye patch h[ m, n] = åg[ k, l] f [ m + k, n + l] k, l f = image g = filter What went wrong? Input Filtered Image Goal: find Matching with filters in image Method 1: filter the image with zero-mean eye mean of g h m, n = 4 g k, l g f[m + k, n + l] <,= True detections False detections Input Filtered Image (scaled) Thresholded Image 3

4 Matching with filters Goal: find in image Method 2: SSD h[ m, n] = å( g[ k, l] - f [ m + k, n + l]) k, l 2 True detections Input 1- sqrt(ssd) Thresholded Image Matching with filters h[ m, n] = å( g[ k, l] - f [ m + k, n + l]) k, l 2 Can SSD be implemented with linear filters? 4

5 Matching with filters Goal: find in image Method 2: SSD h[ m, n] = å( g[ k, l] - f [ m + k, n + l]) k, l What s the potential downside of SSD? 2 Input 1- sqrt(ssd) Matching with filters Goal: find in image Method 3: Normalized cross-correlation mean template mean image patch h[ m, n] = æ ç è å k, l å k, l ( g[ k, l] - g )( f [ m - k, n - l] - f ( g[ k, l] - g) 2 å k, l ( f [ m - k, n - l] - f m, n m, n ) ) 2 ö ø 0.5 Matlab: normxcorr2(template, im) 5

6 Matching with filters Goal: find in image Method 3: Normalized cross-correlation True detections Input Normalized X-Correlation Thresholded Image Matching with filters Goal: find in image Method 3: Normalized cross-correlation True detections Input Normalized X-Correlation Thresholded Image 6

7 Q: What is the best method to use? A: Depends SSD: faster, sensitive to overall intensity Normalized cross-correlation: slower, invariant to local average intensity and contrast But really, neither of these baselines are representative of modern recognition. Q: What if we want to find larger or smaller objects? A1: Image Pyramid 7

8 Image Pyramids How to compute them? Gaussian Pyramids Image Gaussian Filter Low-Pass Filtered Image DownSample Low-Res Image 8

9 3/23/17 Recall Anti-Aliasing 1) Sample more often when possible 2) Get rid of all frequencies that are greater than half the new sampling frequency Will lose information But it s better than aliasing Apply a smoothing filter Recall Anti-Aliasing Source: Collins 9

10 Gaussian Pyramids Source: Forsyth Template Matching with Image Pyramids Input: Image, Template 1. Compute Gaussian pyramid 2. Search over all levels 3. Take responses above some threshold, perhaps with non-maxima suppression 10

11 Template Matching with Image Pyramids Gaussian Pyramids Same filter kernel? Yes Gaussian kernel: G > G >? G >@ f = G >? G >@ f G > f σ C = σ C C D + σ C 11

12 Gaussian Pyramids Gaussian Pyramids For an image with N N pixels, how many pixels in total in a pyramid with P+1 levels? 12

13 3/23/17 Gaussian Pyramids Gaussian / Laplacian Pyramids Recall original = smoothed (5x5) detail f + a ( f - f * g ) = (1 + a ) f - a f * g = f * ((1 + a )e - g ) image unit impulse (identity) blurred image unit impulse Gaussian Laplacian of Gaussian 13

14 Gaussian / Laplacian Pyramids Image = G 1 Smooth, then downsample Downsample (Smooth(G 1 )) G 2 Downsample (Smooth(G 2 )) G 3 G N = L N G 1 - Smooth(Upsample(G 2 )) L 1 L 2 L 3 G 3 - Smooth(Upsample(G 4 )) G 2 - Smooth(Upsample(G 3 )) Use same filter for smoothing in each step (e.g., Gaussian with σ = 2) Downsample/upsample with nearest interpolation Laplacian Pyramids Source: Forsyth 14

15 Ex. Hybrid Image in Laplacian Pyramid High frequency à Low frequency Reconstruction with Gaussian / Laplacian Pyramids We can reconstruct the original image using the coarsest Gaussian level and all the Laplacian levels. 15

16 Spatial Gaussian pyramid Fourier Laplacian pyramid Fourier Spatial Source: keren Application: Image Blending Slides: Fergus 16

17 Application: Image Blending Feathering / Alpha blending = Encoding transparency I(x,y) = (ar, ag, ab, a) I blend = a I left + (1-a) I right Burt and Adelson 1983b Affect of Window Size 1 0 left right 1 0 Ghosting happens if the transition is too low. 17

18 Affect of Window Size Seams happens if the transition is too fast. Good Window Size 1 0 Optimal Window: smooth but not ghosted 18

19 What is the Optimal Window? The size of the transition zone, relative to the size of image features, plays a critical role in image: To avoid seams (visible edge) window >= size of largest prominent feature To avoid ghosting (double exposure) window <= 2*size of smallest prominent feature Both requirements can not be satisfied simultaneously if an image contains both a diffuse background and small foreground objects. A suitable transition zone can only be selected if the images to be splined occupy a relatively narrow spatial frequency band. Natural to cast this in the Fourier domain largest frequency <= 2*size of smallest frequency image frequency content should occupy one octave (power of two) What if the Frequency Spread is Wide FFT Idea (Burt and Adelson) Compute F left = FFT(I left ), F right = FFT(I right ) Decompose Fourier image into octaves (bands) F left = F left1 + F left2 + Feather corresponding octaves F lefti with F right i Can compute inverse FFT and feather in spatial domain Sum feathered octave images in frequency domain Better implemented in spatial domain in practice. 19

20 Coarse structure should blend very slowly between images (lots of feathering), while fine details should transition more quickly Pyramid Blending Left pyramid blend Right pyramid 20

21 Pyramid Blending laplacian level 4 laplacian level 2 laplacian level 0 left pyramid right pyramid blended pyramid 21

22 Laplacian Pyramid: Region Blending General Approach: 1. Build Laplacian pyramids LA and LB from images A and B 2. Build a Gaussian pyramid GR from selected region R 3. Form a combined pyramid LS from LA and LB using nodes of GR as weights: LS i, j = GR i, j, LA i, j + 1 GR i, j LB(I, j) 4. Collapse the LS pyramid to get the final blended image Blending Regions 22

23 Horror Photo david dmartin (Boston College) Summary Template Matching Image Pyramid Gaussian pyramid Laplacian pyramid Application: Image blending 23