Computational Photography

Transcription

1 Computational Photography Si Lu Spring onal_photography.htm 04/0/208

2 Last Time o Digital Camera History of Camera Controlling Camera o Photography Concepts

3 Today o Filters and its applications noisy image naïve denoising Gaussian blur better denoising edge-preserving filter Slide credit: Sylvain Paris and Frédo Durand

4 Slide credit: D. Hoiem The raster image (pixel matrix)

5 The raster image (pixel matrix) Slide credit: D. Hoiem

6 Perception of Intensity Slide credit: C. Dyer

7 Perception of Intensity Slide credit: C. Dyer

8 Color Image R G B Slide credit: D. Hoiem

9 Image Filtering o Image filtering: compute function of local neighborhood at each pixel position o One type of Local operator, Neighborhood operator, Window operator o Useful for: Enhancing images o Noise reduction, smooth, resize, increase contrast, etc. Extracting information from images o Texture, edges, distinctive points, etc. Detecting patterns o Template matching, e.g., eye template Slide credit: C. Dyer Source: D. Hoiem

10 Blurring in the Real World Camera shake = * Source: Fergus, et al. Removing Camera Shake from a Single Photograph, SIGGRAPH 2006 Bokeh: Blur in out-of-focus regions of image Slide credit: C. Dyer Source:

11 Image Correlation Filtering o Select a filter g g is called a filter, mask, kernel, or template o Center filter g at each pixel in image f o Multiply weights by corresponding pixels o Set resulting value in output image h o Linear filtering is sum of dot product at each pixel position o Filtering operation called cross-correlation, and denoted h = f g Slide credit: C. Dyer

12 Example: Box Filter g[, ] Slide credit: David Lowe

13 Image Filtering g[, ] f [.,.] h[.,.] h[ m, n] k, l g[ k, l] f [ m k, n l] Credit: S. Seitz

14 Image Filtering g[, ] f [.,.] h[.,.] h[ m, n] k, l g[ k, l] f [ m k, n l] Credit: S. Seitz

15 Image Filtering g[, ] f [.,.] h[.,.] h[ m, n] g[ k, l] k, l f [ m k, n l] Credit: S. Seitz

16 Image Filtering g[, ] f [.,.] h[.,.] h[ m, n] k, l g[ k, l] f [ m k, n l] Credit: S. Seitz

17 Image Filtering g[, ] f [.,.] h[.,.] h[ m, n] k, l g[ k, l] f [ m k, n l] Credit: S. Seitz

18 Image Filtering g[, ] f [.,.] h[.,.] ? h[ m, n] k, l g[ k, l] f [ m k, n l] Credit: S. Seitz

19 Image Filtering g[, ] f [.,.] h[.,.] ? h[ m, n] k, l g[ k, l] f [ m k, n l] Credit: S. Seitz

20 g[ Image Filtering, ] f [.,.] h[.,.] h[ m, n] k, l g[ k, l] f [ m k, n l] Credit: S. Seitz

21 Box Filter What does it do? Replaces each pixel with an average of its neighborhood g[, ] Achieves smoothing effect (i.e., removes sharp features) Weaknesses: Blocky results Axis-aligned streaks Slide credit: David Lowe

22 Smoothing with Box Filter Slide credit: C. Dyer

23 Properties of Smoothing Filters o Smoothing Values all positive Sum to constant regions same as input Amount of smoothing proportional to mask size Removes high-frequency components low-pass filter Slide credit: C. Dyer

24 Gaussian Filtering o Weight contributions of neighboring pixels by nearness x 5, = o Constant factor at front makes volume sum to o Convolve each row of image with D kernel to produce new image then convolve each column of new image with same D kernel to yield output image Slide credit: Christopher Rasmussen

25 Smoothing with a Gaussian o Smoothing with a box actually doesn t compare at all well with a defocused lens o Most obvious difference is that a single point of light viewed in a defocused lens looks like a fuzzy blob; but the averaging process would give a little square o Gaussian is isotropic (i.e., rotationally symmetric) o A Gaussian gives a good model of a fuzzy blob o It closely models many physical processes (the sum of many small effects) Slide by D.A. Forsyth

26 What does Blurring take away? Slide credit: C. Dyer original

27 What does Blurring take away? Slide credit: C. Dyer smoothed (5x5 Gaussian)

28 Smoothing with Gaussian Filter

29 Smoothing with Box Filter

30 input Slide by S. Paris

31 box average Slide by S. Paris

32 Gaussian blur Slide by S. Paris

33 Gaussian Filters o What parameters matter here? o Standard deviation,, of Gaussian: determines extent of smoothing σ = 2 with 30 x 30 kernel σ = 5 with 30 x 30 kernel Source: D. Hoiem

34 Slide credit: C. Dyer

35 Smoothing with a Gaussian Parameter σ is the scale / width / spread of the Gaussian kernel, and controls the amount of smoothing Slide credit: C. Dyer for sigma=:3:0 h = fspecial('gaussian, hsize, sigma); out = imfilter(im, h); imshow(out); pause; end

36 Gaussian filters = pixel = 5 pixels = 0 pixels = 30 pixels Slide credit: C. Dyer

37 Gaussian Filters o What parameters matter here? o Size of kernel or mask σ = 5 with 0 x 0 kernel σ = 5 with 30 x 30 kernel Slide credit: C. Dyer

38 How big should the filter be? o Gaussian function has infinite support but need a finite-size kernel o Values at edges should be near 0 o 98.8% of area under Gaussian in mask of size 5σ x 5σ o In practice, use mask of size 2k+ x 2k+ where k 3 o Normalize output by dividing by sum of all weights Slide credit: C. Dyer

39 Sharpening Filters ? Original (Note that filter sums to ) Source: D. Lowe

40 Sharpening Filters ? Original (Note that filter sums to ) Source: D. Lowe

41 Sharpening Filters Original Sharpening filter - Sharpen an out of focus image by subtracting a multiple of a blurred version Source: D. Lowe

42 Sharpening Source: D. Lowe

43 Sharpening by Unsharp Masking o h = f + k(f * g) where k is a small positive constant and g = called a Laplacian mask o Called unsharp masking in photography

44 Sharpening using Unsharp Mask Filter Original Filtered result Slide credit: C. Dyer

45 Application: Edge Detection Sobel Slide credit: C. Dyer Vertical Edge (absolute value)

46 Application: Edge Detection Sobel Slide credit: C. Dyer Horizontal Edge (absolute value)

47 Application: Hybrid Images Gaussian Filter I I G G (-G 2 ) I 2 Laplacian Filter unit impulse GaussianLaplacian of Gaussian A. Oliva, A. Torralba, P.G. Schyns, Hybrid Images, SIGGRAPH 2006

48

49 Application: XDoG Filters Input XDoG Input XDoG Gaussian filtering results Winnemoller, H., XDoG: advanced image stylization with extended Difference-of-Gaussians NPAR 20

50 Application: Painterly Filters o Many methods have been proposed to make a photo look like a painting o Today we look at one: Painterly-Rendering with Brushes of Multiple Sizes o Basic ideas: Build painting one layer at a time, from biggest to smallest brushes At each layer, add detail missing from previous layer A. Hertzmann, Painterly rendering with curved brush strokes of multiple sizes, SIGGRAPH 998. Slide credit: S. Chenney

51 Algorithm function paint(sourceimage,r... R n ) // take source and several brush sizes { canvas := a new constant color image // paint the canvas with decreasing sized brushes for each brush radius R i, from largest to smallest do { // Apply Gaussian smoothing with a filter of size const * radius // Brush is intended to catch features at this scale referenceimage = sourceimage * G(fs R i ) // Paint a layer paintlayer(canvas, referenceimage, Ri) } return canvas } Slide credit: S. Chenney

52 Algorithm 2 procedure paintlayer(canvas,referenceimage, R) // Add a layer of strokes { S := a new set of strokes, initially empty D := difference(canvas,referenceimage) // euclidean distance at every pixel for x=0 to imagewidth stepsize grid do // step in size that depends on brush radius for y=0 to imageheight stepsize grid do { // sum the error near (x,y) M := the region (x-grid/2..x+grid/2, y-grid/2..y+grid/2) areaerror := sum(d i,j for i,j in M) / grid 2 if (areaerror > T) then { // find the largest error point (x,y) := max D i,j in M s :=makestroke(r,x,y,referenceimage) add s to S } } paint all strokes in S on the canvas, in random order } Slide credit: S. Chenney

53 Results Original Biggest brush Medium brush added Finest brush added Slide credit: S. Chenney

54 More filters (3 mins Break) Slide credit: S. Chenney

55 Filter Re-cap noisy image naïve denoising Gaussian blur better denoising edge-preserving filter Slide credit: Sylvain Paris and Frédo Durand

56 Median Filter Replace pixel by the median value of its neighbors No new pixel values introduced Removes spikes: good for impulse, salt & pepper noise Slide credit: C. Dyer

57 Median Filter Salt and pepper noise Median filtered Slide credit: M. Hebert, C. Dyer Plots of a row of the image Matlab: output im = medfilt2(im, [h w])

58 Median Filter o Median filter is edge preserving Slide credit: C. Dyer

59 Slide credit: C. Dyer

60 input 9x9 median filter output Slide credit: C. Dyer images by J. Plush

61 Bilateral filter o Tomasi and Manduci CCV98.pdf o Related to SUSAN filter [Smith and Brady 95] Digital-TV [Chan, Osher and Chen 200] sigma filter Slide credit: F. Durand

62 Start with Gaussian filtering o Here, input is a step function + noise J f I output input Slide credit: F. Durand

63 Gaussian filter as weighted average o Weight of x depends on distance to x J(x) f (x,x) I(x) x x x x x output input Slide credit: F. Durand

64 The problem of edges o Here, x o It is too different pollutes our estimate J(x) J(x) f (x,x) I(x) x x x x x output input Slide credit: F. Durand

65 Principle of Bilateral filtering [Tomasi and Manduchi 998] o Penalty g on the intensity difference J(x) k(x) f (x,x) g(i(x) I(x)) I(x) x x x I(x) output Slide credit: F. Durand input

66 Bilateral filtering [Tomasi and Manduchi 998] o Spatial Gaussian f J(x) k(x) f (x,x)g(i(x) I(x)) x I(x) x x x output input Slide credit: F. Durand

67 Bilateral filtering [Tomasi and Manduchi 998] o Spatial Gaussian f o Gaussian g on the intensity difference J(x) k(x) f (x,x) g(i(x) I(x))I(x) x x x I(x) output input Slide credit: F. Durand

68 Normalization factor [Tomasi and Manduchi 998] o k(x)= J(x) k(x) f (x,x) g(i(x) I(x)) x f (x,x) g(i(x) I(x)) I(x) x output input Slide credit: F. Durand

69 Blur from averaging across edges input * output * * Same Gaussian kernel everywhere. Slide credit: P. Sylvain

70 Bilateral filter: no averaging across edges input * output * * The kernel shape depends on the image content. Slide credit: P. Sylvain

71 Parameter for intensity difference Gaussian g r = 0. r = 0.25 r = (Gaussian blur) input s = 2 Parameter for spatial distance Gaussian f s = 6 s = 8 Slide credit: P. Sylvain

72 Parameter for intensity difference Gaussian g r = 0. r = 0.25 r = (Gaussian blur) input s = 2 Parameter for spatial distance Gaussian f s = 6 s = 8 Slide credit: P. Sylvain

73 Result Input Output Tomasi and Manduchi 998

74 Other view o The bilateral filter uses the 3D distance Slide credit: F. Durand

75 Speed o Direct bilateral filtering is slow (minutes) o Accelerations exist: Subsampling in space & range o Durand & Dorsey 2002 o Paris & Durand 2006 Limit to box kernel & intelligent maintenance of histogram o Weiss 2006 Slide credit: F. Durand

76 Local filters o Compute a new value at each pixel using its neighboring pixels o Box filter o Gaussian filter o Median filter o Bilateral filter

77 Non-local means filter o Compute a new value at each pixel from the whole image final value at pixel i weight of pixel j value at pixel j Buades, A., Coll, B., Morel, J.-M. A non-local algorithm for image denoising. CVPR 2005

78 Weight : patch centered at pixel i : patch centered at pixel j Similar pixel neighborhoods give a large weight Reprint from Buades et al. 2005

79 Input Gaussian Anisotropic Total variation Neighborhood NL-means Reprint from Buades et al. 2005

80 Non-local means filter o High-quality o Slow Fast non-local means algorithms available o BM3D:

81 BM3D o Image denoising by sparse 3-D transformdomain collaborative filtering (TIP 2007)

82 BM3D o Patch-based o Non-local method o Two-stage denoising o Collaborative denoising o Denoise in frequency domain

83 BM3D

84 BM3D o Results An Analysis and Implementation of the BM3D Image Denoising Method

85 BM3D An Analysis and Implementation of the BM3D Image Denoising Method

86 Video de-noise o We know how to de-noise an image o How about video? E. P. Bennett and L. McMillan. Video Enhancement using Per-pixel Virtual Exposures SIGGRAPH 2005

87 Gaussian filter in video cube o Blurring artifacts Not edge-preserving Motion blur

88 Bilateral filter in video cube o Cannot remove shot noise Reprint from [Bennett and McMillan 2005]

89 ASTA Filter [Bennett and McMillan 05] o Build upon bilateral filter o Find similar pixels in a video cube for filtering Patch-based similarity measurement o Adaptive Spatial-temporal Accumulation Filter Prefer temporal neighbors

90 Patch-based similarity measurement frame pt frame st

91 Similarity measure Reprint from [Bennett and McMillan 2005]

92 Adaptive Filtering Reprint from [Bennett and McMillan 2005]

93 Results (filtering + tone mapping) Input Naïve method ASTA Reprint from [Bennett and McMillan 2005]

94 Next Time o Color o Lighting