Image Processing /6.865 Frédo Durand A bunch of slides by Bill Freeman (MIT) & Alyosha Efros (CMU)

Size: px

Start display at page:

Download "Image Processing /6.865 Frédo Durand A bunch of slides by Bill Freeman (MIT) & Alyosha Efros (CMU)"

Kelley Lambert
5 years ago
Views:

1 Image Processing 6.815/6.865 Frédo Durand A bunch of slides by Bill Freeman (MIT) & Alyosha Efros (CMU)

2 define cumulative histogram work on hist eq proof rearrange Fourier order discuss complex exponentials with eigenfunctions 2

3 Warning Think about final projects 3

4 Class morph 4

5 Image processing Filtering, Convolution, and our friend Joseph Fourier

6 What is an image? We can think of an image as a function, f, from R 2 to R: f( x, y ) gives the intensity at position ( x, y ) Realistically, we expect the image only to be defined over a rectangle, with a finite range: f: [a,b]x[c,d] [0,1] A color image is just three functions pasted together. We can write this as a vectorvalued function:

7 Images as functions

8 Image Processing image filtering: change range of image f g(x) = h(f(x)) f f h x x image warping: change domain of image g(x) = f(h(x)) h f x x

9 Image Processing image filtering: change range of image g(x) = h(f(x)) h image warping: change domain of image g(x) = f(h(x)) h

10 Point Processing The simplest kind of range transformations are these independent of position x,y: g = t(f) This is called point processing. Important: every pixel for himself spatial information completely ignored!

11 Negative

12 Contrast Stretching input output

13 Image Histograms histogram H(f) = # or % pixels with value f (implies binning of the values) cumulative histogram: C(f) = f f H(f) = # or % of pixels with value f Cumulative Histograms

14 Histogram Equalization point transformation: g(x)=t(f(x)) uniform across image (t does not depend on x) monotonic (preserve intensity ordering) so that histogram of g is uniform perfect uniform only possible with continuous histogram

15 Qualitative Histogram equalization Qualitative 15

16 Derivation Normalized cumulative histogram C: there are C(f)% pixels equal or darker than f In an image in [0 1] with a flat histogram, what is the greyscale value g so that C(f)% pixels are equal or darker than f? C(f) of course! Therefore, histogram equalization: g(x)=c(f(x)) 16

17 Extension: histogram matching Transform image f to match histogram of f f result f 17

18 Extension: histogram matching Transform image f to match histogram of f g(x)=cf -1 (Cf(f(x))) cumulative histogram Cf of f to get the flat case inverse cumulative histogram Cf -1 of f to match that histogram equalization: case where f has flat histogram and Cf -1 is identity 18

19 Questions?

20 Filtering So far we have looked at range-only and domain-only transformation But other transforms need to change the range according to the spatial neighborhood Linear shift-invariant filtering in particular

21 Linear shift-invariant filtering Replace each pixel by a linear combination of its neighbors. only depends on relative position of neighbors The prescription for the linear combination is called the convolution kernel Local image data kernel 7 Modified image data (shown at one pixel)

22 Example of linear NON-shift invariant transformation? e.g. neutral-density graduated filter (darken high y, preserve small y) J(x,y)=I(x,y)*(1-y/ymax) Formally, what does linear mean? For two scalars a & b and two inputs x & y: F(ax+by)=aF(x)+bF(y) What does shift invariant mean? For a translation T: F(T(x))=T(F(x)) If I blur a translated image, I get a translated 22 blurred image

23 More formally: Convolution I

24 Convolution (warm-up slide) coefficient 1.0? original 0 Pixel offset

25 Convolution (warm-up slide) coefficient 1.0 original 0 Pixel offset Filtered (no change)

26 Convolution coefficient 1.0? original 0 Pixel offset

27 shift coefficient 1.0 original 0 Pixel offset shifted

28 Convolution coefficient 0.3? original 0 Pixel offset

29 Blurring coefficient 0.3 original 0 Pixel offset Blurred (filter applied in both dimensions).

30 Blur examples impulse 8 coefficient original 0 Pixel offset filtered

31 Blur examples impulse 8 coefficient original 0 Pixel offset filtered edge 8 4 coefficient original 0 Pixel offset filtered

32 Questions?

33 Convolution (warm-up slide) ? 0 0 original

34 Convolution (no change) original Filtered (no change)

35 Convolution ? 0 0 original

36 (remember blurring) coefficient 0.3 original 0 Pixel offset Blurred (filter applied in both dimensions).

37 Sharpening original Sharpened original

38 Sharpening example 8 coefficient original Sharpened (differences are accentuated; constant areas are left untouched).

39 Sharpening before after

40 Oriented filters Gabor filters at different scales and spatial frequencies top row shows anti-symmetric (or odd) filters, bottom row the symmetric (or even) filters.

Filtered images Reprinted from Shiftable MultiScale Transforms, by

42 Questions?

43 Studying convolutions Convolution is complicated But at least it s linear (f+kg) h = f h +k (g h) We want to find a better expression Let s study functions whose behavior is simple under convolution

44 Blurring: convolution Input Convolution sign Kernel Same shape, just reduced contrast!!! This is an eigenvector (output is the input multiplied by a constant)

45 Big Motivation for Fourier analysis Sine waves are eigenvectors of the convolution operator

46 Other motivation for Fourier analysis: sampling The sampling grid is a periodic structure Fourier is pretty good at handling that We saw that a sine wave has serious problems with sampling Sampling is a linear process but not shift-invariant

47 Sampling Density If we re lucky, sampling density is enough Input Reconstructed

48 Sampling Density If we insufficiently sample the signal, it may be mistaken for something simpler during reconstruction (that's aliasing!)

49 Recap: motivation for sine waves Blurring sine waves is simple You get the same sine wave, just scaled down The sine functions are the eigenvectors of the convolution operator Sampling sine waves is interesting Get another sine wave Not necessarily the same one! (aliasing) If we represent functions (or images) with a sum of sine waves, convolution and sampling are easy to study

50 Questions?

51 Fourier as change of basis Shuffle the data to reveal other information E.g., take average & difference: matrix 3 Basis function 1 Basis function 1 Basis function Basis function 2 0 Signal Geometric interpretation After rotation Pseudo- Fourier

52 Fourier as change of basis Same thing with infinite-dimensional vectors Basis function 1 Basis function 1 Basis function 2 Basis function 2 Signal Geometric interpretation After rotation Pseudo- Fourier

53 Question? 53

54 Fourier as a change of basis Discrete Fourier Transform: just a big matrix But a smart matrix!

55 To get some sense of what basis elements look like, we plot a basis element --- or rather, its real part --- as a function of x,y for some fixed u, v. We get a function that is constant when (ux+vy) is constant. The magnitude of the vector (u, v) gives a frequency, and its direction gives an orientation. The function is a sinusoid with this frequency along the direction, and constant perpendicular to the direction. v u

56 Here u and v are larger than in the previous slide. v u

57 And larger still... v u

58 Question? 58

59 Other presentations of Fourier Start with Fourier series with periodic signal Heat equation more or less special case of convolution iterate -> exponential on eignevalues 59

60 Motivations Insights & mathematical beauty Sampling rate and filtering bandwidth Computation bases FFT: faster convolution E.g. finite elements, fast filtering, heat equation, vibration modes Optics: wave nature of light & diffraction

61 Questions?

62 The Fourier Transform Defined for infinite, aperiodic signals Derived from the Fourier series by extending the period of the signal to infinity The Fourier transform is defined as X(ω) is called the spectrum of x(t) It contains the magnitude and phase of each complex exponential of frequency ω in x(t)

63 The Fourier Transform The inverse Fourier transform is defined as Fourier transform pair x(t) is called the spatial domain representation X(ω) is called the frequency domain representation

64 Beware of differences Different definitions of Fourier transform We use Other people might exclude normalization or include 2π in the frequency X might take ω or jω as argument Physicist use j, mathematicians use i

65 Phase Don t forget the phase! Fourier transform results in complex numbers Can be seen as sum of sines and cosines Or modulus/phase

66 Phase is important!

67 Phase is important!

68 Questions?

69 Duality Up to details (such as factors of 2π or signs): if function a is the Fourier transform of b, then b is the Fourier transform of a For example, the Fourier transform of a box is a sinc, and the Fourier transform of a sinc is a box.

70 Duality Any theorem that involves the primal and Fourier domains is also true when swapping the two domains. e.g. shift theorem: Primal f(x+a) Fourier e -2πiaω F(ω) e -2πiax f(x) F(ω+a)

71 Duality Any theorem that involves the primal and Fourier domains is also true when swapping the two domains. e.g. scaling theorem: Primal f(ax) Fourier 1/a F(x/a) 1/a f(x/a) F(ωa)

72 Convolution/Modulation A convolution in one domain is a multiplication in the other one Primal f g Fourier FG fg F G Recall that Fourier bases are eigenvectors of the convolution

73 Questions?

74 Low pass black means 1, white means 0

75 High pass

76 Filtering in Fourier domain

77 Analysis of our simple filters coefficient 1.0 original 0 Pixel offset Filtered (no change) spectrum: F(ω)=1 (yes, I am now using the definition without 1/sqrt(2pi) 1.0 constant 0

78 Analysis of our simple filters coefficient 1.0 original 0 Pixel offset shifted spectrum: F (ω) =e 2πjωδ 0 Constant magnitude, linearly shifted phase

79 Analysis of our simple filters coefficient 0.3 original 0 Pixel offset blurred spectrum: F(ω)=sinc(ω) =sin(ω)/ω Low-pass filter

80 Analysis of our simple filters original 0 0 sharpened high-pass filter 2.3 spectrum: F(ω)=2-sinc(ω) 1.0 0

81 Convolution versus FFT 1-d FFT: O(NlogN) computation time, where N is number of samples. 2-d FFT: 2N(NlogN), where N is number of pixels on a side Convolution: K N 2, where K is number of samples in kernel Say N=2 10, K= d FFT: , while convolution gives

82 Words of wisdom Careful with the FFT: it assumes a cyclic signal Oftentimes, the answer you get mostly shows wraparound artifacts Proper windowing might be needed to analyze the frequency content of an image e.g. multiply function by a smooth function that falls off away from the center so that the boundary is zero 82

83 Questions?

84 Sampling and aliasing

85 In photos too MIT EECS SMA 5507, Durand and Popović

86 More on Samples In signal processing, the process of mapping a continuous function to a discrete one is called sampling The process of mapping a continuous variable to a discrete one is called quantization To represent or render an image using a computer, we must both sample and quantize Now we focus on the effects of sampling and how to fight them discrete value discrete position

87 Sampling in the Frequency Domain original signal Fourier Transform sampling grid Fourier Transform (multiplication) (convolution) sampled signal Fourier Transform

88 Reconstruction If we can extract a copy of the original signal from the frequency domain of the sampled signal, we can reconstruct the original signal! But there may be overlap between the copies.

89 Guaranteeing Proper Reconstruction Separate by removing high frequencies from the original signal (low pass pre-filtering) Separate by increasing the sampling density If we can't separate the copies, we will have overlapping frequency spectrum during reconstruction aliasing.

90 Sampling Theorem When sampling a signal at discrete intervals, the sampling frequency must be greater than twice the highest frequency of the input signal in order to be able to reconstruct the original perfectly from the sampled version (Shannon, Nyquist, Whittaker, Kotelnikov, Küpfmüller)

91 91

92 Final project brainstorming Fredo Durand MIT EECS 6.815/6.865

93 Final project Groups of 1 or 2 Proposal due soon (with last pset) Deliverables: report + small presentation

94 Your ideas?

95 Some ideas Use CHDK to provide new features to Canon compact cameras Use flickr API to do something creative Explore different types of gradient reconstructions Improve time lapse Handle small parallax in panoramas Exploit flash/no-flash pairs Editing with images+depth (e.g. from stereo) Smart color to greyscale Face-aware image processing Sharpening out-of-focus images using other pictures from the sequences Application of morphing/warping Motion without movements and automatic illusions

ECG782: Multidimensional Digital Signal Processing

Professor Brendan Morris, SEB 3216, brendan.morris@unlv.edu ECG782: Multidimensional Digital Signal Processing Filtering in the Frequency Domain http://www.ee.unlv.edu/~b1morris/ecg782/ 2 Outline Background