Why does a lower resolution image still make sense to us? What do we lose? Image:

Transcription

1 2D FREQUENCY DOMAIN The slides are from several sources through James Hays (Brown); Srinivasa Narasimhan (CMU); Silvio Savarese (U. of Michigan); Bill Freeman and Antonio Torralba (MIT), including their own slides.

2 Why does a lower resolution image still make sense to us? What do we lose? Image: Slide: Hoiem

3 Jean Baptiste Joseph Fourier ( ) Had crazy idea (1807): Any periodic function can be rewritten as a weighted sum of Sines and Cosines of different frequencies. Don t believe it? Neither did Lagrange, Laplace, Poisson and other big wigs Not translated into English until 1878! But it s true! called Fourier Series Possibly the greatest tool used in Engineering

4 Example: sum of sines Our building block: Asin( x Add enough of them to get any signal f(x) you want! f_0 is the average of the signal = 1 the frequencies are f_1,3,5,... because the even f-s are zero

5 To get some sense of what basis elements look like, we plot a basis element --- or rather, its real part --- as a function of x,y for some fixed u, v. We get a function that is constant when (ux+vy) is constant. The magnitude of the vector (u, v) gives a frequency, and its direction gives an orientation. The function is a sinusoid with this frequency along the direction, and constant perpendicular to the direction. v u x,y plane with u,v given. Orientation tan a = v/u Frequency of the sinusoid sqrt(u^2 + v^2). The real component only here as intensity image.

6 Here u and v are larger than in the previous slide. v u

7 And larger still... v u

8 Fourier analysis in images Intensity Image Fourier Image The origin of the Fourier image is in the middle.

9 Signals can be composed + = / moire effect slighty different angles

10 Some Fourier Transforms... Image Magnitude FT

11 Magnitude FT Image

12 Fourier Transform Fourier transform stores the magnitude and phase at each frequency Magnitude encodes how much signal there is at a particular frequency Phase encodes spatial information (indirectly) For mathematical convenience, this is often notated in terms of real and complex numbers Amplitude: A R I 2 2 ( ) ( ) Phase: tan 1 I( ) R( ) F(omega) = R(omega) + i I(omega)

13 See Szeliski Book (3.4) Properties of Fourier Transforms Linearity Fourier transform of a real signal is symmetric about the origin. The energy of the signal is the same as the energy of its Fourier transform.

14 Fourier Amplitude Spectrum A B C fx(cycles/image pixel size) fx(cycles/image pixel size) fx(cycles/image pixel size)

15 Phase and Magnitude Image with cheetah phase (and zebra magnitude) Image with zebra phase (and cheetah magnitude) Computer Vision - A Modern Approach - Set: Pyramids and Texture - Slides by D.A. Forsyth

16 Phase and Magnitude Curious fact all natural images have about the same magnitude transform hence, phase seems to matter, but magnitude largely doesn t Demonstration Take two pictures, swap the phase transforms, compute the inverse - what does the result look like?

17 Randomizing the phase Fourier transform, randomize the phase, inverse transform in the spatial domain!!

18 Convolution Theorem The Fourier transform of the convolution of two functions is the product of their Fourier transforms F[ g h] F[ g]f[ h] The inverse Fourier transform of the product of two Fourier transforms is the convolution of the two inverse Fourier transforms F 1 [ gh] F 1 [ g] [ h] Convolution in spatial domain is equivalent to multiplication in frequency domain! F 1 here g=g h=h

19 Discrete Fourier transform Forward transform F[m,n] M 1N 1 k 0 l 0 f [k,l]e i km M ln N Inverse transform f [k,l] 1 MN M 1N 1 k 0 l 0 F[m,n]e i km M ln N

20 FFT (Fast FT) in Matlab Filtering with fft im = double(imread( '))/255; im = rgb2gray(im); % im should be a gray-scale floating point image [imh, imw] = size(im); hs = 50; % filter half-size fil = fspecial('gaussian', hs*2+1, 10); fftsize = 1024; % should be order of 2 (for speed) and include padding im_fft = fft2(im, fftsize, fftsize); % 1) fft im with padding fil_fft = fft2(fil, fftsize, fftsize); % 2) fft fil, pad to same size as image im_fil_fft = im_fft.* fil_fft; % 3) multiply fft images im_fil = ifft2(im_fil_fft); % 4) inverse fft2 im_fil = im_fil(1+hs:size(im,1)+hs, 1+hs:size(im, 2)+hs); % 5) remove padding Displaying with fft figure(1), imagesc(log(abs(fftshift(im_fft)))), axis image, colormap jet Slide: Hoiem

21 Filtering in spatial domain along the vertical Sobel, high frequency filter * =

22 Filtering in frequency domain FFT FFT = Inverse FFT Slide: Hoiem

23 Filtering Why does the Gaussian give a nice smooth image, but the square filter give edgy artifacts? Gaussian Box filter

24 ...log fft for a image and the output filters! Gaussian filter...

25 ...against the Box filter

26 Image Processing in the Fourier Domain Magnitude of the FT Does not look anything like what we have seen

27 Convolution is Multiplication in Fourier Domain f(x,y) F(s x,s y ) * h(x,y) H(s x,s y ) g(x,y) G(s x,s y )

28 Low-pass Filtering Let the low frequencies pass and eliminating the high frequencies. Images with less details.

29 High-pass Filtering Lets through the high frequencies (the detail), but eliminates the low frequencies (keeps overall shape). It acts like an edge enhancer.

30 Sampling down an image. Why does a lower resolution image still make sense to us? What do we lose? The high frequency details. Now our prior knowledge becomes more important. Image:

31 Subsampling by a factor of 2. Throw away every other row and column to create a 1/2 size image

32 Subsampling without pre-filtering 1/2 1/4 1/8 original size...will result in aliasing... Slide by Steve Seitz

33 Subsampling with Gaussian pre-filtering Gaussian 1/2 original size G 1/4 G 1/8 Slide by Steve Seitz

34 What is a good representation for image analysis? Fourier transform domain tells you what (textural properties), but not where. Pixel domain representation tells you where (pixel location), but not what. Want an image representation that gives you a description of image events what is happening where. This is very difficult to achieve... even with todays state-of-the-art techniques.

35 Human Visual PercepKon Low spakal frequency Medium spakal frequency SpaKal frequency channels High spakal frequency 51

36 Human Visual PercepKon Blur image Low spakal frequency Sharp image Medium spakal frequency SpaKal frequency channels High spakal frequency 51

37 Contrast SensiKvity FuncKon Blackmore & Campbell (1969) Low SpaKal Frequency High c/i Invisible Contrast sensikvity visible Low SpaKal frequency (cycles/degree) High 53

38 Contrast SensiKvity FuncKon Blackmore & Campbell (1969) Low SpaKal Frequency High c/i Contrast sensikvity Invisible Low sensikvity visible Low sensikvity Low SpaKal frequency (cycles/degree) High53