CSCI 1290: Comp Photo

Size: px

Start display at page:

Download "CSCI 1290: Comp Photo"

Ronald McCoy
5 years ago
Views:

1 CSCI 1290: Comp Photo Fall Brown University James Tompkin Many slides thanks to James Hays old CS 129 course, along with all of its acknowledgements.

4 Capture Frequency - Rolling `Shutter James Hays

5 Image Pyramids Project 1 function: vis_hybrid_image.m Wikipedia Image Pyramids

6 Why does a lower resolution image still make sense to us? What information do we lose? Image: Slide: Hoiem

7 Hays Hybrid Images A. Oliva, A. Torralba, P.G. Schyns, Hybrid Images, SIGGRAPH 2006

8 Why do we get different, distance-dependent interpretations of hybrid images?? Slide: Hoiem

9 I understand frequency as in waves but how does this relate to the complex signals we see in natural images? to image frequency?

10 Fourier Series & Fourier Transforms Another way of thinking about frequency

11 Fourier series A bold idea (1807): Any univariate function can be rewritten as a weighted sum of sines and cosines of different frequencies. Jean Baptiste Joseph Fourier ( ) Hays

12 Fourier series A bold idea (1807): Any univariate function can be rewritten as a weighted sum of sines and cosines of different frequencies. Our building block: Asin( t) + B cos( t) Add enough of them to get any signal g(t) you want! Hays

13 Time and Frequency example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t)

14 Time and Frequency example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t) = +

15 Frequency Spectra example : g(t) = sin(2pf t) + (1/3)sin(2p(3f) t) = +

16 Square wave spectra

17 Square wave spectra = + =

18 Square wave spectra = + =

19 Square wave spectra = + =

20 Square wave spectra = + =

21 Square wave spectra = + =

22 Coefficient Square wave spectra = 1 A f sin(2πft) f=1

23 Frequency Spectra

Jean Baptiste Joseph Fourier (1768-1830) A bold idea (1807): Any univariate function can be

Neither did Lagrange, Laplace, Poisson and other big wigs Not translated into English until 1878!

..the manner in which the author arrives at these equations is not exempt of difficulties and.

24 Jean Baptiste Joseph Fourier ( ) A bold idea (1807): Any univariate function can be rewritten as a weighted sum of sines and cosines of different frequencies. Don t believe it? Neither did Lagrange, Laplace, Poisson and other big wigs Not translated into English until 1878! But it s (mostly) true! Called Fourier Series There are some subtle restrictions...the manner in which the author arrives at these equations is not exempt of difficulties and...his analysis to integrate them still leaves something to be desired on the score of generality and even rigour. Laplace Lagrange Legendre Hays

25 Wikipedia Fourier transform

26 Sine/cosine and circle sin t cos t Wikipedia Unit Circle

27 Square wave (approx.) Mehmet E. Yavuz

28 Sawtooth wave (approx.) Mehmet E. Yavuz

29 One series in each of x and y Generative Art, じゃがりきん, Video, 2018

30 Example: Music We think of music in terms of frequencies at different magnitudes Slide: Hoiem

31 Coefficient Spatial domain Frequency Frequency domain

32 Fourier analysis in images Spatial domain images Fourier decomposition frequency amplitude images More:

33 Signals can be composed Spatial domain images + = Fourier decomposition frequency amplitude images More:

34 Natural image Natural image Fourier decomposition Frequency coefficients (amplitude) What does it mean to be at pixel x,y? What does it mean to be more or less bright in the Fourier decomposition image?

35 Can change spectrum, then reconstruct

36 Low and High Pass filtering

37 Brian Pauw demo Live Fourier decomposition images Using FFT2 function I hacked it a bit for MATLAB

38 Think-Pair-Share Match the spatial domain image to the Fourier magnitude image B A C D E Hoiem

39 Fourier Bases Teases away fast vs. slow changes in the image. Blue = sine Green = cosine This change of basis is the Fourier Transform Hays

40 Basis reconstruction Danny Alexander

41 Fourier Transform Stores the amplitude and phase at each frequency: For mathematical convenience, this is often notated in terms of real and complex numbers Related by Euler s formula Hays

42 Fourier Transform Stores the amplitude and phase at each frequency: For mathematical convenience, this is often notated in terms of real and complex numbers Related by Euler s formula Hays

Fourier Transform Stores the amplitude and phase at each frequency: For mathematical convenience, this is often notated in terms of real and complex numbers Related by Euler

43 Fourier Transform Stores the amplitude and phase at each frequency: For mathematical convenience, this is often notated in terms of real and complex numbers Related by Euler s formula Amplitude encodes how much signal there is at a particular frequency: A = Re( 2 ) + Im( ) 2 Phase encodes spatial information (indirectly): = tan 1 Im( ) Re( ) Hays

44 Amplitude / Phase Amplitude tells you how much Phase tells you where Translate the image? Amplitude unchanged Adds a constant to the phase. Morse

45 What about phase? Efros

46 What about phase? Amplitude Phase Efros

47 What about phase? Efros

48 What about phase? Amplitude Phase Efros

49 John Brayer, Uni. New Mexico We generally do not display PHASE images because most people who see them shortly thereafter succumb to hallucinogenics or end up in a Tibetan monastery.

50 Think-Pair-Share In Fourier space, where is more of the information that we see in the visual world? Amplitude Phase

51 Cheebra Zebra phase, cheetah amplitude Cheetah phase, zebra amplitude Efros

52 The frequency amplitude of natural images are quite similar Heavy in low frequencies, falling off in high frequencies Will any image be like that, or is it a property of the world we live in? Most information in the image is carried in the phase, not the amplitude Not quite clear why Efros

53 What is the relationship to phase in audio? In audio perception, frequency is important but phase is not. In visual perception, both are important.??? : (

54 Hays Salvador Dali Gala Contemplating the Mediterranean Sea, which at 30 meters becomes the portrait of Abraham Lincoln, 1976

55 Salvador Dali invented Hybrid Images? Salvador Dali Gala Contemplating the Mediterranean Sea, which at 30 meters becomes the portrait of Abraham Lincoln, 1976

57 Properties of Fourier Transforms Linearity F[ ax ( t) + by( t)] = a F[ x( t)] + b F[ y( t)] Fourier transform of a real signal is symmetric about the origin The energy of the signal is the same as the energy of its Fourier transform See Szeliski Book (3.4)

58 The Convolution Theorem The Fourier transform of the convolution of two functions is the product of their Fourier transforms F[ g h] = F[ g]f[ h] Convolution in spatial domain is equivalent to multiplication in frequency domain! g * h = F 1 [F[ g]f[ h]] Hays

59 Filtering in spatial domain * = Hays

60 Filtering in frequency domain FFT FFT = Inverse FFT Slide: Hoiem

61 Now we can edit frequencies!

62 Low and High Pass filtering

63 Removing frequency bands Brayer

64 High pass filtering + orientation

65 Low-pass, Band-pass, High-pass filters low-pass: High-pass / band-pass:

66 Application: Hybrid Images When we see an image from far away, we are effectively subsampling it! A. Oliva, A. Torralba, P.G. Schyns, SIGGRAPH 2006

67 Hybrid Image in FFT Hybrid Image Low-passed Image High-passed Image

68 Why does the Gaussian filter give a nice smooth image, but the square filter give edgy artifacts? Hays Gaussian Box filter

69 coefficient Why do we have those lines in the image? Sharp edges in the image need _all_ frequencies to represent them. = A k = 1 1 sin(2 kt ) k Efros

70 Box filter / sinc filter duality Hays What is the spatial representation of the hard cutoff (box) in the frequency domain?

71 Evan Wallace demo Made for CS123 1D example Forbes 30 under 30 Figma (collaborative design tools) with Dylan Field

72 Box filter / sinc filter duality Hays What is the spatial representation of the hard cutoff (box) in the frequency domain? Box filter Sinc filter sinc(x) = sin(x) / x Spatial Domain Frequency Domain Frequency Domain Spatial Domain

73 Box filter (spatial) Frequency domain magnitude

74 Gaussian filter duality Fourier transform of one Gaussian is another Gaussian (with inverse variance). Why is this useful? Smooth degradation in frequency components No sharp cut-off No negative values Never zero (infinite extent) Box filter (spatial) Frequency domain magnitude Gaussian filter (spatial) Frequency domain magnitude

75 Is convolution invertible? If convolution is just multiplication in the Fourier domain, isn t deconvolution just division? Sometimes, it clearly is invertible (e.g. a convolution with an identity filter) In one case, it clearly isn t invertible (e.g. convolution with an all zero filter) What about for common filters like a Gaussian?

76 Convolution * = FFT FFT ifft.x = Hays

77 Deconvolution? ifft FFT FFT =./ Hays

78 But under more realistic conditions Random noise, magnitude ifft FFT FFT =./ Hays

79 But under more realistic conditions Random noise,.0001 magnitude ifft FFT FFT =./ Hays

80 But under more realistic conditions Random noise,.001 magnitude ifft FFT FFT =./ Hays

81 But you can t invert multiplication by 0! A Gaussian is only zero at infinity Hays

82 Deconvolution is hard. Active research area. Even if you know the filter (non-blind deconvolution), it is still hard and requires strong regularization to counteract noise. If you don t know the filter (blind deconvolution), then it is harder still.

83 How would math have changed if the onesie had been invented?!?! : ( Hays

84 A few questions If we have infinite frequencies, why does the image end? Sampling theory. Frequencies higher than Nyquist frequency end up falling on an existing sample. i.e., they are aliases for existing samples! Nyquist frequency is half the sampling frequency. (Nyquist frequency is not Nyquist-Shannon rate, which is sampling required to reconstruct aliasfree signal. Both are derived from same theory.)

85 A few questions Why is frequency decomposition centered in middle, and duplicated and rotated? From Euler: cos(x) + i sin(x) = e ix cos ωt = 1 2 e iωt + e +iωt Coefficients for negative frequencies (i.e., backwards traveling waves) FFT of a real signal is conjugate symmetric i.e., f(-x) = f*(x)

86 A few questions How is the Fourier decomposition computed? Intuitively, by correlating the signal with a set of waves of increasing signal! Notes in hidden slides. Plus:

$kernels Fourier Optics Fraunhofer diffraction is Fourier$ transform of slit in the far field.

87 Applications of Fourier analysis Fast filtering with large kernels Fourier Optics Fraunhofer diffraction is Fourier transform of slit in the far field. Light spectrometry for astronomy Circular aperture = Airy disc diffraction pattern Wikipedia for graphics

88 Thinking in Frequency - Compression How is it that a 4MP image can be compressed to a few hundred KB without a noticeable change?

89 Lossy Image Compression (JPEG) 8x8 blocks 8x8 blocks The first coefficient B(0,0) is the DC component, the average intensity The top-left coeffs represent low frequencies, the bottom right represent high frequencies Block-based Discrete Cosine Transform (DCT) Slides: Efros

90 Image compression using DCT Compute DCT filter responses in each 8x8 block Filter responses Quantize to integer (div. by magic number; round) More coarsely for high frequencies (which also tend to have smaller values) Many quantized high frequency values will be zero Quantization divisers (element-wise) Quantized values

91 JPEG Encoding Entropy coding (Huffman-variant) Quantized values Linearize B like this. Helps compression: - We throw away the high frequencies ( 0 ). - The zig zag pattern increases in frequency space, so long runs of zeros.

92 JPEG compression comparison 89k 12k

93 James Hays Color spaces: YCbCr Fast to compute, good for compression, used by TV Y=0 Y=0.5 Y (Cb=0.5,Cr=0.5) Cr Cb Y=1 Cb (Y=0.5,Cr=0.5) Cr (Y=0.5,Cb=05)

94 Most JPEG images & videos subsample chroma

95 JPEG Compression Summary 1. Convert image to YCrCb 2. Subsample color by factor of 2 People have bad resolution for color 3. Split into blocks (8x8, typically), subtract For each block a. Compute DCT coefficients b. Coarsely quantize Many high frequency components will become zero c. Encode (with run length encoding and then Huffman coding for leftovers)

96 Things to Remember Sometimes it makes sense to think of images and filtering in the frequency domain Fourier analysis Can be faster to filter using FFT for large images (N logn vs. N 2 for autocorrelation) Images are mostly smooth Basis for compression Remember to low-pass before sampling

Intelligent Visual Prosthesis

Orientation sensor (IMU) Intelligent Visual Prosthesis Depth image-based obstacle detection Depth camera Wideangle RGB camera Simultaneous object recognition, localization, and hand tracking New projects: