Frequency2: Sampling and Aliasing

Transcription

1 CS 4495 Computer Vision Frequency2: Sampling and Aliasing Aaron Bobick School of Interactive Computing

2 Administrivia Project 1 is due tonight. Submit what you have at the deadline. Next problem set stereo will be out Thursday Sept. 11 and will be due Tuesday evening, Sept 23 rd, 11:55pm. It is easier Readings for this week: Still FP Chapter 4

3 Salvador Dali Gala Contemplating the Mediterranean Sea, which at 30 meters becomes the portrait of Abraham Lincoln, 1976

4

5 Fourier: A nice set of basis Teases away fast vs. slow changes in the image.

6 Fourier Transform We want to understand the frequency ω of our signal. So, let s reparametrize the signal by ω instead of x: f(x) Fourier Transform F ( ω) = R( ω) + ii( ω) F(ω) Inverse Fourier Transform Asin(ωx +φ) F(ω) For every ω from 0 to inf, (actually inf to inf), F(ω) holds the amplitude A and phase φ of the corresponding sine How can F hold both? Complex number trick! Even And we can go back: Odd A = ± R ω + I( φ = tan ( ) ω I( ω) R( ω) f(x) )

7 Frequency Spectra Even/Odd Frequency actually goes from inf to inf. Sinusoid example: Even (cos) Odd (sin) Magnitude ω ω ω Real Imaginary Power

8 2D Fourier Transforms The two dimensional version:. (, ) (, ) And the 2D Discrete FT: 2 π ( ux+ vy ) F u v = f x y e i dx dy 1 x= N 1 y= N 1 Fk (, k ) = f( x, ye ) x y N x= 0 y= 0 i kx+ ky 2π ( ) x N y Works best when you put the origin of k in the middle.

9 Linearity of Sum =

10 Extension to 2D Complex plane Both a Real and Im version

11 Examples B.K. Gunturk

12 Fourier Transform and Convolution Let g = f h Then ( ) g( x) G u i = e 2 πux dx ( τ ) h( x τ ) i2πux = f e dτdx [ ][ ( ) ( ) ] i2πuτ i2πu x τ f e dτ h x τ e ( τ ) = dx [ ( ) ] i2πuτ f τ e dτ h( x' ) [ ] i2πux' e dx = ' = F ( u) H ( u) Convolution in spatial domain Multiplication in frequency domain

13 Fourier Transform and Convolution Spatial Domain (x) Frequency Domain (u) g = f h G = FH g = fh G = F H So, we can find g(x) by Fourier transform g = f h IFT FT FT G = F H

14 Example use: Smoothing/Blurring We want a smoothed function of f(x) g ( x) = f ( x) h( x) Let us use a Gaussian kernel h x ( x) exp = 2 2πσ Convolution in space is multiplication in freq: ( u) F( u) H ( u) 2 σ Fat Gaussian in space is H ( u) G = skinny Gaussian in πσ frequency. Why? h( x) σ 1 2 x H(u) attenuates high frequencies in F(u) (Low-pass Filter)! u

15 2D convolution theorem example f(x,y) h(x,y) * F(s x,s y ) ( or F(u,v) ) H(s x,s y ) g(x,y) G(s x,s y )

16 Low and High Pass filtering Ringing

17 Fourier Transform smoothing pairs

18 Properties of Fourier Transform Spatial Domain (x) Frequency Domain (u) Linearity c1 f ( x) + c2g( x) F( u) c G( u) Scaling f ( ax) c1 + 2 Shifting f ( x x 0 ) e 2πux F( u) Symmetry F ( x) f ( u) Conjugation f ( x) F ( u) Convolution f ( x) g( x) F( u) G( u) n d f ( x) Differentiation n ( i2πu ) F( u) dx n Shrink Stretch Differentiate Multiply by u 1 a F u a

19 Fourier Pairs (from Szeliski)

20 Fourier Transform Sampling Pairs FT of an impulse train is an impulse train

21 Sampling and Aliasing

22 Sampling and Reconstruction

23 Sampled representations How to store and compute with continuous functions? Common scheme for representation: samples write down the function s values at many points S. Marschner

24 Reconstruction Making samples back into a continuous function for output (need realizable method) for analysis or processing (need mathematical method) amounts to guessing what the function did in between S. Marschner

25 1D Example: Audio low frequencies high

26 Sampling in digital audio Recording: sound to analog to samples to disc Playback: disc to samples to analog to sound again how can we be sure we are filling in the gaps correctly? S. Marschner

27 S. Marschner CS 4495 Computer Vision A. Bobick Sampling and Reconstruction Simple example: a sign wave

28 Undersampling What if we missed things between the samples? Simple example: undersampling a sine wave unsurprising result: information is lost S. Marschner

29 Undersampling What if we missed things between the samples? Simple example: undersampling a sine wave unsurprising result: information is lost surprising result: indistinguishable from lower frequency S. Marschner

30 Undersampling What if we missed things between the samples? Simple example: undersampling a sine wave unsurprising result: information is lost surprising result: indistinguishable from lower frequency also was always indistinguishable from higher frequencies aliasing: signals traveling in disguise as other frequencies S. Marschner

31 S. Seitz CS 4495 Computer Vision A. Bobick Aliasing in video

32 Aliasing in images

33 What s happening? Input signal: Plot as image: x = 0:.05:5; imagesc(sin((2.^x).*x)) Alias! Not enough samples

34 Antialiasing What can we do about aliasing? Sample more often Join the Mega-Pixel craze of the photo industry But this can t go on forever Make the signal less wiggly Get rid of some high frequencies Will loose information But it s better than aliasing

35 Preventing aliasing Introduce lowpass filters: remove high frequencies leaving only safe, low frequencies choose lowest frequency in reconstruction (disambiguate) S. Marschner

36 (Anti)Aliasing in the Frequency Domain

37 Impulse Train Define a comb function (impulse train) in 1D as follows comb [ ] [ ] M x = δ x km k = where M is an integer 1 comb [ x] 2 x B.K. Gunturk

38 Impulse Train in 1D comb ( x) 2 1 ( ) 2 comb u x u 2 Remember: 1 2 Scaling f ( ax) 1 a F u a B.K. Gunturk

39 Impulse Train in 2D (bed of nails) ( ) combm, N( x, y) δ x km, y ln k= l= Fourier Transform of an impulse train is also an impulse train: δ 1 k l x km, y ln δ u, v MN = = M N ( ) k= l= k l comb, ( x, y) M N comb 1 1, M N ( u, v) As the comb samples get further apart, the spectrum samples get closer together! B.K. Gunturk

40 Impulse Train comb [ n] 2 1 ( ) 2 comb u n u Remember: 1 2 Scaling f ( ax) 1 a F u a B.K. Gunturk

41 Sampling low frequency signal f( x) Fu ( ) x u comb ( ) M x comb ( ) 1 u M x u Multiply: M f ( x) comb ( x) M Convolve: 1 M F( u)* comb ( u) 1 M x u B.K. Gunturk

42 Sampling low frequency signal f( x) Fu ( ) x W W u f ( x) comb ( x) M F( u)* comb ( u) 1 M x W u M No problem if 1 M > 2W 1 M B.K. Gunturk

43 Sampling low frequency signal f ( x) comb ( x) M F( u)* comb ( u) 1 M x W u M 1 M 1 2M If there is no overlap, the original signal can be recovered from its samples by low-pass filtering. B.K. Gunturk

44 Sampling high frequency signal f( x) Fu ( ) x W W u f ( x) comb ( x) M Overlap: The high frequency energy is folded over into low frequency. It is aliasing as lower frequency energy. And you cannot fix it once it has happened. F( u)* comb ( u) 1 M 1 M W u

45 Sampling high frequency signal f( x) x W Fu ( ) Anti-aliasing filter u W f( x)* hx ( ) W W 1 2M u [ ] f ( x)* h( x) comb ( x) M u 1 M B.K. Gunturk

46 Sampling high frequency signal Without anti-aliasing filter: f ( x) comb ( x) M W u With anti-aliasing filter: 1 M [ ] f ( x)* h( x) comb ( x) M u 1 M B.K. Gunturk

47 Aliasing in Images

48 S. Seitz CS 4495 Computer Vision A. Bobick Image half-sizing This image is too big to fit on the screen. How can we reduce it? How to generate a halfsized version?

49 Image sub-sampling 1/8 1/4 Throw away every other row and column to create a 1/2 size image - called image sub-sampling S. Seitz

50 Image sub-sampling 1/2 1/4 (2x zoom) 1/8 (4x zoom) Aliasing! What do we do? S. Seitz

51 Gaussian (lowpass) pre-filtering G 1/4 G 1/8 Gaussian 1/2 Solution: filter the image, then subsample Filter size should double for each ½ size reduction. Why? S. Seitz

52 Subsampling with Gaussian pre-filtering Gaussian 1/2 G 1/4 G 1/8 S. Seitz

53 Compare with... 1/2 1/4 (2x zoom) 1/8 (4x zoom) S. Seitz

54 Campbell-Robson contrast sensitivity curve The higher the frequency the less sensitive human visual system is

55 Lossy Image Compression (JPEG) Block-based Discrete Cosine Transform (DCT) on 8x8

56 Using DCT in JPEG The first coefficient B(0,0) is the DC component, the average intensity The top-left coeffs represent low frequencies, the bottom right high frequencies

57 Image compression using DCT DCT enables image compression by concentrating most image information in the low frequencies Lose unimportant image info (high frequencies) by cutting B(u,v) at bottom right The decoder computes the inverse DCT IDCT Quantization Table

58 JPEG compression comparison 89k 12k

59 Maybe the end?

60 Or not!!! A teaser on pyramids

61 Image Pyramids Known as a Gaussian Pyramid [Burt and Adelson, 1983] In computer graphics, a mip map [Williams, 1983] A precursor to wavelet transform S. Seitz

62 Band-pass filtering Gaussian Pyramid (low-pass images) Laplacian Pyramid (subband images) Created from Gaussian pyramid by subtraction These are bandpass images (almost).

63 Why almost bandpass? Why are the subtracted sequential Gaussians ( difference of Gaussians ) almost bandpass?

64 Laplacian Pyramid Need this! Original image How can we reconstruct (collapse) this pyramid into the original image?

65 Computing the Laplacian Pyramid Reduce Expand Need G k to reconstruct Don t worry about these details YET! (PS4?)

66 What can you do with band limited imaged?

67 Apples and Oranges in bandpass Fine L 0 L 2 Coarse L 4 Reconstructed

68 What can you do with band limited imaged?

69 Really the end

70 Fourier Transform: Properties Linearity af ( x, y) + bg( x, y) af( u, v) + bg( u, v) Shifting Modulation Convolution f x x y x e Fuv j 2 π ( ux0 vy0 (, ) + ) (, ) 0 0 (, ) (, ) j2 π ( ux 0 + vy 0 ) e f xy Fu u v v f( xy, )* gxy (, ) FuvGuv (, ) (, ) 0 0 Multiplication f( xygxy, ) (, ) Fuv (, )* Guv (, ) Separable functions f( xy, ) = f( x) f( y) Fuv (, ) = FuFv ( ) ( ) Bahadir K. 71

71 Fourier Transform: Properties Separability j 2 π ( ux+ vy) F u v f x y e dxdy = (, ) (, ) = = j 2πux j 2πvy f ( x, y) e dx e dy F( u, y) e j 2π vy dy 2D Fourier Transform can be implemented as a sequence of 1D Fourier Transform operations. Bahadir K. 72

72 Fourier Transform: Properties Energy conservation 2 2 f ( x, y) dxdy = F( u, v) dudv Bahadir K. 73

73 Fourier Transform: 2D Discrete Signals Fourier Transform of a 2D discrete signal is defined as Fuv (, ) f[ mne, ] = m= n= j 2 π ( um+ vn) where 1 1 uv, < 2 2 Inverse Fourier Transform 1/2 1/2 j 2 π ( um+ vn) f [ m, n] F( u, v) e dudv = 1/2 1/2 Bahadir K. 74

74 Fourier Transform: Properties Periodicity: Fourier Transform of a discrete signal is periodic with period 1. Fu ( + kv, + l) = f[ mne, ] Arbitrary integers m= n= = m= n= = m= n= = Fuv (, ) ( ) j2 π ( u+ k) m+ ( v+ l) n ( ) j 2π um+ vn j 2πkm j 2πln f[ mne, ] e e f[ mne, ] j 2 π ( um+ vn) 1 1 Bahadir K. 75

75 Fourier Transform: Properties Linearity, shifting, modulation, convolution, multiplication, separability, energy conservation properties also exist for the 2D Fourier Transform of discrete signals. Bahadir K. 76

76 Fourier Transform: Properties Linearity Shifting Modulation af [ m, n] + bg[ m, n] af( u, v) + bg( u, v) f m m n n e Fuv 2 ( 0 0 [, ] j π um + vn ) (, ) 0 0 j2 π ( um 0 + vn 0 ) e f mn Fu u v v [, ] (, ) 0 0 Convolution Multiplication f[ mn, ]* gmn [, ] FuvGuv (, ) (, ) f[ mngmn, ] [, ] Fuv (, )* Guv (, ) Separable functions Energy conservation f[ m, n] = f[ m] f[ n] F( u, v) = F( u) F( v) 2 2 f [ m, n] = F( u, v) dudv m= n= Bahadir K. 77

77 Fourier Transform: Properties Define Kronecker delta function δ[ mn, ] 1, for m= 0 and n= 0 = 0, otherwise Fourier Transform of the Kronecker delta function ( ) π( ) j2π um+ vn j2 u0+ v0 Fuv (, ) = δ[ mne, ] = e = 1 m= n= Bahadir K. 78

78 Fourier Transform: Properties Fourier Transform of 1 j 2π + ( um vn) f( mn, ) = 1 Fuv (, ) = 1 e = δ( u kv, l) m= n= k= l= To prove: Take the inverse Fourier Transform of the Dirac delta function and use the fact that the Fourier Transform has to be periodic with period 1. Bahadir K. 79

79 Sampling Theorem Continuous signal: f ( x) Shah function (Impulse train): s( x) x ( x) = δ ( x nx ) s 0 n= Sampled function: x 0 ( x) = f ( x) s( x) = f ( x) δ ( x nx ) f s 0 n= x

80 Sampling Theorem Sampled function: F S ( x) = f ( x) s( x) = f ( x) δ ( x nx ) f s 0 n= ( u)= F( u) S( u)= F u F A ( u) ( ) 1 x δ u n x 0 n= 0 F S ( u) A x 0 Sampling frequency 1 x 0 u max u Only if u max 1 2x 0 u max 1 x u 0

81 Nyquist Theorem If u > max 1 2x 0 F S A x 0 ( u) Aliasing u max u 1 x F ( ) ( ) When can we recover u from u? Only if We can use 0 0 F S 1 umax (Nyquist Frequency) 2x C ( u) x < 1 0 u = 2x0 0 otherwise Then F( u) = F ( u) C( u) and f ( x) = IFT[ F( u) ] S Sampling frequency must be greater than 2umax