Computer Vision Lecture 3

Transcription

1 Demo Haribo Classification Computer Vision Lecture 3 Linear Filters Bastian Leibe RWTH Aachen leibe@vision.rwth-aachen.de Code available on the class website... 3 You Can Do It At Home Course Outline Accessin a webcam in Matlab: Imae Processin Basics function out = webcam % uses "Imae Acquisition Toolbox adaptorname = 'winvideo'; vidformat = 'I42_32x24'; vidobj= videoinput(adaptorname,, vidformat); set(vidobj, 'ReturnedColorSpace', 'rb'); set(vidobj, 'FramesPerTrier', ); out = vidobj ; Imae Formation Binary Imae Processin Linear Filters Ede & Structure Extraction Color Sementation Local Features & Matchin Object Reconition and Cateorization cam = webcam(); im=etsnapshot(cam); 3D Reconstruction Motion and Trackin 4 5 Motivation Topics of This Lecture Noise reduction/imae restoration Linear filters What are they? How are they applied? x Application: smoothin Gaussian filter I What does it mean to filter an imae? Nonlinear Filters Structure extraction Median filter Multi-Scale representations How to properly rescale an imae? Filters as templates Correlation as template matchin 6 7

2 Common Types of Noise Gaussian Noise Salt & pepper noise Random occurrences of black and white pixels Impulse noise Random occurrences of white pixels Gaussian noise Variations in intensity drawn from a Gaussian ( Normal ) distribution. Basic Assumption Noise is i.i.d. (independent & identically distributed) 8 Source: Steve Seitz >> noise = randn(size(im)).*sima; >> output = im + noise; 9 Imae Source: Martial Hebert First Attempt at a Solution Movin Averae in 2D Assumptions: Expect pixels to be like their neihbors Expect noise processes to be independent from pixel to pixel ( i.i.d. = independent, identically distributed ) Let s try to replace each pixel with an averae of all the values in its neihborhood Source: S. Seitz Movin Averae in 2D Movin Averae in 2D Source: S. Seitz 3 Source: S. Seitz 2

3 Movin Averae in 2D Movin Averae in 2D Source: S. Seitz 5 Source: S. Seitz Movin Averae in 2D Correlation Filterin Say the averain window size is 2k+ 2k+: Attribute uniform weiht to each pixel Loop over all pixels in neihborhood around imae pixel F[i,j] Now eneralize to allow different weihts dependin on neihborin pixel s relative position: 6 Source: S. Seitz Non-uniform weihts 7 Correlation Filterin Convolution Convolution: Flip the filter in both dimensions (bottom to top, riht to left) Then apply cross-correlation This is called cross-correlation, denoted Filterin an imae Replace each pixel by a weihted combination of its neihbors. The filter kernel or mask is the prescription for the weihts in the linear combination. 2 H 3 4 (,) F (N,N) Notation for convolution operator H H (,) F (N,N) 8 9 3

4 Correlation vs. Convolution Shift Invariant Linear System Correlation Matlab: filter2 imfilter Shift invariant: Operator behaves the same everywhere, i.e. the value of the output depends on the pattern in the imae neihborhood, not the position of the neihborhood. Convolution Note the difference! Matlab: conv2 Linear: Superposition: h * (f + f 2 ) = (h * f ) + (h * f 2 ) Scalin: h * (kf) = k(h * f) Note If H[-u,-v] = H[u,v], then correlation convolution. 2 2 Properties of Convolution Averain Filter Linear & shift invariant Commutative: f = f Associative: (f ) h = f ( h) Often apply several filters in sequence: (((a b ) b 2 ) b 3 ) This is equivalent to applyin one filter: a (b b 2 b 3 ) Identity: f e = f for unit impulse e = [,,,,,, ]. What values belon in the kernel H[u,v] for the movin averae example? = ? box filter 2 3 (f? Smoothin by Averain Smoothin with a Gaussian depicts box filter: white = hih value, black = low value Oriinal Rinin artifacts! Filtered Oriinal Filtered 24 Imae Source: Forsyth & Ponce 25 Imae Source: Forsyth & Ponce 4

5 Smoothin with a Gaussian Comparison Gaussian Smoothin Gaussian kernel Oriinal Filtered Rotationally symmetric Weihts nearby pixels more than distant ones This makes sense as probabilistic inference about the sinal A Gaussian ives a ood model of a fuzzy blob 26 Imae Source: Forsyth & Ponce 27 Imae Source: Forsyth & Ponce Gaussian Smoothin Gaussian Smoothin What parameters matter here? Variance of Gaussian What parameters matter here? Size of kernel or mask Determines extent of smoothin Gaussian function has infinite support, but discrete filters use finite kernels σ = 2 with 33 kernel σ = 5 with 33 kernel σ = 5 with kernel σ = 5 with 33 kernel Rule of thumb: set filter half-width to about 3σ! Gaussian Smoothin in Matlab >> hsize = ; >> sima = 5; >> h = fspecial( aussian hsize, sima); Effect of Smoothin More noise >> mesh(h); >> imaesc(h); >> outim = imfilter(im, h); >> imshow(outim); Wider smoothin kernel outim 3 3 Imae Source: Forsyth & Ponce 5

6 Efficient Implementation Filterin: Boundary Issues Both, the BOX filter and the Gaussian filter are separable: First convolve each row with a D filter x I What is the size of the output? MATLAB: filter2(,f,shape) shape = full : output size is sum of sizes of f and shape = same : output size is same as f shape = valid : output size is difference of sizes of f and Then convolve each column with a D filter y I full same valid Remember: Convolution is linear associative and commutative x? y? I = x? ( y? I) = ( x? y )? I Slide credit: Bernt Schiele 32 f Slide credit: Svetlana Lazebnik f f 33 Filterin: Boundary Issues Filterin: Boundary Issues How should the filter behave near the imae boundary? The filter window falls off the ede of the imae Need to extrapolate Methods: Clip filter (black) Wrap around Copy ede Reflect across ede How should the filter behave near the imae boundary? The filter window falls off the ede of the imae Need to extrapolate Methods (MATLAB): Clip filter (black): imfilter(f,,) Wrap around: imfilter(f,, circular ) Copy ede: imfilter(f,, replicate ) Reflect across ede: imfilter(f,, symmetric ) 34 Source: S. Marschner 35 Source: S. Marschner Topics of This Lecture Why Does This Work? Linear filters What are they? How are they applied? Application: smoothin Gaussian filter What does it mean to filter an imae? A small excursion into the Fourier transform to talk about spatial frequencies Nonlinear Filters Median filter 3 cos(x) Multi-Scale representations How to properly rescale an imae? Filters as templates Correlation as template matchin + cos(3x) +.8 cos(5x) +.4 cos(7x) Source: Michal Irani 6

7 The Fourier Transform in Cartoons Fourier Transforms of Important Functions A small excursion into the Fourier transform to talk about spatial frequencies hih low Frequency spectrum hih Sine and cosine transform to?? 3 cos(x) + cos(3x) +.8 cos(5x) +.4 cos(7x) + Frequency coefficients 38 Source: Michal Irani 39 Imae Source: S. Chenney Fourier Transforms of Important Functions Sine and cosine transform to frequency spikes Fourier Transforms of Important Functions Sine and cosine transform to frequency spikes A Gaussian transforms to? A Gaussian transforms to a Gaussian A box filter transforms to 4 Imae Source: S. Chenney? 4 Imae Source: S. Chenney Fourier Transforms of Important Functions Duality Sine and cosine transform to frequency spikes The better a function is localized in one domain, the worse it is localized in the other. A Gaussian transforms to a Gaussian All of this is symmetric! A box filter transforms to a sinc sin x sinc( x) x This is true for any function 42 Imae Source: S. Chenney 43 7

8 Effect of Convolution Effect of Filterin Convolvin two functions in the imae domain corresponds to takin the product of their transformed versions in the frequency domain. Noise introduces hih frequencies. To remove them, we want to apply a low-pass filter. f? ( F G The ideal filter shape in the frequency domain would be a box. But this transfers to a spatial sinc, which has infinite spatial support. This ives us a tool to manipulate imae spectra. A filter attenuates or enhances certain frequencies throuh this effect. A compact spatial box filter transfers to a frequency sinc, which creates artifacts. A Gaussian has compact support in both domains. This makes it a convenient choice for a low-pass filter Low-Pass vs. Hih-Pass Quiz: What Effect Does This Filter Have? Low-pass filtered Oriinal imae Hih-pass filtered 46 Imae Source: S. Chenney 47 Source: D. Lowe Sharpenin Filter Sharpenin Filter Oriinal Sharpenin filter Accentuates differences with local averae 48 Source: D. Lowe 49 Source: D. Lowe 8

9 Application: Hih Frequency Emphasis Oriinal Hih pass Filter Topics of This Lecture Linear filters What are they? How are they applied? Application: smoothin Gaussian filter What does it mean to filter an imae? Nonlinear Filters Median filter Multi-Scale representations How to properly rescale an imae? Hih Frequency Emphasis Slide credit: Michal Irani Hih Frequency Emphasis + Historam Equalization 5 Imae derivatives How to compute radients robustly? 5 Non-Linear Filters: Median Filter Median Filter Basic idea Replace each pixel by the median of its neihbors. Salt and pepper noise Median filtered Properties Doesn t introduce new pixel values Removes spikes: ood for impulse, salt & pepper noise Linear? Plots of a row of the imae Imae Source: Martial Hebert Median Filter The Median filter is ede preservin. Median vs. Gaussian Filterin 3x3 5x5 7x7 Gaussian Median 54 Slide credit: Svetlana Lazebnik 55 9

10 Topics of This Lecture Motivation: Fast Search Across Scales Linear filters What are they? How are they applied? Application: smoothin Gaussian filter What does it mean to filter an imae? Nonlinear Filters Median filter Multi-Scale representations How to properly rescale an imae? Filters as templates Correlation as template matchin Imae Source: Irani & Basri Imae Pyramid How Should We Go About Resamplin? Low resolution Let s resample the checkerboard by takin one sample at each circle. In the top left board, the new representation is reasonable. Top riht also yields a reasonable representation. Bottom left is all black (dubious) and bottom riht has checks that are too bi. Hih resolution Imae Source: Forsyth & Ponce Fourier Interpretation: Discrete Samplin Fourier Interpretation: Discrete Samplin Samplin in the spatial domain is like multiplyin with a spike function. Samplin in the spatial domain is like multiplyin with a spike function. Samplin in the frequency domain is like... Samplin in the frequency domain is like convolvin with a spike function.? * 6 Source: S. Chenney 6 Source: S. Chenney

11 Samplin and Aliasin Samplin and Aliasin Nyquist theorem: In order to recover a certain frequency f, we need to sample with at least 2f. This corresponds to the point at which the transformed frequency spectra start to overlap (the Nyquist limit) 62 Imae Source: Forsyth & Ponce 63 Imae Source: Forsyth & Ponce Samplin and Aliasin Aliasin in Graphics 64 Imae Source: Forsyth & Ponce 65 Imae Source: Alexej Efros Resamplin with Prior Smoothin The Gaussian Pyramid Low resolution G 4 ( G 3 * aussian) 2 G G * aussian blur ) 2 3 ( 2 blur G 2 ( G * aussian) 2 blur G ( G * aussian) 2 G Imae blur Note: We cannot recover the hih frequencies, but we can avoid artifacts by smoothin before resamplin. 66 Imae Source: Forsyth & Ponce Hih resolution 67 Source: Irani & Basri

12 Gaussian Pyramid Stored Information Summary: Gaussian Pyramid All the extra levels add very little overhead for memory or computation! Construction: create each level from previous one Smooth and sample Smooth with Gaussians, in part because a Gaussian*Gaussian = another Gaussian G( ) * G( 2 ) = G(sqrt( )) Gaussians are low-pass filters, so the representation is redundant once smoothin has been performed. There is no need to store smoothed imaes at the full oriinal resolution. 68 Source: Irani & Basri Slide credit: David Lowe 69 The Laplacian Pyramid Gaussian Pyramid G n G 2 G L G expand( G ) i i i i i G L expand( G ) i - = Laplacian Pyramid Ln G n - L = 2 L Laplacian ~ Difference of Gaussian - = DoG = Difference of Gaussians Cheap approximation no derivatives needed. G L - = - = Why is this useful? 7 Source: Irani & Basri 7 Topics of This Lecture Note: Filters are Templates Linear filters What are they? How are they applied? Application: smoothin Gaussian filter What does it mean to filter an imae? Applyin a filter at some point can be seen as takin a dotproduct between the imae and some vector. Filterin the imae is a set of dot products. Insiht Filters look like the effects they are intended to find. Filters find effects they look like. Nonlinear Filters Median filter Multi-Scale representations How to properly rescale an imae? Filters as templates Correlation as template matchin

13 Where s Waldo? Where s Waldo? Template Template Scene 74 Detected template 75 Where s Waldo? Correlation as Template Matchin Think of filters as a dot product of the filter vector with the imae reion Now measure the anle between the vectors ab ab a b cos cos a b Anle (similarity) between vectors can be measured by normalizin lenth of each vector to. Detected template Correlation map Template a b 76 Imae reion Vector interpretation 77 Summary: Mask Properties Summary Linear Filters Smoothin Values positive Sum to constant reions same as input Amount of smoothin proportional to mask size Remove hih-frequency components; low-pass filter Filters act as templates Hihest response for reions that look the most like the filter Dot product as correlation Linear filterin: Form a new imae whose pixels are a weihted sum of oriinal pixel values Properties Output is a shift-invariant function of the input (same at each imae location) Examples: Smoothin with a box filter Smoothin with a Gaussian Findin a derivative Searchin for a template Pyramid representations Important for describin and searchin an imae at all scales

14 References and Further Readin Backround information on linear filters and their connection with the Fourier transform can be found in Chapters 7 and 8 of D. Forsyth, J. Ponce, Computer Vision A Modern Approach. Prentice Hall,