Image Filtering Slides, adapted from Steve Seitz and Rick Szeliski, U.Washington
The power of blur All is Vanity by Charles Allen Gillbert (1873-1929) Harmon LD & JuleszB (1973) The recognition of faces. Scientific American 229(5):71 82
The power of blur All is Vanity by Charles Allen Gillbert (1873-1929) Harmon LD & JuleszB (1973) The recognition of faces. Scientific American 229(5):71 82
Image filtering applications Today: 1. Noise removal 2. Blur 3. Image gradient 4. Edge detection 5. Image pyramids
Images as functions We can think of an image as a function, f: R 2 -> R f( x, y ) = intensity at position/pixel ( x, y ) Realistically, we expect the image only to be defined over a rectangle, with a finite range: f: [a,b]x[c,d] -> [,1] A color image is just three functions pasted together. We will focus mostly on intensity images.
Images as functions
What is a digital image We usually work with digital (discrete) images: Sample the 2D space on a regular grid Quantize each sample (round to nearest integer)
What is a digital image We usually work with digital (discrete) images: Sample the 2D space on a regular grid Quantize each sample (round to nearest integer) The image can now be represented as a matrix of integer values
Filtering noise Image
Filtering noise 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 Image 9 9 9 9 9 9 9 9 9 9
Filtering noise 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 3 6 9 8 8 5 3 3 6 9 8 8 5 3 3 6 9 9 9 6 3 2 4 6 6 6 4 2 1 2 3 3 3 2 1 Mean image Image 9 9 9 9 9 9 9 9 9 9 1 1 1 1 2 3 3 3 3 2 1 2 4 6 5 5 3 2 3 6 9 8 8 5 3
Filtering noise 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 3 6 9 8 8 5 3 3 6 9 8 8 5 3 3 6 9 9 9 6 3 2 4 6 6 6 4 2 1 2 3 3 3 2 1 Mean image Image 9 9 9 9 9 9 9 9 9 9 1 1 1 1 2 3 3 3 3 2 1 2 4 6 5 5 3 2 3 6 9 8 8 5 3 What happens at boundaries?
Cross-correlation filtering Let s write this down as an equation. Assume the averaging window is (2k+1)x(2k+1): We can generalize this idea by allowing different weights for different neighboring pixels:
Cross-correlation filtering This is called a cross-correlation operation H is called the filter, kernel, or mask. Note: The above allows negative filter indices. When you implement need to use: H[u+k,v+k] instead of H[u,v]
Mean kernel What s the kernel for a 3x3 mean filter? 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
Mean kernel What s the kernel for a 3x3 mean filter? 9 9 9 9 9 9 9 9 9 9 1 1 1 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 1 1 1 1 1 1 1 1 1 9 1
Mean vs. Gaussian filtering = Blur!!!
Gaussian kernel A Gaussian kernel gives less weight to pixels further from the center of the window 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 1 2 1 2 4 2 1 2 1 9
Gaussian filtering This kernel is an approximation of a Gaussian function: What happens if you increase σ?
Mean vs. Gaussian filtering
Image filtering applications Today: 1. Noise removal 2. Blur 3. Image gradient 4. Edge detection 5. Image pyramids
Image gradient as cross-correlation How can we differentiate a digital image F[x,y]? Option 1: reconstruct a continuous image, image, f, then take gradient Option 2: take discrete derivative (finite difference)
Image gradient as cross-correlation How can we differentiate a digital image F[x,y]? Option 1: reconstruct a continuous image, image, f, then take gradient Option 2: take discrete derivative (finite difference) How would you implement this as a cross-correlation?
Image gradient as cross-correlation How can we differentiate a digital image F[x,y]? Option 1:reconstruct a continuous image, image, f, then take gradient Option 2: take discrete derivative (finite difference) How would you implement this as a cross-correlation?
Image gradient as cross-correlation How can we differentiate a digital image F[x,y]? Option 1: reconstruct a continuous image, image, f, then take gradient Option 2: take discrete derivative (finite difference) How would you implement this as a cross-correlation? Forward derivative
Image gradient as cross-correlation How can we differentiate a digital image F[x,y]? Option 1: reconstruct a continuous image, image, f, then take gradient Option 2: take discrete derivative (finite difference) How would you implement this as a cross-correlation? Central derivative
Image gradient
Image gradient The gradient direction is given by: how does this relate to the direction of the edge?
Image gradient The gradient direction is given by: how does this relate to the direction of the edge? The edge strength is given by the gradient magnitude
Convolution A convolution operation is a cross-correlation where the filter is flipped both horizontally and vertically before being applied to the image: It is written: Suppose H is a Gaussian or mean kernel. How does convolution differ from cross-correlation?
Convolution A convolution operation is a cross-correlation where the filter is flipped both horizontally and vertically before being applied to the image: It is written: Suppose H is a Gaussian or mean kernel. How does convolution differ from cross-correlation?
Image filtering applications Today: 1. Noise removal 2. Blur 3. Image gradient 4. Edge detection 5. Image pyramids
Effects of noise Consider a single row or column of the image Plotting intensity as a function of position gives a signal Where is the edge?
Solution: smooth first Where is the edge? Look for peaks in
Derivative theorem of convolution This saves us one operation: Where is the edge? Maxima point
Laplacian of Gaussian Consider Laplacian of Gaussian operator Where is the edge? Zero-crossings of bottom graph
2D edge detection filters Gaussian derivative of Gaussian Laplacian of Gaussian is the Laplacian operator
Laplacian filter Gaussian delta function Laplacian of Gaussian (2 nd derivative operator)
Laplacian image smoothed (5x5 Gaussian) delta function Laplacian of Gaussian (2 nd derivative operator)
Laplacian image smoothed (5x5 Gaussian) original Laplacian of Gaussian (2 nd derivative operator)
Laplacian image smoothed (5x5 Gaussian) original Laplacian image smoothed original (scaled by 4, offset +128)
Laplacian image smoothed (5x5 Gaussian) original Laplacian image smoothed original (scaled by 4, offset +128) What happens as we change σ?
Scale space (Witkin 83) larger Gaussian filtered signal
Scale space: insights As the scale is increased edge position can change edges can disappear new edges are not created Bottom line need to consider information at different scales (or else know what scale you care about)
Image filtering applications Today: 1. Noise removal 2. Blur 3. Image gradient 4. Edge detection 5. Image pyramids
Image Pyramids
Image sub-sampling 1/8 1/4 Throw away every other row and column to create a ½ size image
Image sub-sampling 1/2 1/4 (2x zoom) 1/8(4x zoom) Why does this look so bad?
Even worse for synthetic images
Really bad in video
Gaussian pre-filtering G 1/8 G 1/4 Gaussian 1/2 Solution: filter the image, image, then subsample In video: filter also temporally
Subsampling with Gaussian pre-filtering Gaussian 1/2 Solution: filter the image, G 1/4 G 1/8
Compare with... 1/2 1/4 (2x zoom) 1/8(4x zoom)
Pyramid Creation
Octaves in the Spatial Domain Lowpass Images Bandpass Images
Pyramids Advantages of pyramids Faster than Fourier transform Avoids ringing artifacts Many applications small images faster to process good for multiresolution processing compression progressive transmission
Example: pyramid blending
Laplacian level 4 Laplacian level 2 Laplacian level left pyramid right pyramid blended pyramid
Pyramid Blending
Video Pyramids Same idea in 3 dimensions x,y,t Space-time pyramid: Temporal pyramid:
Example: action recognition
Recognizing Actions Use dense information Gradient orientation <--> space-time shape information (behavior) Gradient magnitude <--> photometric information (clothing, lighting, contrast, etc)
Behavior-Based Distance Measure Temporal pyramid: Distance measure = the χ2 distance between corresponding empirical distributions
Action recognition results
Next week:features and descriptors