Multimedia Databases. 4 Shape-based Features. 4.1 Multiresolution Analysis. 4.1 Multiresolution Analysis. 4.1 Multiresolution Analysis

Transcription

1 4 Shape-based Features Multimedia Databases Wolf-Tilo Balke Silviu Homoceanu Institut für Informationssysteme Technische Universität Braunschweig 4 Multiresolution Analysis and Shape-based Features 4.2 Shape-based Features - Thresholding - Edge detection - Morphological Operators Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 2 In the case of images with many pixels (high resolution) wavelet transforms provide highdimensional equation systems Calculation of long feature vectors by solving linear equations? Far too expensive! The wavelet transform of an image can be computed using fast wavelet transform algorithms in linear time It can be calculated by the repetition of two steps: Converting the image into a representation with reduced resolution (pixel count) Storing the image information lost by this transformation (which provides the wavelet coefficients) The underlying technology is called Multiresolution Analysis Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 3 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Image Resolution Idea: Consider the image in different resolutions The image signal is composed of raster" parts and detail parts Therefore: representation of the image by blocks of detailed information from which the image can be restored in stages Example: Forming the average in different resolutions Summarize blocks of pixels by using their average as one pixel Averaging and Downsampling More detail More detail Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 5 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 6 1

2 Basic idea V k : Pixel raster of the original image V k 1 : Raster of a lower resolution, therefore it has less pixels than V k The process will continue down to V 0, which consists of only one pixel It still has to be defined for each V i how the intensities of pixels are obtained from the intensities of pixels belonging to V i-1 s coarser raster... Usually The intensity of a pixel in the grid V i-1 is the mean of a set of corresponding pixels in the grid V i V 0 has then as intensity the average intensity of the output image V k V i-1 is calculated from V i by halving the number of pixels in width or height Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 7 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Example For each pixel (x, y), in the original image there is in each raster V i a pixel p i (x, y) derived from (x, y) through repeated averaging Let f i (x, y) be the intensity of the pixel p i (x, y) in raster V i For each pixel (x, y) of the original image and each i we have: f i (x, y) = f i-1 (x, y) + d i-1 (x, y) By using the detail information d i (x, y) we can reconstruct the intensity of the pixel (x, y) in the original image f i (x, y): Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 9 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 10 Details are often described by the differences of averages ( Averaging and Differencing ): These two steps correspond to the application of filters in the signal processing: High pass filter: only receives signal components with high frequency (= baby wavelets of higher order) Low pass filter: only receiving signal components with low frequency (= baby wavelets of low order) X(ω) 4.1 Example X HP (ω) X LP (ω) Differences: Advantage: In images, usually neighbor pixels are similar, thus the differences are often 0. Only strong intensity differences are contained in the compressed image. ω ω ω Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 11 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 12 2

3 High-pass filter extract the image details, low-pass filter, the averages Four possible applications of both filters, to reduce the image size, both vertically and horizontally by half: HH, HT, TH, TT (sub-band) Save the results of the high pass filter for the subsequent reconstruction of the image Filtering and Downsampling Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 13 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 14 Various resolutions Feature array Save the expected value and standard deviation of Wavelet coefficients at each resolution E.g., three-stage resolution is a 20-dimensional feature array HH HT TH TT The total number of pixels in each step is the same, i.e. no loss of information! Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 15 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Shape-based Features 4.2 Example: Chair Shape-based retrieval Occurring shapes contribute significantly to the similarity of images In contrast to the purely visual impression made by colors or textures, shapes often carry deeper semantic information A displayed item, is often independent of color, however usually identical items have an identical shape Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 17 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 18 3

4 11/13/ Basic Idea 4.2 Basic Idea Combination of simple shape-features (round, elliptical, triangular, square, trapezoid,...) with other features (color, texture, etc.) brings better retrieval Even more complicated: "Find all coats of arms containing crosses" "Round object in a red-orange image" may be a search for a sunset Shape-based Retrieval Shape segmentation is a fundamental problem How to recognize the shape of things in images? Is a semantic mapping always possible? How do we describe shapes with features? Which shapes are similar and how do you compare different shapes? Which shapes are displayed in the image? All of them? All important? Only foreground motive? Segmentation Segmentation What represents shape and what does not? Is the shape homogeneous in colors or textures? Segmentation Fundamental problems Are all parts of the shape visible? Is the sun round? Segmentation?

5 11/13/ Automatic Segmentation 4.2 IBMs QBIC Prototype Can the segmentation be done automatically? At least semi-automatically? Not in early versions of multimedia retrieval! E.g.: IBM's QBIC Image Classifier Segmentation in QBIC masked marked shape Problem Only segmentation of monochrome surfaces 'End' forms auto-unmask Segmentation in QBIC Manually or semi-automatic with Flood Fill ("seeded region growing") input image input image Automatic Segmentation masked marked shape auto-unmask Automatic Segmentation Many research projects in multimedia retrieval have been working on the topic (e.g., Blobworld, Photobook) Due to segmentation problems, shape features were removed from all commercial databases IBM's QBIC DB2 Image Extender (set) Virage Retrieval Engine Oracle Multimedia Oracle Intermedia Excalibur Technologies Informix Image Foundation DataBlade There are solutions, however mostly for special cases

6 4.2 Automatic Segmentation In principle, a form is defined through the outer perimeter 4.2 Automatic Segmentation How do you get this outline? Segmentation of areas with the same brightness, color and/or texture Edge detection (differences in brightness, gradient, watersheds, etc.) Filling in the spaces with morphological operators (dilation and erosion) Segmentation of the outline as closed curve (polygon, splines,...) And a large number of other procedures... Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 31 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 32 Usually applied for gray value images Idea: Important objects can be differentiated from the background because of their different brightness range A certain threshold separates the regions Supposition: Thematically related areas have similar gray values Can be clearly separated from the background Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 33 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 34 Fixed threshold A fixed threshold is applied to each image Enough for example in the case of binary images Flexible Threshold Depending on the gray value histogram New threshold for each image Often, the histogram is first smoothed but without moving the peaks ISODATA algorithm (Ridler and Calvard, 1978) Divide the gray value histogram into two parts Calculate the expectation values of the gray values in the left and right part Compute a new threshold as the average of the two expected values Iteratively compute the new expected values and a new threshold (until the threshold no longer changes significantly) Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 35 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 36 6

7 4.2 Application example Triangle algorithm (Zack and others, 1977) Connect the highest peak in the histogram with the highest brightness value Maximize the distance to the connecting line Threshold is minimum, shifted by some constant value Medical segmentation Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 37 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 38 There are also area-based algorithms, which evaluate thresholds of individual image areas to segment an image Applicability depends strongly on each image collection Area based algorithms, especially for color images Segmentation with Edge Flow (Ma and Manjunath, 1997) Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 39 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 40 Advantage Very simple procedure Disadvantage Determination of the "right" Thresholds Supposition: strong color or gray value change between foreground object and background Problem: decomposition of complex objects Strong color change between the foreground object and background? Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 41 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 42 7

8 Not the size of the foreground objects will be detected, but limits of such areas The goal is a closed curve around an image object Usually, maxima of the first and second derivative of the brightness function are considered Gradient and Laplace operator How to determine the gradient at (x, y)? Problem: gradients require differentiable (continuous) functions; we only have discrete supporting points Two common solutions: (1) Estimate a differentiable function from the available supporting points and use these (e.g., via Fourier transformation) (2) Estimate the course of this function for each pixel from its immediate neighborhood (e.g., Sobel filter); often much faster than (1) Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 43 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 44 Gradient-based method Calculate the magnitude of the gradient at each point (e.g., Sobel filter) Edges denote high gradient Then use an threshold algorithm to separate the edges from regions Advantage More simple filter Disadvantage Very susceptible to noise (one possibility would be performing noise reduction before applying the Sobel filter) Blurred or merging contours Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 45 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 46 Zero crossing of second derivative ( Laplacian Zero-Crossing ) Is particularly used in "noisy" images with blurred edges The behavior of the gradient is studied starting from an ideal edge Idea: zero passage of the second derivative shows the maximum of the gradient Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 47 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 48 8

9 Unlike the gradient procedure, it is not expected that every point with a sufficiently high gradient value to be assigned to the edge, but only the points on the zero-crossing Applying a smoothing filter (normally Gaussian filter) before calculating the derivative prevents the susceptibility to noise Important: Only real zero crossings, not zero points Mark all pixels with zero crossings and multiply them by the "strength" of the edge (e.g., magnitude of the gradient) Again, we can bring thresholding in performing segmentation Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 49 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 50 Example: Comparison between gradient procedure and zero crossing technique: original gradient zero crossing Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 51 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 52 Sobel and zero crossing filters in Matlab Transform image to gray scale values sobel = edge(img, sobel ) zeroc = edge(img, zerocross ) Sobel and zero crossing filters in Matlab Original image Sobel filter Zero crossing filter Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 53 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 54 9

10 11/13/2009 Sobel and zero crossing filters in Matlab Sobel and zero crossing filters in Matlab Original image Sobel filter Original image Zero crossing filter Watersheds Sobel filter Zero crossing filter Watersheds Watershed transformation Supposition: surfaces are defined by minimal gray values and their zone of influence Idea: Flooding" a surface judging by the minimum gray value, so that different surfaces do not connect Gray values can be seen as topographical surfaces or "Mountains" Watersheds Example: Flood regions based on the minimum gray values Watersheds For image segmentation: Watershed transformation of the gradient : Advantage Enclosed and correct bordering Disadvantage Difficult to implement efficiently Over segmentation original gradient water separation segmentation

11 4.2 Active Contour 4.2 Example Supposition Regions are bordered by a predominantly closed curve ("salient boundary") Method Based on a curve ("snake") iterate towards the best possible separation Minimize the energy of the snake curve Internal energy: curvature and continuity External energy: image energy (gradient) Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 61 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig Active Contour Advantage Fits also "fuzzy" edges Disadvantage Complexity of the curve increases with the accuracy of contour Where does the initial snake curve come from? Morphological operators are applied to surfaces The segmentation with morphological operators can: Use emerging areas directly as the final segmentation Make the contours of surfaces easily recognizable and easy to describe as a preprocessing step Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 63 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 64 Binary neighborhood operations for changing the surfaces with certain structural elements Pixels are removed or added to the object edges by such operations These operations are controlled by an operator mask (the "structure element") Basic operators Dilation inflating the area Erosion shrinking of the surface Typical structural elements are symmetric areas of a pixel Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 65 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 66 11

12 Dilation The structural element will be applied on all pixels of the source image The black pixels of the structural element define a neighborhood around each pixel In the resulting image the black pixels, are exactly the pixels which had a black pixel in their neighborhood in the original image increasing of the area connecting objects with small distance Example of a dilation with various structural elements Original pixel New pixel after dilation Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 67 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 68 Erosion The structural element is again applied on every pixel of the source image The black pixels of the structural element again define neighborhood environments In the resulting image the white pixels, are exactly the pixels which had a white pixel in their neighborhood thin spots disappear separation of areas with small intervals Example Original Dilation Erosion Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 69 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 70 Opening - erosion followed by dilation Elimination of thin and small objects Breaking of thin areas Smoothing of the edges Closing - dilation followed by erosion Small holes are filled Joining close objects Smoothing of the edges Example Original Opening Closing Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 71 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 72 12

13 Advantages Using morphological operators for image processing makes it easier to obtain good shapes Disadvantages Gray values of the areas must be uniform Precise control is relatively difficult Next lecture Query by Visual Examples Shape-based Features Chain Codes Fourier Descriptors Moment Invariants Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 73 Multimedia Databases Wolf-TiloBalke InstitutfürInformationssysteme TU Braunschweig 74 13