Multimedia Databases. Previous Lecture. 4.1 Multiresolution Analysis. 4 Shape-based Features. 4.1 Multiresolution Analysis

Transcription

1 Previous Lecture Multimedia Databases Texture-Based Image Retrieval Low Level Features Tamura Measure, Random Field Model High-Level Features Fourier-Transform, Wavelets Wolf-Tilo Balke Silviu Homoceanu Institut für Informationssysteme Technische Universität Braunschweig Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 2 4 Shape-based Features 4 Multiresolution Analysis and Shape-based Features 4.2 Shape-based Features - Thresholding - Edge detection - Morphological Operators In the case of images with many pixels (high resolution) wavelet transforms provide highdimensional equation systems Calculation of long feature vectors by solving linear equations? Far too expensive! Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 3 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 4 The wavelet transform of an image can be computed using fast wavelet transform algorithms in linear time It can be calculated by the repetition of two steps: Converting the image into a representation with reduced resolution (pixel count) Storing the image information lost by this transformation (which provides the wavelet coefficients) The underlying technology is called Multiresolution Analysis Idea: Consider the image in different resolutions The image signal is composed of raster" parts and detail parts Therefore: representation of the image by blocks of detailed information from which the image can be restored in stages Example: More detail More detail Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 5 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 6 1

2 4.1 Image Resolution Forming the average in different resolutions Summarize blocks of pixels by using their average as one pixel Averaging and Downsampling Basic idea V k : Pixel raster of the original image V k 1 : Raster of a lower resolution, therefore it has less pixels than V k The process will continue down to V 0, which consists of only one pixel... It still has to be defined for each V i how the intensities of pixels are obtained from the intensities of pixels belonging to V i-1 s coarser raster Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 7 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 8 Usually The intensity of a pixel in the grid V i-1 is the mean of a set of corresponding pixels in the grid V i V 0 has then as intensity the average intensity of the output image V k V i-1 is calculated from V i by halving the number of pixels in width or height 4.1 Example Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 9 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 10 For each pixel (x, y), in the original image there is in each raster V i a pixel p i (x, y) derived from (x, y) through repeated averaging Let f i (x, y) be the intensity of the pixel p i (x, y) in raster V i For each pixel (x, y) of the original image and each i we have: f i (x, y) = f i-1 (x, y) + d i-1 (x, y) By using the detail information d i (x, y) we can reconstruct the intensity of the pixel (x, y) in the original image f i (x, y): Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 11 Details are often described by the differences of averages ( Averaging and Differencing ): Differences: Advantage: In images, usually neighbor pixels are similar, thus the differences are often 0. Only strong intensity differences are contained in the compressed image. Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 12 2

3 These two steps correspond to the application of filters in the signal processing: High pass filter: only receives signal components with high frequency (= baby wavelets of higher order) Low pass filter: only receiving signal components with low frequency (= baby wavelets of low order) X(ω) 4.1 Example X HP (ω) X LP (ω) High-pass filter extract the image details, low-pass filter, the averages Four possible applications of both filters, to reduce the image size, both vertically and horizontally by half: HH, HL, LH, LL (sub-band) Save the results of the high pass filter for the subsequent reconstruction of the image ω ω ω Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 13 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 14 Various resolutions LL LH Filtering and Downsampling Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 15 HL HH The total number of pixels in each step is the same, i.e. no loss of information! Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 16 Feature array Save the expected value and standard deviation of Wavelet coefficients at each resolution E.g., three-stage resolution is a 20-dimensional feature array 4.2 Shape-based Features Shape-based retrieval Occurring shapes contribute significantly to the similarity of images In contrast to the purely visual impression made by colors or textures, shapes often carry deeper semantic information A displayed item, is often independent of color, however usually identical items have an identical shape Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 17 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 18 3

4 4.2 Example: Chair 4.2 Basic Idea Combination of simple shape-features (round, elliptical, triangular, square, trapezoid,...) with other features (color, texture, etc.) brings better retrieval "Round object in a red-orange image" may be a search for a sunset... Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 19 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig Basic Idea Even more complicated: "Find all coats of arms containing crosses" 4.2 Shape-based Retrieval Fundamental problems How to recognize the shape of things in images? Is a semantic mapping always possible? How do we describe shapes with features? Which shapes are similar and how do you compare different shapes? Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 21 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig Segmentation Shape segmentation is a fundamental problem Which shapes are displayed in the image? All of them? All important? Only foreground motive? 4.2 Segmentation What represents shape and what does not? Is the shape homogeneous in colors or textures? Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 23 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 24 4

5 4.2 Segmentation Are all parts of the shape visible? Is the sun round? Segmentation? 4.2 Automatic Segmentation Can the segmentation be done automatically? At least semi-automatically? Not in early versions of multimedia retrieval! E.g.: IBM's QBIC Image Classifier Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 25 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig IBMs QBIC Prototype 4.2 Segmentation in QBIC Manually or semi-automatic with Flood Fill ("seeded region growing") input image masked marked shape auto-unmask Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 27 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig Segmentation in QBIC Problem Only segmentation of monochrome surfaces 'End' forms 4.2 Automatic Segmentation Many research projects in multimedia retrieval have been working on the topic (e.g., Blobworld, Photobook) input image masked marked shape auto-unmask There are solutions, however mostly for special cases Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 29 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 30 5

6 4.2 Automatic Segmentation Due to segmentation problems, shape features were removed from all commercial databases IBM's QBIC DB2 Image Extender (set) Virage Retrieval Engine Oracle Multimedia Oracle Intermedia Excalibur Technologies Informix Image Foundation DataBlade 4.2 Automatic Segmentation In principle, a form is defined through the outer perimeter Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 31 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig Automatic Segmentation How do you get this outline? Segmentation of areas with the same brightness, color and/or texture Edge detection (differences in brightness, gradient, watersheds, etc.) Filling in the spaces with morphological operators (dilation and erosion) Segmentation of the outline as closed curve (polygon, splines,...) And a large number of other procedures... Usually applied for gray value images Idea: Important objects can be differentiated from the background because of their different brightness range A certain threshold separates the regions Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 33 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 34 Supposition: Thematically related areas have similar gray values Can be clearly separated from the background Fixed threshold A fixed threshold is applied to each image Enough for example in the case of binary images Flexible Threshold Depending on the gray value histogram New threshold for each image Often, the histogram is first smoothed but without moving the peaks Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 35 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 36 6

7 ISODATA algorithm (Ridler and Calvard, 1978) Divide the gray value histogram into two parts Calculate the expectation values of the gray values in the left and right part Compute a new threshold as the average of the two expected values Iteratively compute the new expected values and a new threshold (until the threshold no longer changes significantly) Triangle algorithm (Zack and others, 1977) Connect the highest peak in the histogram with the highest brightness value Maximize the distance to the connecting line Threshold is minimum, shifted by some constant value Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 37 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig Application example Medical segmentation There are also area-based algorithms, which evaluate thresholds of individual image areas to segment an image Applicability depends strongly on each image collection Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 39 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 40 Area based algorithms, especially for color images Advantage Very simple procedure Disadvantage Determination of the "right" Thresholds Supposition: strong color or gray value change between foreground object and background Problem: decomposition of complex objects Segmentation with Edge Flow (Ma and Manjunath, 1997) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 41 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 42 7

8 Strong color change between the foreground object and background? Not the size of the foreground objects will be detected, but limits of such areas The goal is a closed curve around an image object Usually, maxima of the first and second derivative of the brightness function are considered Gradient and Laplace operator Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 43 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 44 How to determine the gradient at (x, y)? Problem: gradients require differentiable (continuous) functions; we only have discrete supporting points Two common solutions: (1) Estimate a differentiable function from the available supporting points and use these (e.g., via Fourier transformation) (2) Estimate the course of this function for each pixel from its immediate neighborhood (e.g., Sobel filter); often much faster than (1) Gradient-based method Calculate the magnitude of the gradient at each point (e.g., Sobel filter) Edges denote high gradient Then use an threshold algorithm to separate the edges from regions Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 45 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 46 Advantage More simple filter Disadvantage Very susceptible to noise (one possibility would be performing noise reduction before applying the Sobel filter) Blurred or merging contours Zero crossing of second derivative ( Laplacian Zero-Crossing ) Is particularly used in "noisy" images with blurred edges The behavior of the gradient is studied starting from an ideal edge Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 47 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 48 8

9 Idea: zero passage of the second derivative shows the maximum of the gradient Unlike the gradient procedure, it is not expected that every point with a sufficiently high gradient value to be assigned to the edge, but only the points on the zero-crossing Applying a smoothing filter (normally Gaussian filter) before calculating the derivative prevents the susceptibility to noise Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 49 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 50 Important: Only real zero crossings, not zero points Mark all pixels with zero crossings and multiply them by the "strength" of the edge (e.g., magnitude of the gradient) Again, we can bring thresholding in performing segmentation Example: Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 51 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 52 Comparison between gradient procedure and zero crossing technique: Sobel and Zero-crossing filters in Matlab Transform image to gray scale values sobel = edge(img, sobel ) zeroc = edge(img, zerocross ) original gradient zero crossing Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 53 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 54 9

10 4.2 Watersheds 4.2 Watersheds Watershed transformation Supposition: surfaces are defined by minimal gray values and their zone of influence Idea: Flooding" a surface judging by the minimum gray value, so that different surfaces do not connect Gray values can be seen as topographical surfaces or "Mountains" Example: Flood regions based on the minimum gray values Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 55 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig Watersheds For image segmentation: Watershed transformation of the gradient : 4.2 Watersheds Advantage Enclosed and correct bordering Disadvantage Difficult to implement efficiently Over segmentation original gradient water separation segmentation Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 57 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig Active Contour 4.2 Example Supposition Regions are bordered by a predominantly closed curve ("salient boundary") Method Based on a curve ("snake") iterate towards the best possible separation Minimize the energy of the snake curve Internal energy: curvature and continuity External energy: image energy (gradient) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 59 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 60 10

11 4.2 Active Contour Advantage Fits also "fuzzy" edges Disadvantage Complexity of the curve increases with the accuracy of contour Where does the initial snake curve come from? Problem: Noise can make shape recognition difficult Goal: Make the contours of surfaces easily recognizable and easy to describe Solution: Apply morphological operators as a preprocessing step Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 61 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 62 Morphological operators are binary neighborhood operations for changing the surfaces Pixels are removed or added to the object edges by such operations These operations are controlled by an operator mask (the "structure element") Basic operators Dilation inflating, adding pixels to the area Erosion shrinking, removing pixels from the area Typical structural elements are symmetric areas of a pixel Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 63 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 64 Dilation The structural element will be applied on all pixels of the source image The structural element defines a neighborhood around each pixel In the dilated image the black pixels, are exactly the pixels which had a black pixel in their neighborhood in the original image Effects: increasing of the area connecting objects with small distance Example of a dilation with various structural elements Original pixel New pixel after dilation Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 65 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 66 11

12 Erosion The structural element is again applied on every pixel of the source image The structural element again define neighborhood environments In the resulting image the white pixels, are exactly the pixels which had a white pixel in their neighborhood Effects: thin spots disappear separation of areas with small intervals Example Original Dilation Erosion Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 67 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 68 Opening - erosion followed by dilation Elimination of thin and small objects Breaking of thin areas Smoothing of the edges Closing - dilation followed by erosion Small holes are filled Joining close objects Smoothing of the edges Example Original Opening Closing Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 69 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 70 Advantages Using morphological operators for image processing makes it easier to obtain good shapes Disadvantages Gray values of the areas must be uniform Precise control is relatively difficult This Lecture Multiresolution Analysis Shape-based Features - Thresholding - Edge detection - Morphological Operators Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 71 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 72 12

13 Next lecture Query by Visual Examples Shape-based Features Chain Codes Fourier Descriptors Moment Invariants Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 73 13