#1 Lecture 8: Interest Point Detection Saad J Bedros sbedros@umn.edu
Last Lecture : Edge Detection Preprocessing of image is desired to eliminate or at least minimize noise effects There is always tradeoff between sensitivity and accuracy in edge detection The parameters that we can set for edge detection and labeling include the size of the edge detection mask and the value of the threshold A larger mask or a higher gray level threshold will tend to reduce noise effects, but may result in a loss of valid edge points #2
Example original image (Lena)
Compute Gradients X-Derivative of Gaussian Y-Derivative of Gaussian Gradient Magnitude
Get Orientation at Each Pixel Threshold at minimum level Get orientation theta = atan2(gy, gx)
#6
Edge Thinning: Non-maximum suppression for each orientation At q, we have a maximum if the value is larger than those at both p and at r. Interpolate to get these values.
Today s Lecture Goal: Find features between multiple images taken from different position or time to be robustly matched Approaches: Patch/Template Matching Interest Point Detection
Applications Comparison Between 2 or more images Image alignment Image Stitching 3D reconstruction Object recognition ( matching ) Indexing and database retrieval Object tracking Robot navigation
What points would you choose? Known as: Template Matching Method, with option to use correlation as a metric
Correlation #11
Correlation #12 Use Cross Correlation to find the template in an image Maximum indicate high similarity
Template/Patch Matching Options What do we do for different scales of the patch? Compute multiple Templates at different sizes Match for different scales of the template What can we do to identify the shape of the pattern at different mean brightness levels Remove the mean of the template Example in 1D
We need more robust feature descriptors for matching!!!!!
Interest Points Local features associated with a significant change of an image property of several properties simultaneously (e.g., intensity, color, texture).
Why Extract Interest Points? Corresponding points (or features) between images enable the estimation of parameters describing geometric transforms between the images.
What if we don t know the correspondences?? ( ) ( ) feature descriptor Need to compare feature descriptors of local patches surrounding interest points =? feature descriptor
Properties of good features Local: features are local, robust to occlusion and clutter (no prior segmentation!). Accurate: precise localization. Invariant / Covariant Robust: noise, blur, compression, etc. do not have a big impact on the feature. Repeatable Distinctive: individual features can be matched to a large database of objects. Efficient: close to real-time performance.
Invariant / Covariant A function f( ) is invariant under some transformation T( ) if its value does change when the transformation is applied to its argument: if f(x) = y then f(t(x))=y A function f( ) is covariant when it commutes with the transformation T( ): if f(x) = y then f(t(x))=t(f(x))=t(y)
Invariance Features should be detected despite geometric or photometric changes in the image. Given two transformed versions of the same image, features should be detected in corresponding locations.
Example: Panorama Stitching How do we combine these two images?
Panorama stitching (cont d) Step 1: extract features Step 2: match features
Panorama stitching (cont d) Step 1: extract features Step 2: match features Step 3: align images
What features should we use? Use features with gradients in at least two (significantly) different orientations patches? Corners?
What features should we use? (cont d) (auto-correlation)
Corners vs Edges #27
Corners in Images #28
The Aperture Problem #29
Corner Detection #30
Corner Detection-Basic Idea #31
Corner Detector: Basic Idea flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions
Corner Detection Using Intensity: Basic Idea Image gradient has two or more dominant directions near a corner. Shifting a window in any direction should give a large change in intensity. flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions
Corner Detection Using Edge Detection? Edge detectors are not stable at corners. Gradient is ambiguous at corner tip. Discontinuity of gradient direction near corner.
Corner Types Example of L-junction, Y-junction, T-junction, Arrow-junction, and X-junction corner types
Mains Steps in Corner Detection 1. For each pixel in the input image, the corner operator is applied to obtain a cornerness measure for this pixel. 2. Threshold cornerness map to eliminate weak corners. 3. Apply non-maximal suppression to eliminate points whose cornerness measure is not larger than the cornerness values of all points within a certain distance.
Mains Steps in Corner Detection (cont d)
Moravec Detector (1977) Measure intensity variation at (x,y) by shifting a small window (3x3 or 5x5) by one pixel in each of the eight principle directions (horizontally, vertically, and four diagonals).
Moravec Detector (1977) Calculate intensity variation by taking the sum of squares of intensity differences of corresponding pixels in these two windows. 8 directions x, y in {-1,0,1} S W (-1,-1), S W (-1,0),...S W (1,1)
Moravec Detector (cont d) The cornerness of a pixel is the minimum intensity variation found over the eight shift directions: Cornerness(x,y) = min{s W (-1,-1), S W (-1,0),...S W (1,1)} Cornerness Map (normalized) Note response to isolated points!
Moravec Detector (cont d) Use Non-maximal suppression will yield the final corners. Process of Zero out all pixels that are not the maximum along the direction of the gradient (look at 1 pixel on each side)
Moravec Detector (cont d) Does a reasonable job in finding the majority of true corners. Edge points not in one of the eight principle directions will be assigned a relatively large cornerness value.
Moravec Detector (cont d) The response is anisotropic (directionally sensitive) as the intensity variation is only calculated at a discrete set of shifts (i.e., not rotationally invariant)
Corner Detection: Mathematics Change in appearance of window w(x,y) for the shift [u,v]: xy, 2 Euv (, ) wxy (, ) I( x u, y v) I( x, y) I(x, y) E(u, v) E(3,2) w(x, y)
Corner Detection: Mathematics Change in appearance of window w(x,y) for the shift [u,v]: xy, 2 Euv (, ) wxy (, ) I( x u, y v) I( x, y) I(x, y) E(u, v) E(0,0) w(x, y)
Harris Detector: Mathematics Change of intensity for the shift [u,v]: xy, 2 Euv (, ) wxy (, ) I( x u, y v) I( x, y) Window function Shifted intensity Intensity Window function w(x,y) = or 1 in window, 0 outside Gaussian
Harris Detector #47 Improves the Moravec operator by avoiding the use of discrete directions and discrete shifts. Uses a Gaussian window instead of a square window.
Harris Detector (cont d) Using first-order Taylor approximation: Reminder: Taylor expansion ( n) f ( a) f ( a) 2 f ( a) n n 1 f( x) f( a) ( x a) ( x a) ( x a) O( x ) 1! 2! n!
Since Harris Detector (cont d)
Harris Detector (cont d) A W (x,y)= f ( xi, yi) ( ) x 2 f ( xi, yi) ( ) y 2 2 x 2 matrix (auto-correlation or 2 nd order moment matrix)
Harris Detector General case use window function: 2 S ( x, y) w( x, y ) f( x, y ) f( x x, y y) W i i i i i i x, y i i default window function w(x,y) : 1 in window, 0 outside A W 2 wxy (, ) fx wxy (, ) fx fy 2 xy, xy, fx fx f y w( x, y) 2 2 xy, fx fy fy wxy (, ) fx fy wxy (, ) f y xy, xy,
Harris Detector (cont d) Harris uses a Gaussian window: w(x,y)=g(x,y,σ I ) where σ I is called the integration scale window function w(x,y) : Gaussian A W 2 wxy (, ) fx wxy (, ) fx fy 2 xy, xy, fx fx f y w( x, y) 2 2 xy, fx fy fy wxy (, ) fx fy wxy (, ) f y xy, xy,
Harris Detector A W 2 fx fx f y 2 xy, fx fy fy Describes the gradient distribution (i.e., local structure) inside window! Does not depend on x, y
Harris Detector (cont d) 0 AW R R 0 2 1 1 Since M is symmetric, we have: We can visualize A W as an ellipse with axis lengths determined by the eigenvalues and orientation determined by R Ellipse equation: x [ x y] AW const y direction of the fastest change (λ min ) -1/2 (λ max ) -1/2 direction of the slowest change
Harris Corner Detector (cont d) The eigenvectors of A W encode direction of intensity change. v 1 The eigenvalues of A W encode strength of intensity change. v 2
Harris Detector (cont d) Eigenvectors encode edge direction Eigenvalues encode edge strength direction of the fastest change (λ min ) -1/2 (λ max ) -1/2 direction of the slowest change
Distribution of fx and fy
Distribution of f x and f y (cont d)
Harris Detector (cont d) % 2 (assuming that λ 1 > λ 2 )
Harris Detector Measure of corner response: (k empirical constant, k = 0.04-0.06) (Shi-Tomasi variation: use min(λ1,λ2) instead of R) http://www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/invariantfeatures.ppt Darya Frolova, Denis Simakov The Weizmann Institute of Science
Harris Detector: Mathematics R depends only on eigenvalues of M R is large for a corner R is negative with large magnitude for an edge Edge R < 0 Corner R > 0 R is small for a flat region Flat R small Edge R < 0
Harris Corner Detector (cont d) Classification of pixels using the eigenvalues of A W : 2 Edge 2 >> 1 Corner 1 and 2 are large, 1 ~ 2 ; intensity changes in all directions 1 and 2 are small; intensity is almost constant in all directions Flat region Edge 1 >> 2 1
Harris Detector (cont d) f( x, y ) i x f( xi, yi) y i x y Gxy (,, )* f( x, y) D i i Gxy (,, )* f( x, y ) D i i σ D is called the differentiation scale
Harris Detector The Algorithm: Find points with large corner response function R (R > threshold) Take the points of local maxima of R
http://www.wisdom.weizmann.ac.il/~deniss/vision_spring04/files/invariantfeatures.ppt Darya Frolova, Denis Simakov The Weizmann Institute of Science Harris corner detector algorithm Compute image gradients Ix Iy for all pixels For each pixel Compute by looping over neighbors x,y compute Find points with large corner response function R (R > threshold) Take the points of locally maximum R as the 65 detected feature points (ie, pixels where R is bigger than for all the 4 or 8 neighbors). 65
Harris Corner Detector (cont d) To avoid computing the eigenvalues explicitly, the Harris detector uses the following function: R(A W ) = det(a W ) α trace 2 (A W ) α: is a const which is equal to: R(A W ) = λ 1 λ 2 - α (λ 1 + λ 2 ) 2
Harris Corner Detector (cont d) Classification of image points using R(A W ): Edge R < 0 Corner R > 0 R(A W ) = det(a W ) α trace 2 (A W ) α: is usually between 0.04 and 0.06 Flat region R small Edge R < 0
Harris Corner Detector (cont d) Other functions: a 2 1 det A tra 1 2 1 2
Harris Corner Detector - Steps 1. Compute the horizontal and vertical (Gaussian) derivatives fx Gxy (,, D)* f( xi, yi) f y Gxy (,, D)* f( xi, yi) x y σ D is called the differentiation scale 2. Compute the three images involved in A W : A W 2 fx fx f y w( x, y) 2 x W, y W fx fy fy
Harris Detector - Steps 3. Convolve each of the three images with a larger Gaussian w(x,y) : 4. Determine cornerness: Gaussian σ I is called the integration scale R(A W ) = det(a W ) α trace 2 (A W ) 5. Find local maxima
Harris Detector - Example
Harris Detector - Example
Harris Detector - Example Compute corner response R
Harris Detector - Example Find points with large corner response: R>threshold
Harris Detector - Example Take only the points of local maxima of R
Harris Detector - Example Map corners on the original image
Harris Detector Scale Parameters The Harris detector requires two scale parameters: (i) a differentiation scale σ D for smoothing prior to the computation of image derivatives, & (ii) an integration scale σ I for defining the size of the Gaussian window (i.e., integrating derivative responses). A W (x,y) A W (x,y,σ I,σ D ) Typically, σ I =γσ D
Invariance to Geometric/Photometric Changes Is the Harris detector invariant to geometric and photometric changes? Geometric Rotation Scale Affine Photometric Affine intensity change: I(x,y) a I(x,y) + b
Harris Detector: Rotation Invariance Rotation Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation
Harris Detector: Rotation Invariance (cont d)
Harris Detector: Photometric Changes Affine intensity change Only derivatives are used => invariance to intensity shift I(x,y) I (x,y) + b Intensity scale: I(x,y) a I(x,y) R R threshold x (image coordinate) x (image coordinate) Partially invariant to affine intensity change
Harris Detector: Scale Invariance Scaling Corner All points will be classified as edges Not invariant to scaling (and affine transforms)
Harris Detector: Repeatability In a comparative study of different interest point detectors, Harris was shown to be the most repeatable and most informative. Repeatability = # correspondences min( m, m ) 1 2 best results: σ I =1, σ D =2 C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of Interest Point Detectors", International Journal of Computer Vision 37(2), 151.172, 2000.
Harris Detector: Disadvantages Sensitive to: Scale change Significant viewpoint change Significant contrast change
How to handle scale changes? A W must be adapted to scale changes. If the scale change is known, we can adapt the Harris detector to the scale change (i.e., set properly σ I, σ D ). What if the scale change is unknown?
Multi-scale Harris Detector Detects interest points at varying scales. R(A W ) = det(a W (x,y,σ I,σ D )) α trace 2 (A W (x,y,σ I,σ D )) scale σ n =k n σ σ D = σ n σ I =γσ D σ n y Harris x
How to cope with transformations? Exhaustive search Invariance Robustness
Multi-scale approach Exhaustive search Slide from T. Tuytelaars ECCV 2006 tutorial
Multi-scale approach Exhaustive search
Multi-scale approach Exhaustive search
Multi-scale approach Exhaustive search
How to handle scale changes? Not a good idea! There will be many points representing the same structure, complicating matching! Note that point locations shift as scale increases. The size of the circle corresponds to the scale at which the point was detected
How to handle scale changes? (cont d) How do we choose corresponding circles independently in each image?
How to handle scale changes? (cont d) Alternatively, use scale selection to find the characteristic scale of each feature. Characteristic scale depends on the feature s spatial extent (i.e., local neighborhood of pixels). scale selection scale selection
How to handle scale changes? Only a subset of the points computed in scale space are selected! The size of the circles corresponds to the scale at which the point was selected.
Automatic Scale Selection Design a function F(x,σ n ) which provides some local measure. Select points at which F(x,σ n ) is maximal over σ n. F(x,σ n ) max of F(x,σ n ) corresponds to characteristic scale! σ n T. Lindeberg, "Feature detection with automatic scale selection" International Journal of Computer Vision, vol. 30, no. 2, pp 77-116, 1998.
Invariance Extract patch from each image individually
Automatic scale selection Solution: Design a function on the region, which is scale invariant (the same for corresponding regions, even if they are at different scales) Example: average intensity. For corresponding regions (even of different sizes) it will be the same. For a point in one image, we can consider it as a function of region size (patch width) f Image 1 f Image 2 scale = 1/2 region size region size
Automatic scale selection Common approach: Take a local maximum of this function Observation: region size, for which the maximum is achieved, should be invariant to image scale. Important: this scale invariant region size is found in each image independently! f Image 1 f Image 2 scale = 1/2 s 1 region size s 2 region size
Automatic Scale Selection f I (, )) ( (, i x f I x )) 1 im i im ( 1 Same operator responses if the patch contains the same image up to scale factor. 100
Automatic Scale Selection Function responses for increasing scale (scale signature) f ( I (, )) i i 1 x i i m f ( I 1 ( x, )) m 101
Automatic Scale Selection Function responses for increasing scale (scale signature) f ( I (, )) i i 1 x i i m f ( I 1 ( x, )) m 102
Automatic Scale Selection Function responses for increasing scale (scale signature) f ( I (, )) i i 1 x i i m f ( I 1 ( x, )) m 103
Automatic Scale Selection Function responses for increasing scale (scale signature) f ( I (, )) i i 1 x i i m f ( I 1 ( x, )) m 104
Automatic Scale Selection Function responses for increasing scale (scale signature) f ( I (, )) i i 1 x i i m f ( I 1 ( x, )) m 105
Automatic Scale Selection Function responses for increasing scale (scale signature) f I (, )) i i i i m f ( I 1 ( x, )) m ( 1 x 106
Scale selection Use the scale determined by detector to compute descriptor in a normalized frame
Automatic Scale Selection (cont d) Characteristic scale is relatively independent of the image scale. Scale factor: 2.5 The ratio of the scale values corresponding to the max values, is equal to the scale factor between the images. Scale selection allows for finding spatial extend that is covariant with the image transformation.
Automatic Scale Selection What local measures should we use? Should be rotation invariant Should have one stable sharp peak
How should we choose F(x,σ n )? Typically, F(x,σ n ) is defined using derivatives, e.g.: Square gradient L x L x LoG L x L x 2 2 2 : ( x(, ) y(, )) 2 : ( xx (, ) yy (, )) DoG : I( x)* G( ) I( x)* G( ) n 1 n :det( 2 W) ( W) Harris function A trace A C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of Interest Point Detectors", International Journal of Computer Vision, 37(2), pp. 151-172, 2000.
Review Patch/Template Matching #111 Interest Points Corner Detection: Harris Detector Properties of Harris Detector Scaling approach