INTEREST POINTS AT DIFFERENT SCALES
Thank you for the slides. They come mostly from the following sources. Dan Huttenlocher Cornell U David Lowe U. of British Columbia Martial Hebert CMU
Intuitively, junctions of contours. Generally more stable features over changes of viewpoint Intuitively, large variations in the neighborhood of the point in all directions 2
The distribution of the x and y derivatives is very different for all three types of patches The distribution of x and y derivatives can be characterized by the shape and size of the principal component ellipse 3
5
Corner detection: the basic idea At a corner, shifting a window in any direction should give a large change in intensity flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions 6
A simple corner detector Define the sum squared difference (SSD) between an image patch and a patch shifted by offset (x,y): where w(u,v) = or 1 in window, 0 outside Gaussian If s(x,y) is high for shifts in all 8 directions, declare a corner. Problem: not isotropic 7
Harris corner detector derivation Second-order Taylor series approximation: where A is defined in terms of partial derivatives I x = I/ x and I y = I/ y summed over (u,v): For constant t, S(x,y) < t is an ellipse 8
Eigenvector analysis The eigenvectors v 1, v 2 of A give an orthogonal basis for the ellipse I.e. directions of fastest and slowest change for λ 2 > λ 1, v 1 is the direction of fastest change (minor axis of ellipse) and v 2 is the direction of slowest change (major axis) direction of the fastest change v 2 direction of the slowest change (λ 2 ) -1/2 (λ 1 ) -1/2 v 1 9
Classify points based on eigenvalues Classification of image points using eigenvalues of M: λ 2 Edge λ 2 >> λ 1 Corner λ 1 and λ 2 are large, λ 1 ~ λ 2 ; E increases in all directions λ 1 and λ 2 are small; E is almost constant in all directions Flat region Edge λ 1 >> λ 2 λ 1 10
Harris corner detection But square roots are expensive Approximate corner response function that avoids square roots: R ( ) 2 = λ λ k λ + λ 1 2 1 2 with k is set empirically After thresholding, keep only local maxima of R as corners prevents multiple detections of the same corner 13
Harris detector, step-by-step 14
Harris detector, step-by-step Compute corner response R 15
Harris detector, step-by-step Threshold on corner response R 16
Harris detector, step-by-step Take only local maxima of R 17
Harris detector properties Invariant to intensity shift: I = I + b only derivatives are used, not original intensity values Insensitive to intensity scaling: I = a I R R threshold x (image coordinate) x (image coordinate) So Harris is insensitive to affine intensity changes I.e. linear scaling plus a constant offset, I = a I + b 21
Rotation invariance Harris detector properties Ellipse (eigenvectors) rotate but shape (eigenvalues) remain the same Corner response R is invariant to image rotation 20
Rotation Invariance Cordelia Schmid & Roger Mohr (97) Apply Harris corner detector Use rotational invariants at corner points However, not scale invariant. Sensitive to viewpoint and illumination change.
Experimental evaluation C.Schmid et.al. Evaluation of Interest Point Detectors. IJCV 2000 24
Harris detector properties But Harris is not invariant to image scale All points will be classified as edges Corner! 22
Experimental evaluation Quality of Harris detector for different scale changes Repeatability rate: # correspondences # possible correspondences C.Schmid et.al. Evaluation of Interest Point Detectors. IJCV 2000 23
Scale invariant interest point detection Consider regions (e.g. circles) of different sizes around a point Regions of corresponding sizes will look the same in both images 25
Scale invariant detection The problem: how do we choose corresponding circles independently in each image? 26
A solution Design a function which is scale invariant I.e. value is the same for two corresponding regions, even if they are at different scales Example: average intensity is the same for corresponding regions, even of different sizes For a given point in an image, consider the value of f as a function of region size (circle radius) f scale = 1/2 f region size region 27 size
A solution Take a local maximum of this function The region size at which maximum is achieved should be invariant to image scale This scale invariant region size is determined independently in each image f scale = 1/2 f s 1 region size s 2 region 28 size
Choosing a function A good function for scale detection has one sharp peak f f f Good! bad bad region size region size region size A function that responds to image contrast is a good choice e.g. convolve with a kernel like the Laplacian or the Difference of Gaussians 29
ML = µ ( xl, ΣL) MR = µ ( xr, ΣR) x Ax 1 x M 2 x L 1 x M 2 x R whitening 1 2 R 2 ( M x ) = R( M x R 1 L L ) Isotropic neighborhoods related by rotation Localization: Compute location of interest point using local maximum of the Harris measure Iterate Scale: Compute the scale at that point using Laplacian Affine neighborhood: Normalize neighborhood with second moment matrix Yes, it converges 5
Why invariance? Reminder: Finding scale invariant features Laplacian Characteristic scale : - maximum in scale space - scale invariant 4
Affine invariant detection Harris-Affine [Mikolajczyk & Schmid, IJCV04]: Use Harris moment matrix to select dominant directions and anisotropy 23
Affine invariant detection Matching Widely Separated Views Based on Affine Invariant Regions, T. TUYTELAARS and L. VAN GOOL, IJCV 2004 24
Affine invariant detection The regions found may not exactly correspond, so we approximate them with ellipses Geometric Moments: mpq p q = x y f( x, y) dxdy Fact: moments m pq uniquely 2 determine the function f Taking f to be the characteristic function of a region (1 inside, 0 outside), moments of orders up to 2 allow to approximate the region by an ellipse This ellipse will have the same moments of orders up to 2 as the original region 26
Affine invariant detection Algorithm summary (detection of affine invariant region): Start from a local intensity extremum point Go in every direction until the point of extremum of some function f Curve connecting the points is the region boundary Compute geometric moments of orders up to 2 for this region Replace the region with ellipse 27
Scale Invariant Detection: Summary Given: two images of the same scene with a scale difference between them Goal: find the same interest points independently in each image Solution: search for maxima of suitable functions in scale and in space (over the image) Methods: 1. Harris-Laplacian [Mikolajczyk, Schmid]: maximize Laplacian over scale, Harris measure of corner response over the image 2. SIFT [Lowe]: maximize Difference of Gaussians over scale and space 14
Scale-Invariant Local Features Image content is transformed into local feature coordinates that are invariant to translation, rotation, scale, and other imaging parameters SIFT Features
Advantages of invariant local features Locality: features are local, so robust to occlusion and clutter (no prior segmentation) Distinctiveness: individual features can be matched to a large database of objects Quantity: many features can be generated for even small objects Efficiency: close to real-time performance Extensibility: can easily be extended to wide range of differing feature types, with each adding robustness
Resam Bl ple ur Sub tract Build Scale-Space Pyramid All scales must be examined to identify scale-invariant features An efficient function is to compute the Difference of Gaussian (DOG) pyramid (Burt & Adelson, 1983) Resample Blur Subtract
Scale space processed one octave at a time
Sampling frequency for scale More points are found as sampling frequency increases, but accuracy of matching decreases after 3 scales/octave
Resam ple ur Sub tract Key point localization Detect maxima and minima of difference-of-gaussian in scale space Bl
Example of keypoint detection Threshold on value at DOG peak and on ratio of principle curvatures (Harris approach) (a) 233x189 image (b) 832 DOG extrema (c) 729 left after peak value threshold (d) 536 left after testing ratio of principle curvatures
Select canonical orientation Create histogram of local gradient directions computed at selected scale Assign canonical orientation at peak of smoothed histogram Each key specifies stable 2D coordinates (x, y, scale, orientation) 0 2π
SIFT Orientation Assignment Orientation determined by largest peak Assign second orientation if second peak at least.8 height About 15% of keypoints have two orientations Parabolic fit to three values around peak to interpolate orientation 0 2π 17
SIFT Descriptor 8 orientations and 4x4 array computed by histogramming over 16x16 image region 128 dimensional feature vector Illustration shows 2x2 array and 8x8 region 19
SIFT Matching Example Example objects in cluttered environment Rectangle around detected objects based on model images at left 20
Location recognition
Scale Invariant Detectors Experimental evaluation of detectors w.r.t. scale change Repeatability rate: # correspondences # possible correspondences K.Mikolajczyk, C.Schmid. Indexing Based on Scale Invariant Interest Points. ICCV 2001 13
Feature Matching & Geometric Relations Features from training image(s) Features from input image Feature Matching & Geometric Relations Features from training image(s) Features from input image Aspect is different between training and test images invariant features Local feature similarity is not sufficient use global geometric consistency Large number of features define distance in feature space + efficient indexing 3
Lucas-Kanade (1981) is an image alignment algorithm in a optical motion routines. Align a (2D) template T(x) to a (2D) image I(x) where the image have a parameterized warps, e.g., affine with six parameters, or homography with eight. See the reading list: Lucas-Kanade 20 years on: A unifying framework.