nstance-level recognition: ocal invariant features Cordelia Schmid NRA Grenoble
Overview ntroduction to local features Harris interest t points + SSD ZNCC SFT Scale & affine invariant interest point detectors
ocal features local descriptor Several / man local descriptors per image Robust to occlusion/clutter + no object segmentation required Photometric : distinctive nvariant : to image transformations + illumination changes
ocal features nterest Points Contours/lines Region segments
ocal features nterest Points Contours/lines Region segments Patch descriptors i.e. SFT Mi-points angles Color/teture histogram
nterest points / invariant regions Harris detector Scale/affine inv. detector presented in this lecture
Contours / lines Etraction de contours Zero crossing of aplacian ocal maima of gradients Chain contour points hsteresis Cann detector Recent detectors Global probabilit of boundar gpb detector [Mali et al. UC Berele] Structured forests for fast edge detection SED [Dollar and Zitnic] student presentation
Regions segments / superpiels Simple linear iterative clustering SC Normalized cut [Shi & Mali] Mean Shift [Comaniciu & Meer].
Application: matching Find corresponding locations in the image
llustration Matching nterest t points etracted t with Harris detector t ~ 500 points
llustration Matching Matching nterest t points matched based on cross-correlation 188 pairs
llustration Global constraints Matching Global constraint - Robust estimation of the fundamental matri 99 inliers 89 outliers
Application: Panorama stitching mages courtes of A. Zisserman.
Application: nstance-level recognition Search for particular objects and scenes in large databases
Difficulties Finding the object despite possibl large changes in scale viewpoint lighting and partial occlusion requires invariant description Scale Viewpoint ighting Occlusion
Difficulties Ver large images collection need for efficient i indeing i Flicr has billion photographs h more than 1 million added d dail Faceboo has 15 billion images ~7 million added dail arge personal collections Video collections i.e. YouTube
Applications Search photos on the web for particular places Find these landmars...in these images and 1M more
Applications Tae a picture of a product or advertisement find relevant information on the web
Applications Cop detection for images and videos Quer video Search in 00h of video
Overview ntroduction to local features Harris interest t points + SSD ZNCC SFT Scale & affine invariant interest point detectors
Harris detector [Harris & Stephens 88] Based on the idea of auto-correlation ti mportant t difference in all directions => interest t point
Harris detector Auto-correlation function for a point and a shift A W W
Harris detector Auto-correlation function for a point and a shift A W A { W small in all directions uniform region { large in one directions contour large in all directions interest point
Harris detector Harris detector Discret shifts are avoided based on the auto-correlation matri with first order approimation A W W W
Harris detector Harris detector W W W W W W Auto-correlation matri the sum can be smoothed with a Gaussian the sum can be smoothed with a Gaussian G
Harris detector Auto-correlation matri A G captures the structure of the local neighborhood measure based on eigenvalues of this matri strong eigenvalues => interest point 1 strong eigenvalue => contour 0 eigenvalue => uniform region
nterpreting the eigenvalues Classification of image points using eigenvalues of autocorrelation matri: Edge >> 1 Corner 1 and are large 1 ~ ; \ 1 and are small; Flat region Edge 1 >> 1
R Corner response function det A trace A α: constant 0.04 to 0.06 Edge R <0 1 1 Corner R > 0 Flat region R small Edge R < 0
Harris detector Cornerness function R det A trace A 1 1 Reduces the effect of a strong contour nterest point detection Treshold absolut relatif number of corners ocal maima f thresh 8 neighbourhood f f
Harris Detector: Steps
Compute corner response R Harris Detector: Steps
Harris Detector: Steps Find points with large corner response: R>threshold
Harris Detector: Steps Tae onl the points of local maima of R
Harris Detector: Steps
Harris detector: Summar of steps 1. Compute Gaussian derivatives at each piel. Compute second moment matri A in a Gaussian window around each piel 3. Compute corner response function R 4. Threshold R 5. Find local maima of response function non-maimum suppression
Harris - invariance to transformations Geometric transformations translation rotation similitude ilit rotation ti + scale change affine valid for local planar objects Photometric transformations Affine intensit changes a + b
Harris Detector: nvariance Properties Rotation Ellipse rotates but its shape i.e. eigenvalues remains the same Corner response R is invariant to image rotation
Harris Detector: nvariance Properties Scaling Corner All points will be classified as edges Not invariant to scaling
Harris Detector: nvariance Properties Affine intensit change Onl derivatives are used => invariance to intensit shift + b ntensit scale: a R threshold R image coordinate image coordinate ffi i i h Partiall invariant to affine intensit change dependent on tpe of threshold
Comparison of patches - SSD Comparison of the intensities in the neighborhood of two interest points 1 1 image 1 image SSD : sum of square difference 1 i j i j N 1 N N i N jn 1 1 1 Small difference values similar patches
Cross-correlation ZNCC Cross correlation ZNCC ZNCC: zero normalized cross correlation 1 1 1 1 1 1 m j i m j i N N N 1 1 N i N j N ZNCC values between 1 and 1 1 when identical patches ZNCC values between -1 and 1 1 when identical patches in practice threshold around 0.5 Robust to illumination change -> a+b
ocal descriptors Piel values Grevalue derivatives Differential invariants i [Koenderin 87] SFT descriptor [owe 99]
ocal descriptors ocal descriptors Grevalue derivatives G e a ue de at es Convolution with Gaussian derivatives G G * * * G G G v * * G G d d G G ep 1 G
ocal descriptors ocal descriptors Notation for grevalue derivatives [Koenderin 87] G Notation for grevalue derivatives [Koenderin 87] * G G * * G G v * G i? nvariance?
ocal descriptors rotation invariance ocal descriptors rotation invariance i t i t ti diff ti l i i t nvariance to image rotation : differential invariants [Koen87] gradient magnitude aplacian
aplacian of Gaussian OG OG G G
SFT descriptor [owe 99] Approach 8 orientations of the gradient 44 spatial grid Dimension 18 soft-assignment to spatial bins normalization of the descriptor to norm one comparison with Euclidean distance image patch gradient 3D histogram
ocal descriptors - rotation invariance Estimation of the dominant orientation ti etract gradient orientation histogram over gradient orientation pea in this histogram 0 Rotate patch in dominant direction
ocal descriptors illumination change Robustness to illumination changes in case of an affine transformation 1 a b Normalization of the image patch with mean and variance
nvariance to scale changes nvariance to scale changes Scale change between two images Scale change between two images Scale factor s can be eliminated Support region for calculation!! n case of a convolution with Gaussian derivatives defined b n case of a convolution with Gaussian derivatives defined b d d G G ep 1 G d d G G
Overview ntroduction to local features Harris interest t points + SSD ZNCC SFT Scale & affine invariant interest point detectors
Scale invariance - motivation Description regions have to be adapted to scale changes nterest t points have to be repeatable for scale changes
Harris detector + scale changes Repeatabilit rate { ai bi dist H ai b R ma a b i i i }
Scale adaptation Scale adaptation Scale change bet een t o images Scale change between two images 1 1 1 1 1 s s Scale adapted derivative calculation
Scale adaptation Scale adaptation Scale change bet een t o images Scale change between two images 1 1 1 1 1 s s Scale adapted derivative calculation 1 1 1 1 1 s G s G n n i i n i i s n s
Harris detector adaptation to scale R { a b dist H a b } i i i i
Scale selection For a point compute a value gradient aplacian etc. at several scales Normalization ation of the values with the scale factor e.g. aplacian s Select scale s at the maimum characteristic scale s scale Ep. results show that t the aplacian gives best results
Scale selection Scale invariance of the characteristic scale s norm. ap. scale
Scale selection Scale invariance of the characteristic scale s norm. a p. norm. a p. scale scale Relation between characteristic scales s s 1 s
Scale-invariant detectors Harris-aplace Miolajcz & Schmid 01 aplacian detector indeberg 98 Difference of Gaussian owe 99 Harris-aplace aplacian
Harris-aplace multi-scale Harris points selection of points at maimum of aplacian invariant points + associated regions [Miolajcz & Schmid 01]
Matching results 13 / 190 detected interest points
Matching results 58 points are initiall matched
Matching results 3 points are matched after verification all correct
OG detector Convolve image with scalenormalized aplacian at several scales OG s G G Detection of maima and minima of aplacian in scale space
Efficient implementation Difference of Gaussian DOG approimates the aplacian DOG G G Error due to the approimation
DOG detector Fast computation scale space processed one octave at a time David G. owe. "Distinctive image features from scale-invariant epoints. JCV 60 student presentation
Affine invariant regions - Motivation Scale invariance is not sufficient for large baseline changes detected scale invariant region A j t d i i i t h l ll projected regions viewpoint changes can locall be approimated b an affine transformation A
Affine invariant regions - Motivation
Affine invariant regions - Eample
Harris/Hessian/aplacian-Affine nitialize with scale-invariant Harris/Hessian/aplacian points Estimation of the affine neighbourhood with the second moment matri [indeberg 94] Appl affine neighbourhood estimation to the scale- invariant interest points [Miolajcz & Schmid 0 Schmid0 Schaffalitz & Zisserman 0] Ecellent results in a comparison [Miolajcz et al. 05]
Affine invariant regions Affine invariant regions Based on the second moment matri indeberg 94 Based on the second moment matri indeberg 94 D D G M D D D D D D G M Normalization with eigenvalues/eigenvectors 1 M
Affine invariant regions A R M 1 R M 1 R R R R sotropic neighborhoods related b image rotation
Affine invariant regions - Estimation terative estimation initial points
Affine invariant regions - Estimation terative estimation iteration #1
Affine invariant regions - Estimation terative estimation iteration #
Harris-Affine versus Harris-aplace Harris-Affine Harris-aplace
Harris/Hessian-Affine Harris-Affine Hessian-Affine
Harris-Affine
Hessian-Affine
Matches correct matches
Matches 33 correct matches
Maimall stable etremal regions MSER [Matas 0] Etremal regions: connected components in a thresholded image all piels above/below a threshold Maimall stable: minimal change of the component area for a change of the threshold i.e. region remains stable for a change of threshold Ecellent results in a recent comparison
Maimall stable etremal regions MSER Eamples of thresholded h d images high threshold low threshold
MSER