Instance-level recognition: Local invariant features. Cordelia Schmid INRIA, Grenoble

Transcription

1 nstance-level recognition: ocal invariant features Cordelia Schmid NRA Grenoble

2 Overview ntroduction to local features Harris interest points + SSD ZNCC SFT Scale & affine invariant interest point detectors Evaluation and comparison of different detectors Region descriptors and their performance

3 ocal features local descriptor Several / man local descriptors per image Robust to occlusion/clutter + no object segmentation required Photometric : distinctive nvariant : to image transformations + illumination changes

4 ocal features: interest points

5 ocal features: Contours/segments

6 ocal features: segmentation

7 Application: Matching Find corresponding locations in the image

8 llustration Matching nterest points etracted with Harris detector ~ 500 points

9 llustration Matching Matching nterest points matched based on cross-correlation 188 pairs

10 llustration Global constraints Matching Global constraint - Robust estimation of the fundamental matri 99 inliers 89 outliers

11 Application: Panorama stitching mages courtes of A. Zisserman.

12 Application: nstance-level recognition Search for particular objects and scenes in large databases

13 nstance-level recognition: Approach mage content is transformed into local features invariant to geometric and photometric transformations Matching local invariant descriptors ocal Features e.g. SFT K. Grauman B. eibe Slide credit: David owe 13

14 Difficulties Finding the object despite possibl large changes in scale viewpoint lighting and partial occlusion requires invariant description Scale Viewpoint ighting Occlusion

15 Difficulties Ver large images collection need for efficient indeing Flickr has billion photographs more than 1 million added dail Facebook has 15 billion images ~7 million added dail arge personal collections Video collections i.e. YouTube

16 Applications Take a picture of a product or advertisement find relevant information on the web [Piee Milpi]

17 Applications Cop detection for images and videos Quer video Search in 00h of video

18 Applications Son Aibo Robotics Recognize docking station Place recognition oop closure in SAM K. Grauman B. eibe Slide credit: David owe 18

19 ocal features - histor ine segments [owe 87 Aache 90] nterest points & cross correlation [Z. Zhang et al. 95] Rotation invariance with differential invariants [Schmid&Mohr 96] Scale & affine invariant detectors [indeberg 98 owe 99 Tutelaars&VanGool 00 Mikolajczk&Schmid 0 Matas et al. 0] Dense detectors and descriptors [eung&malik 99 Fei-Fei& Perona 05 azebnik et al. 06] Contour and region segmentation descriptors [Shotton et al. 05 Opelt et al. 06 Ferrari et al. 06 eordeanu et al. 07]

20 ocal features 1 Etraction of local features Contours/segments nterest points & regions Regions b segmentation Dense features points on a regular grid Description of local features Dependant on the feature tpe Contours/segments angles length ratios nterest points grelevels gradient histograms Regions segmentation teture + color distributions

21 ine matching Etraction de contours Zero crossing of aplacian ocal maima of gradients Chain contour points hsteresis Etraction of line segments Description of segments Mi-point length orientation angle between pairs etc.

22 Eperimental results line segments images

23 Eperimental results line segments 48 / 1 line segments etracted

24 Eperimental results line segments 89 matched line segments - 100% correct

25 Eperimental results line segments 3D reconstruction

26 Problems of line segments Often onl partial etraction ine segments broken into parts Missing parts nformation not ver discriminative 1D information Similar for man segments Potential solutions Pairs and triplets of segments nterest points

27 Overview ntroduction to local features Harris interest points + SSD ZNCC SFT Scale & affine invariant interest point detectors Evaluation and comparison of different detectors Region descriptors and their performance

28 Harris detector [Harris & Stephens 88] Based on the idea of auto-correlation mportant difference in all directions => interest point

29 Harris detector Auto-correlation function for a point and a shift A = k k k + k + k k W W

30 Harris detector Auto-correlation function for a point and a shift A = k k k + k + k k W W { A small in all directions large in one directions large in all directions uniform region contour interest point

31 Harris detector

32 Harris detector Discret shifts are avoided based on the auto-correlation matri + = + + with first order approimation = W k k k k k k + = + + k k k k k k k k A k k k W k k k + + =

33 Harris detector = W k k W k k k k W k k k k W k k k k k k k k k k Auto-correlation matri the sum can be smoothed with a Gaussian = G

34 Harris detector Auto-correlation matri A G = captures the structure of the local neighborhood measure based on eigenvalues of this matri strong eigenvalues 1 strong eigenvalue 0 eigenvalue => interest point => contour => uniform region

35 nterpreting the eigenvalues Classification of image points using eigenvalues of autocorrelation matri: λ Edge λ >> λ 1 Corner λ 1 and λ are large λ 1 ~ λ ; \ λ 1 and λ are small; Flat region Edge λ 1 >> λ λ 1

36 R Corner response function = det A A α: constant 0.04 to 0.06 α trace = λ1λ α λ1 + Edge R < 0 Corner R > 0 λ Flat region R small Edge R < 0

37 Harris detector Cornerness function f = det A k trace A = 1λ k λ λ + λ 1 Reduces the effect of a strong contour nterest point detection Treshold absolut relatif number of corners ocal maima f > thresh 8 neighbourhood f f

38 Harris Detector: Steps

39 Compute corner response R Harris Detector: Steps

40 Harris Detector: Steps Find points with large corner response: R>threshold

41 Harris Detector: Steps Take onl the points of local maima of R

42 Harris Detector: Steps

43 Harris detector: Summar of steps 1. Compute Gaussian derivatives at each piel. Compute second moment matri A in a Gaussian window around each piel 3. Compute corner response function R 4. Threshold R 5. Find local maima of response function non-maimum suppression

44 Harris - invariance to transformations Geometric transformations translation rotation similitude rotation + scale change affine valide for local planar objects Photometric transformations Affine intensit changes a + b

45 Harris Detector: nvariance Properties Rotation Ellipse rotates but its shape i.e. eigenvalues remains the same Corner response R is invariant to image rotation

46 Harris Detector: nvariance Properties Affine intensit change Onl derivatives are used => invariance to intensit shift + b ntensit scale: a R threshold R image coordinate image coordinate Partiall invariant to affine intensit change dependent on tpe of threshold

47 Harris Detector: nvariance Properties Scaling Corner All points will be classified as edges Not invariant to scaling

48 Comparison of patches - SSD Comparison of the intensities in the neighborhood of two interest points 1 1 image 1 image SSD : sum of square difference N N + i + j + i + N i= N j= N 1 j Small difference values similar patches

49 Comparison of patches j i j i N N i N N j N = = + SSD : nvariance to photometric transformations? ntensit changes + b => Normalizing with the mean of each patch ntensit changes a + b m j i m j i N N i N N j N = = + => Normalizing with the mean of each patch = = N N i N N j N m j i m j i σ σ => Normalizing with the mean and standard deviation of each patch

50 Cross-correlation ZNCC = = N N i N N j N m j i m j i σ σ zero normalized SSD ZNCC: zero normalized cross correlation = = σ σ m j i m j i N N i N N j N ZNCC values between -1 and 1 1 when identical patches in practice threshold around 0.5

51 ocal descriptors Grevalue derivatives Differential invariants [Koenderink 87] SFT descriptor [owe 99]

52 Grevalue derivatives: mage gradient The gradient of an image: The gradient points in the direction of most rapid increase in intensit The gradient direction is given b how does this relate to the direction of the edge? The edge strength is given b the gradient magnitude

53 Differentiation and convolution Recall for D function f: f = lim ε 0 f + ε ε f ε We could approimate this as f f n+1 f n Convolution with the filter -1 1

54 Finite difference filters Other approimations of derivative filters eist:

55 Effects of noise Consider a single row or column of the image Plotting intensit as a function of position gives a signal Where is the edge?

56 Solution: smooth first f g f * g d d f g To find edges look for peaks in d f g d

57 Derivative theorem of convolution Differentiation is convolution and convolution is associative: d d f g = d f g d This saves us one operation: f d d g f d d g

58 ocal descriptors σ σ σ G G Grevalue derivatives Convolution with Gaussian derivatives ep 1 σ πσ σ G + = = d d G G σ σ = M * * * * σ σ σ σ G G G G v

59 ocal descriptors * G G G σ σ σ Notation for grevalue derivatives [Koenderink 87] = = M M * * * * G G G G σ σ σ σ v nvariance?

60 ocal descriptors rotation invariance nvariance to image rotation : differential invariants [Koen87] gradient magnitude K K K K aplacian

61 aplacian of Gaussian OG OG = G σ + G σ

62 SFT descriptor [owe 99] Approach 8 orientations of the gradient 44 spatial grid Dimension 18 soft-assignment to spatial bins normalization of the descriptor to norm one comparison with Euclidean distance image patch gradient 3D histogram

63 ocal descriptors - rotation invariance Estimation of the dominant orientation etract gradient orientation histogram over gradient orientation peak in this histogram 0 π Rotate patch in dominant direction

64 ocal descriptors illumination change Robustness to illumination changes in case of an affine transformation = a + b 1

65 ocal descriptors illumination change Robustness to illumination changes in case of an affine transformation = a + b 1 Normalization of derivatives with gradient magnitude + +

66 ocal descriptors illumination change Robustness to illumination changes in case of an affine transformation = a + b 1 Normalization of derivatives with gradient magnitude + + Normalization of the image patch with mean and variance

67 nvariance to scale changes Scale change between two images Scale factor s can be eliminated Support region for calculation!! n case of a convolution with Gaussian derivatives defined b ep 1 σ πσ σ G + = = d d G G σ σ σ

68 Overview ntroduction to local features Harris interest points + SSD ZNCC SFT Scale & affine invariant interest point detectors Evaluation and comparison of different detectors Region descriptors and their performance

69 Scale invariance - motivation Description regions have to be adapted to scale changes nterest points have to be repeatable for scale changes

70 Harris detector + scale changes Repeatabilit rate R ε = { a i b i dist H ai b ma a b i i i < ε}

71 Scale adaptation Scale change between two images = = s s 1 1 Scale adapted derivative calculation

72 Scale adaptation = = s s Scale change between two images σ σ s G s G n n i i n i i K K = Scale adapted derivative calculation sσ n s

73 Scale adaptation σ i where are the derivatives with Gaussian convolution ~ σ σ σ σ σ G

74 Scale adaptation σ i where are the derivatives with Gaussian convolution ~ σ σ σ σ σ G ~ σ σ σ σ σ s s s s s G s Scale adapted auto-correlation matri

75 Harris detector adaptation to scale R ε { a b dist H a b < ε} = i i i i

76 Multi-scale matching algorithm s = 1 s = 3 s = 5

77 Multi-scale matching algorithm s = 1 8 matches

78 Multi-scale matching algorithm Robust estimation of a global affine transformation s = 1 3 matches

79 Multi-scale matching algorithm s = 1 3 matches s = 3 4 matches

80 Multi-scale matching algorithm s = 1 3 matches s = 3 4 matches highest number of matches correct scale s = 5 16 matches

81 Matching results Scale change of 5.7

82 Matching results 100% correct matches 13 matches

83 Scale selection We want to find the characteristic scale of the blob b convolving it with aplacians at several scales and looking for the maimum response However aplacian response decas as scale increases: original signal radius=8 increasing σ Wh does this happen?

84 Scale normalization The response of a derivative of Gaussian filter to a perfect step edge decreases as σ increases 1 σ π

85 Scale normalization The response of a derivative of Gaussian filter to a perfect step edge decreases as σ increases To keep response the same scale-invariant must multipl Gaussian derivative b σ aplacian is the second Gaussian derivative so it must be aplacian is the second Gaussian derivative so it must be multiplied b σ

86 Effect of scale normalization Original signal Unnormalized aplacian response Scale-normalized aplacian response maimum

87 Blob detection in D aplacian of Gaussian: Circularl smmetric operator for blob detection in D g = g + g

88 Blob detection in D aplacian of Gaussian: Circularl smmetric operator for blob detection in D Scale-normalized: g g g = σ + norm

89 Scale selection The D aplacian is given b / σ + σ + e up to scale For a binar circle of radius r the aplacian achieves a maimum at σ = r / r aplacian response image r / scale σ

90 Characteristic scale We define the characteristic scale as the scale that produces peak of aplacian response characteristic scale T. indeberg Feature detection with automatic scale selection. nternational Journal of Computer Vision 30 : pp

91 Scale selection For a point compute a value gradient aplacian etc. at several scales Normalization of the values with the scale factor e.g. aplacian s + Select scale s at the maimum characteristic scale s + scale Ep. results show that the aplacian gives best results

92 Scale selection Scale invariance of the characteristic scale s norm. ap. scale

93 Scale selection Scale invariance of the characteristic scale s norm. ap. norm. ap. scale scale Relation between characteristic scales s s 1 = s

94 Scale-invariant detectors Harris-aplace Mikolajczk & Schmid 01 aplacian detector indeberg 98 Difference of Gaussian owe 99 Harris-aplace aplacian

95 Harris-aplace multi-scale Harris points selection of points at maimum of aplacian invariant points + associated regions [Mikolajczk & Schmid 01]

96 Matching results 13 / 190 detected interest points

97 Matching results 58 points are initiall matched

98 Matching results 3 points are matched after verification all correct

99 OG detector Convolve image with scalenormalized aplacian at several scales OG = s G σ + G σ Detection of maima and minima of aplacian in scale space

100 Hessian detector Hessian matri H = Determinant of Hessian matri DET = Penalizes/eliminates long structures with small derivative in a single direction

101 Efficient implementation Difference of Gaussian DOG approimates the aplacian DOG = G kσ G σ Error due to the approimation

102 DOG detector Fast computation scale space processed one octave at a time David G. owe. "Distinctive image features from scale-invariant kepoints. JCV 60.