Instance-level recognition: Local invariant features. Cordelia Schmid INRIA, Grenoble

Transcription

1 nstance-level recognition: ocal invariant features Cordelia Schmid NRA Grenoble

2 Overview ntroduction to local features Harris interest t points + SSD ZNCC SFT Scale & affine invariant interest point detectors

3 ocal features local descriptor Several / man local descriptors per image Robust to occlusion/clutter + no object segmentation required Photometric : distinctive nvariant : to image transformations + illumination changes

4 ocal features nterest Points Contours/lines Region segments

5 ocal features nterest Points Contours/lines Region segments Patch descriptors i.e. SFT Mi-points angles Color/teture histogram

6 nterest points / invariant regions Harris detector Scale/affine inv. detector presented in this lecture

7 Contours / lines Etraction de contours Zero crossing of aplacian ocal maima of gradients Chain contour points hsteresis Cann detector Recent detectors Global probabilit of boundar gpb detector [Mali et al. UC Berele] Structured forests for fast edge detection SED [Dollar and Zitnic] student presentation

8 Regions segments / superpiels Simple linear iterative clustering SC Normalized cut [Shi & Mali] Mean Shift [Comaniciu & Meer].

9 Application: matching Find corresponding locations in the image

10 llustration Matching nterest t points etracted t with Harris detector t ~ 500 points

11 llustration Matching Matching nterest t points matched based on cross-correlation 188 pairs

12 llustration Global constraints Matching Global constraint - Robust estimation of the fundamental matri 99 inliers 89 outliers

13 Application: Panorama stitching mages courtes of A. Zisserman.

14 Application: nstance-level recognition Search for particular objects and scenes in large databases

15 Difficulties Finding the object despite possibl large changes in scale viewpoint lighting and partial occlusion requires invariant description Scale Viewpoint ighting Occlusion

16 Difficulties Ver large images collection need for efficient i indeing i Flicr has billion photographs h more than 1 million added d dail Faceboo has 15 billion images ~7 million added dail arge personal collections Video collections i.e. YouTube

17 Applications Search photos on the web for particular places Find these landmars...in these images and 1M more

18 Applications Tae a picture of a product or advertisement find relevant information on the web

19 Applications Cop detection for images and videos Quer video Search in 00h of video

20 Overview ntroduction to local features Harris interest t points + SSD ZNCC SFT Scale & affine invariant interest point detectors

21 Harris detector [Harris & Stephens 88] Based on the idea of auto-correlation ti mportant t difference in all directions => interest t point

22 Harris detector Auto-correlation function for a point and a shift A W W

23 Harris detector Auto-correlation function for a point and a shift A W A { W small in all directions uniform region { large in one directions contour large in all directions interest point

24 Harris detector Harris detector Discret shifts are avoided based on the auto-correlation matri with first order approimation A W W W

25 Harris detector Harris detector W W W W W W Auto-correlation matri the sum can be smoothed with a Gaussian the sum can be smoothed with a Gaussian G

26 Harris detector Auto-correlation matri A G captures the structure of the local neighborhood measure based on eigenvalues of this matri strong eigenvalues => interest point 1 strong eigenvalue => contour 0 eigenvalue => uniform region

27 nterpreting the eigenvalues Classification of image points using eigenvalues of autocorrelation matri: Edge >> 1 Corner 1 and are large 1 ~ ; \ 1 and are small; Flat region Edge 1 >> 1

28 R Corner response function det A trace A α: constant 0.04 to 0.06 Edge R <0 1 1 Corner R > 0 Flat region R small Edge R < 0

29 Harris detector Cornerness function R det A trace A 1 1 Reduces the effect of a strong contour nterest point detection Treshold absolut relatif number of corners ocal maima f thresh 8 neighbourhood f f

30 Harris Detector: Steps

31 Compute corner response R Harris Detector: Steps

32 Harris Detector: Steps Find points with large corner response: R>threshold

33 Harris Detector: Steps Tae onl the points of local maima of R

34 Harris Detector: Steps

35 Harris detector: Summar of steps 1. Compute Gaussian derivatives at each piel. Compute second moment matri A in a Gaussian window around each piel 3. Compute corner response function R 4. Threshold R 5. Find local maima of response function non-maimum suppression

36 Harris - invariance to transformations Geometric transformations translation rotation similitude ilit rotation ti + scale change affine valid for local planar objects Photometric transformations Affine intensit changes a + b

37 Harris Detector: nvariance Properties Rotation Ellipse rotates but its shape i.e. eigenvalues remains the same Corner response R is invariant to image rotation

38 Harris Detector: nvariance Properties Scaling Corner All points will be classified as edges Not invariant to scaling

39 Harris Detector: nvariance Properties Affine intensit change Onl derivatives are used => invariance to intensit shift + b ntensit scale: a R threshold R image coordinate image coordinate ffi i i h Partiall invariant to affine intensit change dependent on tpe of threshold

40 Comparison of patches - SSD Comparison of the intensities in the neighborhood of two interest points 1 1 image 1 image SSD : sum of square difference 1 i j i j N 1 N N i N jn Small difference values similar patches

41 Cross-correlation ZNCC Cross correlation ZNCC ZNCC: zero normalized cross correlation m j i m j i N N N 1 1 N i N j N ZNCC values between 1 and 1 1 when identical patches ZNCC values between -1 and 1 1 when identical patches in practice threshold around 0.5 Robust to illumination change -> a+b

42 ocal descriptors Piel values Grevalue derivatives Differential invariants i [Koenderin 87] SFT descriptor [owe 99]

43 ocal descriptors ocal descriptors Grevalue derivatives G e a ue de at es Convolution with Gaussian derivatives G G * * * G G G v * * G G d d G G ep 1 G

44 ocal descriptors ocal descriptors Notation for grevalue derivatives [Koenderin 87] G Notation for grevalue derivatives [Koenderin 87] * G G * * G G v * G i? nvariance?

45 ocal descriptors rotation invariance ocal descriptors rotation invariance i t i t ti diff ti l i i t nvariance to image rotation : differential invariants [Koen87] gradient magnitude aplacian

46 aplacian of Gaussian OG OG G G

47 SFT descriptor [owe 99] Approach 8 orientations of the gradient 44 spatial grid Dimension 18 soft-assignment to spatial bins normalization of the descriptor to norm one comparison with Euclidean distance image patch gradient 3D histogram

48 ocal descriptors - rotation invariance Estimation of the dominant orientation ti etract gradient orientation histogram over gradient orientation pea in this histogram 0 Rotate patch in dominant direction

49 ocal descriptors illumination change Robustness to illumination changes in case of an affine transformation 1 a b Normalization of the image patch with mean and variance

50 nvariance to scale changes nvariance to scale changes Scale change between two images Scale change between two images Scale factor s can be eliminated Support region for calculation!! n case of a convolution with Gaussian derivatives defined b n case of a convolution with Gaussian derivatives defined b d d G G ep 1 G d d G G

51 Overview ntroduction to local features Harris interest t points + SSD ZNCC SFT Scale & affine invariant interest point detectors

52 Scale invariance - motivation Description regions have to be adapted to scale changes nterest t points have to be repeatable for scale changes

53 Harris detector + scale changes Repeatabilit rate { ai bi dist H ai b R ma a b i i i }

54 Scale adaptation Scale adaptation Scale change bet een t o images Scale change between two images s s Scale adapted derivative calculation

55 Scale adaptation Scale adaptation Scale change bet een t o images Scale change between two images s s Scale adapted derivative calculation s G s G n n i i n i i s n s

56 Harris detector adaptation to scale R { a b dist H a b } i i i i

57 Scale selection For a point compute a value gradient aplacian etc. at several scales Normalization ation of the values with the scale factor e.g. aplacian s Select scale s at the maimum characteristic scale s scale Ep. results show that t the aplacian gives best results

58 Scale selection Scale invariance of the characteristic scale s norm. ap. scale

59 Scale selection Scale invariance of the characteristic scale s norm. a p. norm. a p. scale scale Relation between characteristic scales s s 1 s

60 Scale-invariant detectors Harris-aplace Miolajcz & Schmid 01 aplacian detector indeberg 98 Difference of Gaussian owe 99 Harris-aplace aplacian

61 Harris-aplace multi-scale Harris points selection of points at maimum of aplacian invariant points + associated regions [Miolajcz & Schmid 01]

62 Matching results 13 / 190 detected interest points

63 Matching results 58 points are initiall matched

64 Matching results 3 points are matched after verification all correct

65 OG detector Convolve image with scalenormalized aplacian at several scales OG s G G Detection of maima and minima of aplacian in scale space

66 Efficient implementation Difference of Gaussian DOG approimates the aplacian DOG G G Error due to the approimation

67 DOG detector Fast computation scale space processed one octave at a time David G. owe. "Distinctive image features from scale-invariant epoints. JCV 60 student presentation

68 Affine invariant regions - Motivation Scale invariance is not sufficient for large baseline changes detected scale invariant region A j t d i i i t h l ll projected regions viewpoint changes can locall be approimated b an affine transformation A

69 Affine invariant regions - Motivation

70 Affine invariant regions - Eample

71 Harris/Hessian/aplacian-Affine nitialize with scale-invariant Harris/Hessian/aplacian points Estimation of the affine neighbourhood with the second moment matri [indeberg 94] Appl affine neighbourhood estimation to the scale- invariant interest points [Miolajcz & Schmid 0 Schmid0 Schaffalitz & Zisserman 0] Ecellent results in a comparison [Miolajcz et al. 05]

72 Affine invariant regions Affine invariant regions Based on the second moment matri indeberg 94 Based on the second moment matri indeberg 94 D D G M D D D D D D G M Normalization with eigenvalues/eigenvectors 1 M

73 Affine invariant regions A R M 1 R M 1 R R R R sotropic neighborhoods related b image rotation

74 Affine invariant regions - Estimation terative estimation initial points

75 Affine invariant regions - Estimation terative estimation iteration #1

76 Affine invariant regions - Estimation terative estimation iteration #

77 Harris-Affine versus Harris-aplace Harris-Affine Harris-aplace

78 Harris/Hessian-Affine Harris-Affine Hessian-Affine

79 Harris-Affine

80 Hessian-Affine

81 Matches correct matches

82 Matches 33 correct matches

83 Maimall stable etremal regions MSER [Matas 0] Etremal regions: connected components in a thresholded image all piels above/below a threshold Maimall stable: minimal change of the component area for a change of the threshold i.e. region remains stable for a change of threshold Ecellent results in a recent comparison

84 Maimall stable etremal regions MSER Eamples of thresholded h d images high threshold low threshold

85 MSER