Instance-level l recognition. Cordelia Schmid INRIA

Transcription

1 nstance-level l recognition Cordelia Schmid NRA

2 nstance-level recognition Particular objects and scenes large databases

3 Application Search photos on the web for particular places Find these landmars...in these images and M more

4 Applications Tae a picture of a product or advertisement find relevant information on the web [Google Goggles Milpi Piee]

5 Applications Cop detection for images and videos Quer video Search in 00h of video

6 Difficulties Find the object despite large changes in scale viewpoint lighting crop and occlusion not much teture/structure requires local invariant descriptors Scale Viewpoint ighting Occlusion

7 Difficulties Ver large images collection need for efficient indeing Flicr has billions photographs more than million added dail Faceboo has 5 billions images ~7 million added dail arge personal collections Video collections with a large number of videos i.e. YouTube

8 Approach: matching local invariant descriptors mage content is transformed into local features that are invariant to geometric and photometric transformations ocal Features e.g. SFT K. Grauman B. eibe [owe04] Slide credit: David owe 8

9 Overview ocal invariant features Matching and recognition with local features Efficient visual search Ver large scale search

10 ocal features local descriptor Several / man local descriptors per image Robust to occlusion/clutter + no object segmentation required Photometric : distinctive nvariant : to image transformations + illumination changes

11 ocal features: interest points

12 ocal features: Contours/segments

13 ocal features: segmentation

14 ocal features Etraction of local features Contours/segments nterest points & regions Regions b segmentation Dense features points on a regular grid Description of local features Dependant on the feature tpe Contours/segments angles length ratios nterest t points grelevels l gradient histograms Regions segmentation teture + color distributions

15 ine matching Etraction de contours Zero crossing of aplacian ocal maima of gradients Chain contour points hsteresis Etraction of line segments Description of segments Mi-point length orientation angle between pairs etc.

16 Eperimental results line segments images

17 Eperimental results line segments 48 / line segments etracted

18 Eperimental results line segments 89 matched line segments - 00% correct

19 Eperimental results line segments 3D reconstruction

20 Problems of line segments Often onl partial etraction ine segments broen into parts Missing parts nformation not ver discriminative D information Similar for man segments Potential solutions Pairs and triplets of segments nterest points

21 Eample results - interest points nterest t points etracted t with Harris detector t ~ 500 points

22 Matching interest points Find corresponding locations in the image

23 Matching Matching interest points nterest t points matched based on cross-correlation 88 pairs

24 Matching constrai interest points Global constraint - Robust estimation of the fundamental matri 99 inliers 89 outliers

25 Application: Panorama stitching mages courtes of A. Zisserman.

26 Overview Harris interest points + SSD ZNCC SFT Scale & affine invariant interest point detectors Evaluation and comparison of different detectors Region descriptors and their performance

27 Harris detector [Harris & Stephens 88] Based on the idea of auto-correlation ti mportant t difference in all directions => interest t point

28 Harris detector Auto-correlation function for a point and a shift A W W

29 Harris detector Auto-correlation function for a point and a shift A W A { W small in all directions uniform region { large in one directions contour large in all directions interest point

30 Harris detector

31 Harris detector Harris detector Discret shifts are avoided based on the auto-correlation matri with first order approimation A W W W

32 Harris detector Harris detector W W W W W W Auto-correlation matri the sum can be smoothed with a Gaussian the sum can be smoothed with a Gaussian G

33 Harris detector Auto-correlation matri A G captures the structure of the local neighborhood measure based on eigenvalues of this matri strong eigenvalues => interest point strong eigenvalue => contour 0 eigenvalue => uniform region

34 nterpreting the eigenvalues Classification of image points using eigenvalues of autocorrelation matri: Edge >> Corner and are large ~ ; \ and are small; Flat region Edge >>

35 R Corner response function det A trace A α: constant 0.04 to 0.06 Edge R <0 Corner R > 0 Flat region R small Edge R < 0

36 Harris detector Cornerness function f det A trace A Reduces the effect of a strong contour nterest point detection Treshold absolut relatif number of corners ocal maima f thresh 8 neighbourhood f f

37 Harris Detector: Steps

38 Compute corner response R Harris Detector: Steps

39 Harris Detector: Steps Find points with large corner response: R>threshold

40 Harris Detector: Steps Tae onl the points of local maima of R

41 Harris Detector: Steps

42 Harris detector: Summar of steps. Compute Gaussian derivatives at each piel. Compute second moment matri A in a Gaussian window around each piel 3. Compute corner response function R 4. Threshold R 5. Find local maima of response function non-maimum suppression

43 Harris - invariance to transformations Geometric transformations translation rotation similitude ilit rotation ti + scale change affine valide for local planar objects Photometric transformations Affine intensit changes a + b

44 Harris Detector: nvariance Properties Rotation Ellipse rotates but its shape i.e. eigenvalues remains the same Corner response R is invariant to image rotation

45 Harris Detector: nvariance Properties Affine intensit change Onl derivatives are used => invariance to intensit shift + b ntensit scale: a R threshold R image coordinate image coordinate ffi i i h Partiall invariant to affine intensit change dependent on tpe of threshold

46 Harris Detector: nvariance Properties Scaling Corner All points will be classified as edges Not invariant to scaling

47 Comparison of patches - SSD Comparison of the intensities in the neighborhood of two interest points image image SSD : sum of square difference i j i j N N N i N jn Small difference values similar patches

48 Comparison of patches Comparison of patches j i j i N N SSD j i j i N i N j N SSD : nvariance to photometric transformations? nvariance to photometric transformations? ntensit changes + b > N li i ith th f h t h m j i m j i N N i N N j N => Normalizing with the mean of each patch ntensit changes a + b N i N j N N m j i m j i => Normalizing with the mean and standard deviation of each patch N i N j N m j i m j i

49 Cross-correlation ZNCC Cross correlation ZNCC li d SSD N N m j i m j i zero normalized SSD N i N j N ZNCC: zero normalized cross correlation m j i m j i N N N N i N j N ZNCC values between - and when identical patches ZNCC values between and when identical patches in practice threshold around 0.5

50 ntroduction to local descriptors Grevalue derivatives Differential invariants i [Koenderin 87] SFT descriptor [owe 99]

51 Grevalue derivatives: mage gradient The gradient of an image: The gradient points in the direction of most rapid increase in intensit The gradient direction is given b how does this relate to the direction of the edge? The edge strength is given b the gradient magnitude

52 Differentiation and convolution Recall for D function f: f lim f 0 f f We could approimate this as f f n f n Convolution with the filter -

53 Finite difference filters Other approimations of derivative filters eist:

54 Effects of noise Consider a single row or column of the image Plotting intensit as a function of position gives a signal Where is the edge?

55 Solution: smooth first f g f* g d d f g To find edges loo for peas in d f g d

56 Derivative theorem of convolution Differentiation is convolution and convolution is associative: d d f g d f g d This saves us one operation: f d d g d f d g

57 ocal descriptors ocal descriptors Grevalue derivatives G e a ue de at es Convolution with Gaussian derivatives G G * * * G G G v * * G G d d G G ep G

58 ocal descriptors ocal descriptors Notation for grevalue derivatives [Koenderin 87] G Notation for grevalue derivatives [Koenderin 87] * G G * * G G v * G i? nvariance?

59 ocal descriptors rotation invariance ocal descriptors rotation invariance i t i t ti diff ti l i i t nvariance to image rotation : differential invariants [Koen87] gradient magnitude aplacian

60 aplacian of Gaussian OG OG G G

61 SFT descriptor [owe 99] Approach 8 orientations of the gradient 44 spatial grid Dimension 8 soft-assignment to spatial bins normalization of the descriptor to norm one comparison with Euclidean distance image patch gradient 3D histogram

62 ocal descriptors - rotation invariance Estimation of the dominant orientation ti etract gradient orientation histogram over gradient orientation pea in this histogram 0 Rotate patch in dominant direction

63 ocal descriptors illumination change Robustness to illumination changes in case of an affine transformation a b

64 ocal descriptors illumination change Robustness to illumination changes in case of an affine transformation a b Normalization of derivatives with gradient magnitude

65 ocal descriptors illumination change Robustness to illumination changes in case of an affine transformation a b Normalization of derivatives with gradient magnitude Normalization of the image patch with mean and variance

66 nvariance to scale changes nvariance to scale changes Scale change between two images Scale change between two images Scale factor s can be eliminated Support region for calculation!! n case of a convolution with Gaussian derivatives defined b n case of a convolution with Gaussian derivatives defined b d d G G ep G d d G G