Blobs & Scale Invariance

Transcription

1 Blobs & Scale Invariance Prof. Didier Stricker Doz. Gabriele Bleser Computer Vision: Object and People Tracking With slides from Bebis, S. Lazebnik & S. Seitz, D. Lowe, A. Efros 1

2 Apertizer: some videos (results from the research group Augmented Vision ) Illustration of ifficulties of tracking in videos: Video foot Video ball Video Volleyball Artefacts

3 Reminder: Harris Detector - Steps Compute Gaussian derivatives at each pixel Compute second moment matrix H in a Gaussian window around each pixel Recall: f ( xi, yi) x f ( xi, yi) y G( x, y, D)* f ( xi, yi ) x G( x, y, D)* f ( xi, yi ) y σ D is called the differentiation scale 3

4 Harris Detector (cont d) Using a window function w(x,y), we rewrite E : w(x,y) : Then H becomes: H A H W ( x, y) 1 in window, 0 outside fx fx f y xw, yw f x f y f y HA W w( x, y) f x w( x, y) f x f y fx fx f y x, y x, y w( x, y) xy, f x f y f y w( x, y) fx f y w( x, y) f y x, y x, y 4

5 Harris Detector (cont d) Harris uses a Gaussian window: w(x,y)=g(x,y,σ I ) where σ I is called the integration scale: w(x,y) : Gaussian HA W w( x, y) f x w( x, y) f x f y fx fx f y x, y x, y w( x, y) xy, f x f y f y w( x, y) fx f y w( x, y) f y x, y x, y 5

6 Harris Detector - Example H H H 6

7 Harris Detector Scale Parameters The Harris detector requires two scale parameters: (i) a differentiation scale σ D for smoothing prior to the computation of image derivatives, & (ii) an integration scale σ I for defining the size of the Gaussian window (i.e., integrating derivative responses). H W (x,y) H W (x,y,σ I,σ D ) Typically, σ I =γσ D 7

8 Invariance to Geometric/Photometric Changes Is the Harris detector invariant to geometric and photometric changes? Rotation Scale Affine Linear intensity change: I(x,y) a I(x,y) + b 8

9 Harris Detector: Rotation Invariance Rotation Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation 9

10 Harris Detector: Rotation Invariance (cont d) 10

11 Harris Detector: Photometric Changes Linear intensity change: - Only derivatives are used => invariance to intensity shift I(x,y) I (x,y) + b - Intensity scale: I(x,y) a I(x,y) R threshold R x (image coordinate) x (image coordinate) Partially invariant to affine intensity change 11

12 Harris Detector: Scale Invariance Scaling Corner All points will be classified as edges rather than corners. Not invariant to scaling (and affine transforms) 1

13 Harris Detector: Repeatability 13

14 Harris Detector: Repeatability (cont d) 14

15 Harris Detector: Repeatability (cont d) When two points correspond? 15

16 Harris Detector: Repeatability (cont d) How do we find correspondences? 16

17 Harris Detector: Repeatability (cont d) In a comparative study of different interest point detectors, Harris was shown to be the most repeatable. (σ I =1, σ D =) C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of Interest Point Detectors", International Journal of Computer Vision 37(), ,

18 Harris Detector: Disadvantages Sensitive to: Scale change Significant viewpoint change Significant contrast change 18

19 How can we handle scale changes? H W (x,y,σ I,σ D ) must be adapted to scale changes. If scale change is known, adapt the Harris detector to it by properly setting σ I and σ D. If scale change is unknown, detect interest points at multiple scales. 19

20 Multi-scale Harris Detector Detects interest points at varying scales. R(H W ) = det(h W (x,y,σ I,σ D )) α trace (H W (x,y,σ I,σ D )) scale σ n =k n σ σ D = σ n σ I =γσ D σ n y Harris x 0

21 Multi-scale Harris Detector (cont d) Interest points detected at varying scales: M. Brown, R. Szeliski, and S. Winder, Multi-image matching using multi-scale oriented Patches, IEEE Conference on Computer Vision and Pattern Recognition, vol. I, pages ,

22 Multi-scale Harris Detector (cont d) The same interest point will be detected at multiple consecutive scales. Interest point location will shift as scale increases (i.e., due to smoothing). The size of each circle corresponds to the scale at which the interest point was detected.

23 How do we match them? Corresponding features might appear at different scales. How do we determine these scales? Need a scale selection mechanism! 3

24 Scale Selection: Characteristic Scale Scale selection by finding the characteristic scale of each feature. This the scale revealing the spatial extent of an interest point. characteristic scale characteristic scale 4

25 Scale Selection: Characteristic Scale (cont d) Only a subset of interest points are selected using the characteristic scale of each feature matching can be simplified. The size of the circles is related to the scale at which the interest points were selected. 5

26 Automatic Scale Selection Design a function F(x,σ n ) which provides some local measure. Select points at which F(x,σ n ) is maximal over σ n. F(x,σ n ) max of F(x,σ n ) corresponds to characteristic scale! σ n T. Lindeberg, "Feature detection with automatic scale selection" International Journal of Computer Vision, vol. 30, no., pp ,

27 Lindeberg et al, 1996 Slide from Tinne Tuytelaars 7

28 8

29 9

30 30

31 31

32 3

33 33

34 34

35 Automatic Scale Selection (cont d) Using characteristic scale, the spatial extent of interest points becomes covariant to scale transformations. The ratio σ 1 /σ reveals the scale factor between the images. σ 1 σ σ 1 /σ =.5 35

36 How should we choose F(x,σ n )? What local measure F(x,σ n ) should we use? Should be rotation invariant. Should have one stable sharp peak. 36

37 How should we choose F(x,σ n )? (cont d) Typically, F(x,σ n ) is defined using derivatives, e.g.: Square gradient L x L x LoG L x L x : ( x(, ) y(, )) : ( xx (, ) yy (, )) DoG : I( x)* G( ) I( x)* G( ) n1 Harris function A trace A : det( W) ( W) LoG yielded best results in a evaluation study; DoG was second best. n C. Schmid, R. Mohr, and C. Bauckhage, "Evaluation of Interest Point Detectors", International Journal of Computer Vision, 37(), pp ,

38 How should we choose F(x,σ n )? (cont d) Let s see how LoG responds at blobs 38

39 What about internal structure? Edges & Corners convey boundary information What about interior texture of the object? 39

40 Blob detection with scale selection 40

41 Recall: Edge detection f Edge d dx g Derivative of Gaussian f d dx g Edge = maximum of derivative 41 Source: S. Seitz

42 Edge detection, Take f Edge d dx g Second derivative of Gaussian (Laplacian) d dx f g Edge = zero crossing of second derivative 4 Source: S. Seitz

43 From edges to blobs Edge = ripple Blob = superposition of two ripples maximum Spatial selection: the magnitude of the Laplacian response will achieve a maximum at the center of the blob, provided the scale of the Laplacian is matched to the scale of the blob. 43

44 Scale selection We want to find the characteristic scale of the blob by convolving it with Laplacians at several scales and looking for the maximum response However, Laplacian response decays as scale increases: original signal (radius=8) increasing σ Why does this happen? 44

45 Scale normalization The response of a derivative of Gaussian filter to a perfect step edge decreases as σ increases 1 45

46 Scale normalization The response of a derivative of Gaussian filter to a perfect step edge decreases as σ increases To keep response the same (scale-invariant), must multiply Gaussian derivative by σ Laplacian is the second Gaussian derivative, so it must be multiplied by σ 46

47 Effect of scale normalization Original signal Unnormalized Laplacian response Scale-normalized Laplacian response maximum 47

48 Blob detection in D y g x g g Laplacian of Gaussian: Circularly symmetric operator for blob detection in D y x e y x 48

49 Blob detection in D Laplacian of Gaussian: Circularly symmetric operator for blob detection in D Scale-normalized: g g g norm x y 49

50 Scale selection At what scale does the Laplacian achieve a maximum response to a binary circle of radius r? r image Laplacian 50

51 LoG response Spatial extent of circle At what scale does the LoG achieve a maximum response for a binary circle of radius r? LoG is maximized at r / (characteristic scale) r image r / scale (σ) 51

52 Scale selection At what scale does the Laplacian achieve a maximum response to a binary circle of radius r? To get maximum response, the zeros of the Laplacian have to be aligned with the circle Zeros of Laplacian is given by (up to scale): Therefore, the maximum response occurs at x y 1 r /. 0 r image circle Laplacian 5

53 Characteristic scale We define the characteristic scale of a blob as the scale that produces peak of Laplacian response in the blob center Spatial extent r can be determined using: r characteristic scale T. Lindeberg (1998). "Feature detection with automatic scale selection." 53 International Journal of Computer Vision 30 (): pp

54 Scale-space blob detector 1. Convolve image with scale-normalized Laplacian at several scales. Find maxima of squared Laplacian response in scale-space 54

55 Scale-space blob detector: Example 55

56 Scale-space blob detector: Example 56

57 Scale-space blob detector: Example 57

58 Efficient implementation Approximating the Laplacian with a difference of Gaussians: L Gxx x y Gyy x y (,, ) (,, ) (Laplacian) DoG G( x, y, k) G( x, y, ) (Difference of Gaussians) 58

59 Recall: Implement Harris-Laplace using DoG LoG can be approximated by DoG: G( x, y, k ) G( x, y, ) ( k 1) G Note: DoG already incorporates the σ scale normalization factor. 59

60 Reminder: Implement Harris-Laplace using DoG Gaussian-blurred image L( x, y, ) G( x, y, )* I( x, y) Result using DoG: DoG( x, y, ) L( x, y, k ) L( x, y, ) - = 60

61 Scale-space Extrema Detection Extract local extrema (i.e., minima or maxima) in DoG pyramid. -Compare each point to its 8 neighbors at the same level, 9 neighbors in the level above, and 9 neighbors in the level below (i.e., 6 total). 61

62 Efficient implementation David G. Lowe. "Distinctive image features from scale-invariant keypoints. IJCV 60 (), pp , 004. Key-point detector and descriptor: SIFT (Scale Invariant Feature Transform) 6

63 Scale Invariant Detectors Experimental evaluation of detectors w.r.t. scale change Repeatability rate: # correspondences # possible correspondences K.Mikolajczyk, C.Schmid. Indexing Based on Scale Invariant Interest Points. ICCV

64 Matching with Features Problem: For each point correctly recognize the corresponding one? We need a reliable and distinctive descriptor 64

65 Thanks! 65