SURF Features. Jacky Baltes Dept. of Computer Science University of Manitoba WWW:

Transcription

1 SURF Features Jacky Baltes Dept. of Computer Science University of Manitoba WWW:

2 Salient Spatial Features Trying to find interest points Points that can be found independent of perspective transformations Distinctive (Unique in local region) Surrounding of pixel is rich in structure Repeatable (Different views) Stable under geometric, photometric transformations Stable to noise Well defined position in the image

3 Applications Tracking Object recognition Human action recognition Panorama stitching Robot localization Texture recognition

4 Covariant to Scale Change Changes in scale do not alter the structure of the image

5 Salient Features Scale Invariant Feature Transform (SIFT) David Lowe, 1999 Harris Affine Corner Detector Hessian Affine Corner Detector Edge Based Regions Intensity Based Regions Maximally Stable Extremal Regions (MSER) Entropy Based Salient Regions

6 Scale Invariant Feature Transform (SIFT) Scale-space Theorem: A local 3D Maximum of NLOG in (x,y,σ) Can be identified at different scales (scale invariant keypoint) Laplacian Kernel NLoG x, y, σ =σ 2 2 G

7 The Laplacian of Gaussian (LoG) Convolution of an image by the following kernel g x, y,t = 1 2 t 2 e x 2 y 2 / 2t L x, y ; t =g x, y, t I x, y Diameter t, L is called scale-space representation 2 norm L x, y ;t =t L xx L yy Based on Laplacian operator, which is sum of partial derivatives in Euclidean space A popular blob detector is based on LoG

8 Everything Clear? Maybe if you are a mathematician Another derivation Sobel, Prewitt edge detectors are gradient based Maximum of 1 st derivative Or zero-crossing of 2 nd derivative How to calculate 2 nd derivative?

9 Laplacian Need to calculate 2 nd derivative Change in 1 st derivative a b c d e f g h i 4 connectedness (f e) (e d) + (h-e) (e-b) = +f +d +h+b-4e 8 Connectedness (f e) (e d) + (h-e) (e-b) + (i-e) (e a) + (c-e) (e g) = +f + d + h + b + i + a + c + g - 8e

10 Laplacian Detect the zero crossing of the Laplacian to detect edges Because of the use of the 2 nd derivative, very sensitive to noise Remove noise by blurring the image first with a Gaussian kernel of size σ L og x, y = 1 [1 x2 y 2 x 2 y 2 2 ]e

11 Gradient based procedure Sobel Sobel

12 Laplacian

13 Sobel

14 Laplacian

15 Zero-crossing based procedure LoG

16 Laplacian of Gaussian

17 Plot in Scilab Gaussian

18 Edge-based Segmentation: examples Prewitt: needs edge linking Canny: needs cleaning

19 Difference of Gaussian Calculate the difference of Gaussian Radius 1 = 1.0, Radius 2 = 2.0

20 Difference of Gaussian Approximation of Laplacian NLoG D og x, y, =G x, y, k G x, y, Invariant to scale and rotation

21 SIFT Features

22 Scale Space Representation L is the scale-space representation Obtained by convoluting with a Gaussian kernel of size t g x, y,t = 1 2 t 2 e x 2 y 2 / 2t L x, y ; t =g x, y, t I x, y And partial derivatives of L L x = L x, L y= L y

23 Structure Tensor Also called 2 nd moment matrix Derived from the gradient Measures the pre-dominant gradient in a neighborhood and its coherence [ I 2 S w p = x p I x p I y p ] I x p I y p I 2 y p

24 Harris Affine Corner Detector Gradient distribution matrix (M) [ M = 2 D g 1 I 2 x p, D I x p, D I y p, D ] I x p, D I y p, D I 2 y p, D Calculate Eigenvalues of M

25 Eigenvectors and Eigenvalues Are vectors and scalars such for a matrix M such that M x= x Only exists if (M-λI) has no inverse. Characteristic/secular equation det(m-λi)=0

26 Eigenvectors and Eigenvalues Given a 2*2 matrix M M =[ a 11 a 12 a 21 a 22 ] det M I =0 1,2 = a 11 a 22 ± a 11 a a 11 a 22 a 12 a 21 2

27 Harris Affine Corner Detector Eigenvalues show the direction of change Curvature C=det M k trace 2 M C= 1 2 k Corners are stable under different lighting conditions

28 Hessian Affine Corner Detector The Hessian of a function with two arguments is defined as [ 2 I x p 2 I 2 x y p ] H p = 2 I y x p 2 I y p 2 If function is continuous, then 2 I x y p = 2 I y x p

29 Hessian Approximation Hessian approximation using Gaussians L xx p, = 2 I g I p 2 x Convolution of image by Gaussian H p, =[ L xx p, L xy p, ] L xy p, L yy p, Blob detector using maximums of the determinant

30 SURF Algorithm Uses the integral image to compute averages over areas efficiently (4 lookups and 3 arithmetic operations)

31 Integral Image + Bottum Right Sum of all the pixels to the left and top of the pixel Sum of any rectangular region can be extracted in constant time using four lookups and arithmetic - Top Right - Bottum Left + Top Left

32 SURF and Hessian Matrix SURF is based on calculating the Hessian matrix Authors claim more robust than Harris detector Hessian is an approximation of 2 nd order derivative large values for maxima and minima

33 Approximation of Gaussian 2 nd order partial derivatives Cropped and discretized L_yy L_xy

34 Box Filters Coarse approximation allows computation of value by integral image L_yy L_xy

35 Scale Space Transform Useful for finding interest points Can scale filter without increased computational cost

36 Scale Space What filter sizes do we need to use? 9x9 box filters are approximations of Gaussian with σ = 1.2 det H =D xx D yy wd xy 2 W is a weight that corrects for the approximation of the Gaussian. Analysis shows that w=0.9 is a good enough approximation

37 Scale Space The approximation of the Hessian determinant is equivalent to finding blobs Used to detect local maximas in the scale space using different sized filters Stored in the so-called blob response map

38 Scale Space Doubling of σ represents one octave of the scale space Each octave has a constant number of scale levels Filter sized needs to be increased by 6 pixels Lobes are set of 1/3 of filter Needs to be increased by 2 To keep a central pixel

39 Scale Space

40 Scale Space 9,15,21,27 are first octave Corresponds to a change of σ of 1.2 to 3.2 Min and max scale leves per octave are used to suppress maximas that are not maximas in scale space Filter size increase doubles per octave 15,27,39,51 Large change in σ in first two octaves can be avoided by scaling image first

41 Scale Space

42 Haar Wavelets

43 Orientation S is the scale at which an interest point was detected Calculate response of Haar wavelets in the x and y direction around the interest point Radius is 6*s Sampling is s So take 6 samples along one direction Size of the Haar wavelets is 4*s

44 Orientation Responses weighted by Gaussian with σ=2s X direction is Haar wavelet response in x Y direction is Haar wavelet response in y Sum all points in 60 deg. window Import parameter Orientation is window with the longest vector

45 Feature Descriptors Similar to SIFT (David Lowe) Generate square window Size is 20s Orientation along the orientation calculated Split window into 4x4 square subregions Calculate Haar wavelet response in x and y Rotate along the orientation Weight with a Gaussian of σ=3.3s

46 Feature Descriptor

47 Feature Descriptor Calculate sum of changes as well as absolute sum of changes 4 entries per field, 16 sub-regions = 64 entries

48 Robustness to Noise

49 Evaluation The set of features of the feature descriptors was arrived at by experimentation Evaluations Camera calibration Object detection Faster and more robust than other detectors

50 References Herbert Bay, Andreas Ess, Tinne Tuytelaars, Luc Van Gool, "SURF: Speeded Up Robust Features", Computer Vision and Image Understanding (CVIU), Vol. 110, No. 3, pp , 2008 SURF, David Tam (Ryerson), Computer Robotics Vision (CRV) Tutorials 2010, Salient Feature Detectors and Descriptors: Affine-Hessian, Harris, MSER, SIFT, SURF, Amir-Hossein Shabani, Computer Robotics Vision (CRV) Tutorials 2009, Chris Evans. Notes on the OpenSURF Library. January 18, 2009,