Achieving scale covariance

Transcription

1 Achieving scale covariance Goal: independently detect corresponding regions in scaled versions of the same image Need scale selection mechanism for finding characteristic region size that is covariant with the image transformation S.Lazebnik, UNC

2 Blobs (and scale selection) S.Lazebnik, UNC

3 Blob detection in 2D Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D 2 2 g g 2 g 2 2 x y S.Lazebnik, UNC of Gaussian controls the radius of the operator

4 Blob detection in 2D Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D 2 2 g 2 2 g Scale-normalized norm g 2 2 y x need this to make filter response insensitive to the scale S.Lazebnik, UNC

5 LoG Blob Finding and Scale Lapacian of Gaussian (LoG) filter extrema locate blobs maxima = dark blobs on light background minima = light blobs on dark background Scale of blob (size ; radius in pixels) is determined by the sigma parameter of the LoG filter. LoG sigma = 2 LoG sigma = 10

6 Scale Selection Laplacian operator.

7 Scale selection At what scale does the Laplacian achieve a maximum response to a binary circle of radius r? r image Laplacian

8 Scale selection At what scale does the Laplacian achieve a maximum response to a binary circle of radius r? To get maximum response, the zeros of the Laplacian have to be aligned with the circle The Laplacian is given by (up to scale): ( x y 2 ) e ( x 2 y 2 ) / 2 2 Therefore, the maximum response occurs at r circle image Laplacian r / 2.

9 Characteristic scale We define the characteristic scale of a blob as the scale that produces peak of Laplacian response in the blob center characteristic scale T. Lindeberg (1998). "Feature detection with automatic scale selection." International Journal of Computer Vision 30 (2): pp

10 Scale-space blob detector 1. Convolve image with scale-normalized Laplacian at several scales 2. Find maxima of squared Laplacian response in scale-space

11 Scale-space blob detector: Example S.Lazebnik, UNC

12 Scale-space blob detector: Example S.Lazebnik, UNC

13 Scale-space blob detector: Example S.Lazebnik, UNC

14 Efficient implementation Approximating the Laplacian with a difference of Gaussians: L 2 Gxx ( x, y, ) G yy ( x, y, ) (Laplacian) DoG G ( x, y, k ) G ( x, y, ) (Difference of Gaussians) Why do this: 2D Gaussian filter is separable into two 1D filters, making it more efficient to compute.

15 Structure Per-pixel transformations Interest points: edges, corners, blobs Feature descriptors Computer vision: models, learning and inference Simon J.D. Prince 15

16 Feature Descriptors Most features descriptors can be thought of either: Templates Intensity, gradients, etc. Histograms Color, texture, SIFT descriptors, etc. or combinations of both James Hays, Brown U.

17 Textons An attempt to characterize texture Replace each pixel with integer representing the texture type Computer vision: models, learning and inference Simon J.D. Prince 17

18 Computing Textons Take a bank of filters and apply to lots of images Cluster in filter space For new pixel, filter surrounding region with same filter bank, and assign to nearest cluster Computer vision: models, learning and inference Simon J.D. Prince 18

19 Bag of words descriptor Compute visual features in image Compute descriptor around each Find closest match in library and assign index Compute histogram of these indices over the region Dictionary computed using K-means Computer vision: models, learning and inference Simon J.D. Prince 19

20 Sift Descriptor Goal: produce a scale and rotation invariant vector that describes the region around an interest point. All calculations are relative to the orientation and scale of the keypoint Makes descriptor invariant to rotation and scale Computer vision: models, learning and inference Simon J.D. Prince 20

21 HoG Descriptor HOG=Histogram of Oriented Gradients Computer vision: models, learning and inference Simon J.D. Prince 21

22 Shape Context Descriptor Count the number of points inside each bin, e.g.: Count = 4... Count = 10 Log-polar binning: more precision for nearby points, more flexibility for farther points. Belongie & Malik, ICCV 2001 K. Grauman, B. Leibe

23 Shape Context Descriptor K. Grauman, B. Leibe

24 SIFT Detector/Descriptor Algorithm Steps Let s look under the hood at one of the most popular image feature detectors. Generating Gaussian and DOG pyramid Locating scale-space extrema Eliminating edge responses (related to Harris corners) Computing patch orientation Spatial histogram of gradients References: Lowe, Distinctive Image Features from Scale Invariant Keypoints International Journal of Computer Vision, Vol.60(2):91-110, Crowley et.al., Fast Computation of Characteristic Scale using a HalfOctave Pyramid, Cognitive Vision Workshop, Zurich Switzerland, Sept 19-20, 2002.

25 Generating a Gaussian Pyramid Basic Functions: Blur (convolve with Gaussian to smooth image) DownSample (reduce image size by half) Upsample (double image size)

26 Using a Half-Octave Pyramid From Crowley et.al., Fast Computation of Characteristic Scale using a Half-Octave Pyramid.. General idea: cascaded filtering using [ ] kernel (sigma = 1) to generate a pyramid with two images per octave (power of 2 change in resolution). When we reach a full octave, downsample the image. Crowley etal.

27 Half Octave Gaussian and DOG blur blur tmp blur downsample downsample blur blur tmp blur

28 Half Octave Gaussian and DOG L1 L2 blur G1 blur G2 G1,L1: N x M G2,L2: N x M G3,L3: N/2 x M/2 G4,L4: N/2 x M/2 G5,L5: N/4 x M/4 and so on tmp blur G5 downsample downsample G3 L3 blur G4 blur L4 tmp blur

29 Find Extrema in Space and Scale DOG level L-1 Scale DOG level L DOG level L+1 Space Hint: when finding maxima or minima at level L, we must DownSample or UpSample as necessary to make DOG images at level L-1 and L+1 the same size as L.

30 Filtering out edge responses Either Using Harris corner matrix (2x2 matrix computed from first partial image derivatives) H Or use Hessian matrix (2x2 matrix computed from second partial image derivatives) Dxx Dxy Dxy Dyy Lowe s notation, Section 4.1

31 Filtering out edge responses Would be removed by Harris method Since the goal is filtering out edges, I like this one better Would be removed by Hessian method

32 Any Significant Difference? Not much difference Yellow=extrema, blue=removed by Harris corners, red=removed by Hessian

33 Computing Patch Orientation

34 Which Gaussian is Closest in Scale? L G(i) Sigma = Sigma = 1.18 (Crowley paper) G(i+1) Sigma = sqrt(2)

35 Which Gaussian is Closest in Scale? L G(i) Sigma = Sigma = 1.18 (Crowley paper) G(i+1) Sigma = sqrt(2) G(I) is closest in scale to L

36 Computing Orientation

37 Gradient Magnitude and Angle Dx Dy M

38 Computing Orientation

39 Sampling/Weighting over a Circular Region Gaussian-weighted circular window == 2D Gaussian kernel. Weight magnitude by Gaussian kernel. What size kernel? Lowe suggests sigma value of weighting kernel should be 3 times bigger than scale of the keypoint. Design decision: make the gaussian kernel have sigma =3, in the pixel coordinate system of G(I), the image from the Gaussian pyramid that magnitude and angle were computed with. Plus or minus 3 sigma, with sigma = 3, gives a width of 19 pixels. 19x19

40 Example Keypoint location = extrema location Keypoint scale is scale of the DOG image

41 Example (continued) gradient magnitude gaussian image (at closest scale, from pyramid) gradient orientation

42 Example (continued).* gradient magnitude = weighted by 2D gaussian kernel weighted gradient magnitude

43 Example (continued) weighted gradient magnitude weighted orientation histogram. Each bucket contains sum of weighted gradient magnitudes corresponding to angles that fall within that bucket. gradient orientation 36 buckets 10 degree range of angles in each bucket, i.e. 0 <=ang<10 : bucket 1 10<=ang<20 : bucket 2 20<=ang<30 : bucket 3

44 Computing Orientation

45 Example (continued) weighted gradient magnitude weighted orientation histogram. peak 80% of peak value gradient orientation degrees Orientation of keypoint is approximately 25 degrees

46 Example (continued) There may be multiple orientations. peak Second peak 80% of peak value In this case, generate duplicate SIFT patches, one with orientation at 25 degrees, one at 155 degrees. Design decision: one might want to limit number of possible multiple peaks to two.

47 Forming the Sift Descriptor 1. Compute image gradients 2. Pool into local histograms 3. Concatenate histograms 4. Normalize histograms Computer vision: models, learning and inference Simon J.D. Prince 47

48 Spatial Histogram of Gradients Computed on rotated and scaled version of window according to computed orientation & scale resample the window Based on gradients weighted by a Gaussian of variance half the window (for smooth falloff)

49 Spatial Histogram of Gradients 4x4 array of gradient orientation histograms each orientation increment is weighted by magnitude 8 orientations x 4x4 array = 128 dimensions Motivation: some sensitivity to spatial layout, but not too much. showing only 2x2 here but is 4x4

50 Ensure smoothness Gaussian weight Trilinear interpolation a given gradient contributes to 8 bins: 4 in space times 2 in orientation

51 Reduce effect of illumination 128-dim vector normalized to 1 Threshold gradient magnitudes to avoid excessive influence of high gradients after normalization, clamp gradients >0.2 renormalize

52 SIFT Invariance and covariance properties Laplacian blob response (detection) is invariant w.r.t. rotation and scaling Blob location covariant w.r.t. rotation and scale Normalized patch is invariant wrt rotation/scale Normalized vector is insensitive to illumination Not invariant/covariant with respect to affine transformations (such as due to small changes in viewing angle) S.Lazebnik, UNC

53 Achieving affine covariance Consider the second moment matrix of the window containing the blob: I x2 M w( x, y ) x, y I x I y IxI y R R 2 I y 0 2 direction of the fastest change Recall: u [u v] M const v ( max)-1/2 direction of the slowest change ( min)-1/2 This ellipse visualizes the characteristic shape of the window S.Lazebnik, UNC

54 Affine adaptation example Scale-invariant regions (blobs) S.Lazebnik, UNC

55 Affine adaptation example Affine-adapted blobs S.Lazebnik, UNC

56 Affine adaptation Problem: the second moment window determined by weights w(x,y) must match the characteristic shape of the region Solution: iterative approach Use a circular window to compute second moment matrix Perform affine adaptation to find an ellipse-shaped window Recompute second moment matrix using new window and iterate S.Lazebnik, UNC

57 Iterative affine adaptation K. Mikolajczyk and C. Schmid, Scale and Affine invariant interest point detectors, IJCV 60(1):63-86,

58 Affine covariance Affinely transformed versions of the same neighborhood will give rise to ellipses that are related by the same transformation What to do if we want to compare these image regions? Affine normalization: transform these regions into samesize circles S.Lazebnik, UNC

59 Affine normalization Problem: There is no unique transformation from an ellipse to a unit circle We can rotate or flip a unit circle, and it still stays a unit circle S.Lazebnik, UNC

60 Eliminating rotation ambiguity To assign a unique orientation to circular image windows: Create histogram of local gradient directions in the patch Assign canonical orientation at peak of smoothed histogram 0 S.Lazebnik, UNC 2

61 From covariant regions to invariant features Extract affine regions Normalize regions Eliminate rotational ambiguity Compute appearance descriptors SIFT (Lowe 04) S.Lazebnik, UNC