Scale-space image processing

Scale-space image processing Corresponding image features can appear at different scales Like shift-invariance, scale-invariance of image processing algorithms is often desirable. Scale-space representation is useful to process an image in a manner that is both shift-invariant and scale-invariant Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 1

Scale-space image processing Scale-space theory Laplacian of Gaussian (LoG) and Difference of Gaussian (DoG) Scale-space edge detection Scale-space keypoint detection Harris-Laplacian SIFT detector SURF detector Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 2

Scale-space representation of a signal Parametric family of signals f t (x) where fine-scale information is successively attenuated scale t Successive smoothing with a Gaussian filter Zero-crossings of 2 nd derivative Fewer edges at coarser scales ( ) f t x Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 3

Scale-space representation of images Parametric family of images smoothed by Gaussian filter t Coarser scales Shift-invariance Original image f (x,y) Rotation-invariance Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 4

Scale-space representation of images (cont.) Commutative semigroup property Separability Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 5

Scale-space representation of images (cont.) Non-creation of local extrema (for f (x,y) and all of its partial derivatives) since and unimodal. Solution to diffusion equation (heat equation) Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 6

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 7

Laplacian of Gaussian LoG vs. DoG Difference of Gaussians t = σ 2 = 1 t = σ 2 = 1, k = 1.1 1 2 2 f t ( x, y)= t f t x, y ( ) 0.05 0.05 0 0-0.05-0.05-0.1-0.1-0.15-0.15-0.2-0.2-0.25-0.25 4 4 2 4 2 0 2 0 0-2 -2-2 Y -4-4 X Y -4-4 -2 X 0 2 4 Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 8

LoG vs. DoG (cont.) Laplacian of Gaussian Difference of Gaussians t = σ 2 = 1 t = σ 2 = 1, k = 1.1 0.8 0.8 0.6 0.6 H 0.4 H 0.4 0.2 0.2 0 0 2 0-2 -2 0 2 2 0-2 -2 0 2 ω y ω x ω y ω x Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 9

Scale space: Laplacian images t = 1 t = 4 t = 16 t = 64 Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 10

Scale space: Binarized Laplacian images t = 1 t = 4 t = 16 t = 64 Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 11

Scale space: edge detection Zero crossings of Laplacian images t = 1 t = 4 t = 16 t = 64 Low-gradient-magnitude edges removed Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 12

Laplacian zero-crossings Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 13

Keypoint detection with automatic scale selection Scale-space representation provides all scales; which scale is best for keypoint detection? Harris-Laplacian 1. Detect Harris corners at some initial scale 2. For each Harris corner detect characteristic scale t h = argmax t t 2 f t ( x h, y ) h 3. Apply Harris detector in a spatial neighborhood at scale t h to refine keypoint location 4. Repeat 2. and 3. until convergence scale t y Harris Harris Harris x Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 15

Keypoint detection with automatic scale selection Harris-Laplacian example (150 strongest peaks) Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 16

Keypoint detection with automatic scale selection Harris-Laplacian example (200 strongest peaks) Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 17

Scale (next octave) t = 16 t = 8 2 t = 8 t = 4 2 t = 4 SIFT keypoint detection SIFT - Scale-Invariant Feature Transform Decompose image into DoG scale-space representation Detect minima and maxima locally and across scales Fit 3-d quadratic function to localize extrema with subpixel/sub-scale accuracy [Brown, Lowe, 2002] Eliminate edge responses based on Hessian t = 4 t = 2 2 Scale (first octave) t = 2 t = 2 Scale t = 1 Gaussian Difference of Gaussian (DoG) Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 18 [Lowe, 1999, 2004]

SIFT scale space pyramid: octave 1 - - - Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 19

SIFT scale space pyramid: octave 2 - - - Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 20

SIFT scale space pyramid: octave 3 - - - Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 21

SIFT scale space pyramid: octave 4 - - - Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 22

SIFT scale space pyramid: octave 5 - - - Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 23

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 24 SIFT keypoints

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 25 SIFT keypoints

Robustness against scaling Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 26 [Mikolajczyk, Schmid, 2001]

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 28 Hessian keypoints in scale space

SURF keypoint detection SURF Speeded Up Robust Features [Bay, Tuytelaars, Van Gool, ECCV 2006] No subsampling all resolution levels at full spatial resolution Simple approximation of scale space Gaussian derivatives using integral images Determinant of Hessian D t xy Non-maximum suppression in 3x3x3 [x,y,t] neighborhood Interpolation of maximum of det(h) in image space x,y and scale t Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 29

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 30 SURF keypoints

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 31 SIFT keypoints

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 32 SURF keypoints

Digital Image Processing: Bernd Girod, 2013 Stanford University -- Scale Space 33 SIFT keypoints