Invariant local eatures Invariant Local Features Tuesday, February 6 Subset o local eature types designed to be invariant to Scale Translation Rotation Aine transormations Illumination 1) Detect distinctive interest points ) Extract invariant descriptors y 1 y y d x 1 x x d [Mikolajczyk & Schmid, Matas et al., Tuytelaars & Van Gool, Lowe, Kadir et al., ] (Good) invariant local eatures Reliably detected Distinctive Robust to noise, blur, etc. Description normalized properly Classes o transormations Euclidean/rigid: Translation + rotation Similarity: Translation + rotation + uniorm scale Aine: Similarity + shear Projective: Aine + projective warps Projective Similarity Translation Aine Translation transormation and Scaling Case study: panorama stitching How do we build panorama? We need to match (align) images [Brown, Szeliski, and Winder, CVPR 005] [These slides are rom Darya Frolova and Denis Simakov] 1
Detect eature points in both images Detect eature points in both images Find corresponding pairs Detect eature points in both images Find corresponding pairs Use these pairs to align images Problem 1: Detect the same point independently in both images no chance to match! We need a repeatable detector Problem : For each point correctly recognize the corresponding one? Interest operators: an introductory example Harris corner detector We need a reliable and distinctive descriptor C.Harris, M.Stephens. A Combined Corner and Edge Detector. 1988
The Basic Idea Harris Detector: Basic Idea We should easily recognize the point by looking through a small window Shiting a window in any direction should give a large change in intensity lat region: no change in all directions edge : no change along the edge direction corner : signiicant change in all directions Harris Detector: Basic Idea Window-averaged change o intensity or the shit [u,v]: xy, [ ] Euv (, ) = wxy (, ) I( x+ uy, + v) I( xy, ) Window unction Shited intensity Intensity Corner: signiicant change in all directions. [Figure rom C. Schmid] Window unction w(x,y) = 1 in window, 0 outside or Gaussian A bilinear approximation or average intensity change or small shits in direction [u, v]: u Euv (, ) [ uv, ] M v where M is a matrix computed rom image derivatives: I x II x y M = w( x, y) xy, I xiy Iy Intensity change in shiting window: eigenvalues tell us how intensity changes in dierent directions u Euv (, ) [ uv, ] M v direction o the astest change (λ max ) -1/ (λ min )-1/ λ 1, λ eigenvalues o M direction o the slowest change 3
Classiication o image points using eigenvalues o M: λ 1 and λ are small; E is almost constant in all directions λ Edge λ >> λ 1 Flat region Corner λ 1 and λ are large, λ 1 ~ λ ; E increases in all directions Edge λ 1 >> λ Measure o corner response: ( trace ) R = det M k M det M = λ1λ trace M = λ + λ 1 (k empirical constant, k = 0.04-0.06) Avoid computing eigenvalues themselves. λ 1 Harris Detector λ R depends only on eigenvalues o M R is large or a corner R is negative with large magnitude or an edge Edge R < 0 Corner R > 0 The Algorithm: Find points with large corner response unction R (R > threshold) Take the points o local maxima o R R is small or a lat region Flat R small Edge R < 0 λ 1 Compute corner response R 4
Find points with large corner response: R>threshold Take only the points o local maxima o R Harris Detector: Some Properties Rotation invariance Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation Harris Detector: Some Properties Not invariant to image scale! All points will be classiied as edges Corner! [Images rom T. Tuytelaars] 5
Consider regions (e.g. circles) o dierent sizes around a point Regions o corresponding content will look the same in both images The problem: how do we choose corresponding circles independently in each image? Solution: Design a unction on the region (circle), which is scale invariant (the same or corresponding regions, even i they are at dierent scales) Example: average intensity. For corresponding regions (even o dierent sizes) it will be the same. For a point in one image, we can consider it as a unction o (circle radius) Common approach: Take a local maximum o this unction Observation:, or which the maximum is achieved, should be invariant to image scale. Important: this scale invariant is ound in each image independently! Image 1 scale = 1/ Image Image 1 scale = 1/ Image s 1 s A good unction or scale detection: has one stable sharp peak bad bad Good! For usual images: a good unction would be a one which responds to contrast (sharp local intensity change) Functions or determining scale Kernels: = Kernel Image ( xx (,, ) yy (,, )) L= G x y + G x y σ σ σ (Laplacian) DoG= Gxyk (,, σ ) Gxy (,, σ ) (Dierence o Gaussians) where Gaussian x + y 1 πσ σ Gxy (,, σ ) = e Note: both kernels are invariant to scale and rotation 6
Scale Invariant Detectors Harris-Laplacian 1 Find local maximum o: Harris corner detector in space (image coordinates) Laplacian in scale scale SIFT (Lowe) scale Find local maximum o: Dierence o Gaussians in space and scale y y Harris x Laplacian DoG Scale Invariant Detectors Experimental evaluation o detectors w.r.t. scale change Repeatability rate: # correspondences # possible correspondences DoG x 1 K.Mikolajczyk, C.Schmid. Indexing Based on Scale Invariant Interest Points. ICCV 001 D.Lowe. Distinctive Image Features rom Scale-Invariant Keypoints. IJCV 004 K.Mikolajczyk, C.Schmid. Indexing Based on Scale Invariant Interest Points. ICCV 001 : Summary Given: two images o the same scene with a large scale dierence between them Goal: ind the same interest points independently in each image Solution: search or maxima o suitable unctions in scale and in space (over the image) Methods: 1. Harris-Laplacian [Mikolajczyk, Schmid]: maximize Laplacian over scale, Harris measure o corner response over the image. SIFT [Lowe]: maximize Dierence o Gaussians over scale and space Aine Invariant Detection Above we considered: Similarity transorm (rotation + uniorm scale) Now we go on to: Aine transorm (rotation + non-uniorm scale) Aine Invariant Detection Intensity-based regions (IBR): Start rom a local intensity extrema Consider intensity proile along rays Select maximum o (t) along each ray Connect local maxima Fit an ellipse Aine Invariant Detection Maximally Stable Extremal Regions (MSER) Threshold image intensities: I > I 0 Extract connected components ( Extremal Regions ) Seek extremal regions that remain Maximally Stable under range o thresholds T.Tuytelaars, L.V.Gool. Wide Baseline Stereo Matching Based on Local, Ainely Invariant Regions. BMVC 000. Matas et al. Robust Wide Baseline Stereo rom Maximally Stable Extremal Regions. BMVC 00. 7
Point Descriptors We know how to detect points Next question: How to describe them or matching?? Rotation Invariant Descriptors Harris corner response measure: depends only on the eigenvalues o the matrix M Point descriptor should be: 1. Invariant. Distinctive C.Harris, M.Stephens. A Combined Corner and Edge Detector. 1988 Rotation Invariant Descriptors Find local orientation Dominant direction o gradient Scale Invariant Descriptors Use the scale determined by detector to compute descriptor in a normalized rame Rotate description relative to dominant orientation 1 K.Mikolajczyk, C.Schmid. Indexing Based on Scale Invariant Interest Points. ICCV 001 D.Lowe. Distinctive Image Features rom Scale-Invariant Keypoints. Accepted to IJCV 004 [Images rom T. Tuytelaars] Aine Invariant Descriptors Compute rotation invariant descriptor in the aine normalized rame (deskew) Applications Wide baseline stereo Motion tracking Panoramas Mobile robot navigation 3D reconstruction Recognition Speciic objects Textures Categories [Image rom T. Tuytelaars] 8
Wide baseline stereo Panorama stitching [Image rom T. Tuytelaars ECCV 006 tutorial] Brown, Szeliski, and Winder, 005 Recognition o speciic objects, scenes Recognition o categories Constellation model Bags o words Schmid and Mohr 1997 Sivic and Zisserman, 003 Rothganger et al. 003 Lowe 00 Weber et al. (000) Fergus et al. (003) Csurka et al. (004) Dorko & Schmid (005) Sivic et al. (005) Lazebnik et al. (006), [Slide rom Lazebnik, Sicily 006] Comparative evaluations Testing various detector and descriptor options or relative repeatability and distinctiveness Planar objects / lat scenes: Mikolajczyk & Schmid (004) 3D objects: Moreels & Perona (005) [Images rom Lazebnik, Sicily 006] Issues For speciic-level recognition scaling the search? Complexity Distinctiveness For category-level recognition are eatures most appropriate? Sparse Strict appearance description Texture vs. shape Expense o detecting interest points 9