Lecture 6: Finding Features (part 1/2)

Transcription

1 Lecture 6: Finding Features (part 1/2) Professor Fei- Fei Li Stanford Vision Lab Lecture 6 -! 1

2 What we will learn today? Local invariant features MoHvaHon Requirements, invariances Keypoint localizahon Harris corner detector Scale invariant region selechon AutomaHc scale selechon Difference- of- Gaussian (DoG) detector SIFT: an image region descriptor Next lecture (#7) Lecture 6 -! 2

3 What we will learn today? Local invariant features MoHvaHon Requirements, invariances Keypoint localizahon Harris corner detector Scale invariant region selechon AutomaHc scale selechon Difference- of- Gaussian (DoG) detector Some SIFT: background an image reading: region descriptor Rick Szeliski, Chapter ; David Lowe, IJCV 2004 Lecture 6 -! 3

4 Image matching: a challenging problem Lecture 6 -! 4

5 Image matching: a challenging problem by Diva Sian by swashford Slide credit: Steve Seitz Lecture 6 -! 5

6 Harder Case by Diva Sian by scgbt Slide credit: Steve Seitz Lecture 6 -! 6

7 Harder SHll? NASA Mars Rover images Slide credit: Steve Seitz Lecture 6 -! 7

8 Answer Below (Look for Hny colored squares) NASA Mars Rover images with SIFT feature matches (Figure by Noah Snavely) Slide credit: Steve Seitz Lecture 6 -! 8

9 MoHvaHon for using local features Global representahons have major limitahons Instead, describe and match only local regions Increased robustness to Occlusions ArHculaHon d d q Intra- category variahons θ φ θ q φ Lecture 6 -! 9

10 General Approach 1. Find a set of distinctive keypoints A 1 A 2 A 3 B 3 B 1 B 2 2. Define a region around each keypoint 3. Extract and normalize the region content N pixels N pixels f A e.g. color Similarity measure d( f A, fb) < T f B e.g. color 4. Compute a local descriptor from the normalized region 5. Match local descriptors Slide credit: Bastian Leibe Lecture 6 -! 10

11 Common Requirements Problem 1: Detect the same point independently in both images No chance to match! We need a repeatable detector! Slide credit: Darya Frolova, Denis Simakov Lecture 6 -! 11

12 Common Requirements Problem 1: Detect the same point independently in both images Problem 2: For each point correctly recognize the corresponding one? We need a reliable and distinctive descriptor! Slide credit: Darya Frolova, Denis Simakov Lecture 6 -! 12

13 Invariance: Geometric TransformaHons Slide credit: Steve Seitz Lecture 6 -! 13

14 Levels of Geometric Invariance CS131 CS231a Slide credit: Bastian Leibe Lecture 6 -! 14

15 Invariance: Photometric TransformaHons Ofen modeled as a linear transformahon: Scaling + Offset Slide credit: Tinne Tuytelaars Lecture 6 -! 15

16 Requirements Region extrachon needs to be repeatable and accurate Invariant to translahon, rotahon, scale changes Robust or covariant to out- of- plane ( affine) transformahons Robust to lighhng variahons, noise, blur, quanhzahon Locality: Features are local, therefore robust to occlusion and cluier. QuanHty: We need a sufficient number of regions to cover the object. DisHncHveness: The regions should contain intereshng structure. Efficiency: Close to real- Hme performance. Slide credit: Bastian Leibe Lecture 6 -! 16

17 Many ExisHng Detectors Available Hessian & Harris [Beaudet 78], [Harris 88] Laplacian, DoG [Lindeberg 98], [Lowe 99] Harris- /Hessian- Laplace [Mikolajczyk & Schmid 01] Harris- /Hessian- Affine [Mikolajczyk & Schmid 04] EBR and IBR [Tuytelaars & Van Gool 04] MSER [Matas 02] Salient Regions [Kadir & Brady 01] Others Those detectors have become a basic building block for many recent applica8ons in Computer Vision. Slide credit: Bastian Leibe Lecture 6 -! 17

18 Keypoint LocalizaHon Goals: Repeatable detechon Precise localizahon InteresHng content Look for two- dimensional signal changes Slide credit: Bastian Leibe Lecture 6 -! 18

19 Finding Corners Key property: In the region around a corner, image gradient has two or more dominant direchons Corners are repeatable and dis8nc8ve C.Harris and M.Stephens. "A Combined Corner and Edge Detector. Proceedings of the 4th Alvey Vision Conference, Slide credit: Svetlana Lazebnik Lecture 6 -! 19

20 Corners as DisHncHve Interest Points Design criteria We should easily recognize the point by looking through a small window (locality) Shifing the window in any direc8on should give a large change in intensity (good localiza8on) flat region: no change in all directions edge : no change along the edge direction Lecture 6 -! 20 corner : significant change in all directions Slide credit: Alyosha Efros

21 Harris Detector FormulaHon Change of intensity for the shif [u,v]: xy, [ ] 2 Euv (,) = wxy (, ) I( x+ uy, + v) I(, xy) Window function Shifted intensity Intensity Window function w(x,y) = 1 in window, 0 outside or Gaussian Slide credit: Rick Szeliski Lecture 6 -! 21

22 Harris Detector FormulaHon Slide credit: Rick Szeliski This measure of change can be approximated by: E ( u, v) [ u v] u v where M is a 2 2 matrix computed from image derivahves: M M M 2 Ix IxI y = w(, x y) 2 xy, II x y Iy Sum over image region the area we are checking for corner Gradient with respect to x, times gradient with respect to y Lecture 6 -! 22

23 Harris Detector FormulaHon Slide credit: Rick Szeliski Image I I x where M is a 2 2 matrix computed from image derivahves: M Sum over image region the area we are checking for corner M 2 Ix IxI y = w(, x y) 2 xy, II x y Iy I y I x I y Gradient with respect to x, times gradient with respect to y Lecture 6 -! 23

24 What Does This Matrix Reveal? First, let s consider an axis- aligned corner: M = I I x 2 x I y I I x y 2 I y = λ1 0 0 λ 2 This means: Dominant gradient direchons align with x or y axis If either λ is close to 0, then this is not a corner, so look for locahons where both are large. What if we have a corner that is not aligned with the image axes? Slide credit: David Jacobs Lecture 6 -! 24

25 What Does This Matrix Reveal? First, let s consider an axis- aligned corner: M = I I x 2 x I y I I x y 2 I y = λ1 0 0 λ 2 This means: Dominant gradient direchons align with x or y axis If either λ is close to 0, then this is not a corner, so look for locahons where both are large. What if we have a corner that is not aligned with the image axes? Slide credit: David Jacobs Lecture 6 -! 25

26 General Case Since M is symmetric, we have (Eigenvalue decomposition) λ = R R 0 λ2 We can visualize M as an ellipse with axis lengths determined by the eigenvalues and orientahon determined by R Direction of the fastest change M (λ max ) -1/2 (λ min ) -1/2 Direction of the slowest change adapted from Darya Frolova, Denis Simakov Lecture 6 -! 26

27 InterpreHng the Eigenvalues ClassificaHon of image points using eigenvalues of M: λ 2 Edge λ 2 >> λ 1 Corner λ 1 and λ 2 are large, λ 1 ~ λ 2 ; E increases in all direchons Slide credit: Kristen Grauman λ 1 and λ 2 are small; E is almost constant in all direchons Flat region Edge λ 1 >> λ 2 λ 1 Lecture 6 -! 27

28 Corner Response FuncHon θ = det(m ) α trace(m ) 2 = λ 1 λ 2 α(λ 1 + λ 2 ) 2 λ 2 Edge θ < 0 Corner θ > 0 Slide credit: Kristen Grauman Fast approximahon Avoid compuhng the eigenvalues α: constant (0.04 to 0.06) Flat region Edge θ < 0 λ 1 Lecture 6 -! 28

29 Window FuncHon w(x,y) M OpHon 1: uniform window Sum over square window Problem: not rotahon invariant 2 Ix IxI y = w(, x y) 2 xy, II x y Iy 2 Ix IxI y M = 2 xy, II x y Iy 1 in window, 0 outside OpHon 2: Smooth with Gaussian Gaussian already performs weighted sum 2 Ix IxI y M = g( σ ) 2 II x y Iy Result is rotahon invariant Gaussian Slide credit: Bastian Leibe Lecture 6 -! 29

30 Summary: Harris Detector [Harris88] Compute second moment matrix (autocorrelahon matrix) 2 Ix( σd) IxIy( σ ) D M( σi, σd) = g( σi) 2 II x y( σd) Iy( σd) 2. Square of derivatives 1. Image derivatives I x I y I x 2 I y 2 I x I y Slide credit: Krystian Mikolajczyk 3. Gaussian filter g(σ I ) 4. Cornerness function two strong eigenvalues θ = det[m (σ I,σ D )] α[trace(m (σ I,σ D ))] 2 = gi ( ) gi ( ) [ gii ( )] α[ gi ( ) + gi ( )] x y x y x y 5. Perform non-maximum suppression g(i x2 ) g(i y2 ) g(i x I y ) Lecture 6 -! 30 R

31 Harris Detector: Workflow Slide adapted from Darya Frolova, Denis Simakov Lecture 6 -! 31

32 Harris Detector: Workflow - computer corner responses θ Slide adapted from Darya Frolova, Denis Simakov Lecture 6 -! 32

33 Harris Detector: Workflow - Take only the local maxima of θ, where θ>threshold Slide adapted from Darya Frolova, Denis Simakov Lecture 6 -! 33

34 Harris Detector: Workflow - ResulHng Harris points Slide adapted from Darya Frolova, Denis Simakov Lecture 6 -! 34

35 Harris Detector Responses [Harris88] Effect: A very precise corner detector. Slide credit: Krystian Mikolajczyk Lecture 6 -! 35

36 Harris Detector Responses [Harris88] Slide credit: Krystian Mikolajczyk Lecture 6 -! 36

37 Harris Detector Responses [Harris88] Results are well suited for finding stereo correspondences Slide credit: Kristen Grauman Lecture 6 -! 37

38 Harris Detector: ProperHes TranslaHon invariance? Slide credit: Kristen Grauman Lecture 6 -! 38

39 Harris Detector: ProperHes TranslaHon invariance RotaHon invariance? Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response θ is invariant to image rotation Slide credit: Kristen Grauman Lecture 6 -! 39

40 Harris Detector: ProperHes TranslaHon invariance RotaHon invariance Scale invariance? Corner Not invariant to image scale! All points will be classified as edges! Slide credit: Kristen Grauman Lecture 6 -! 40

41 What we have learned today? Local invariant features MoHvaHon Requirements, invariances Keypoint localizahon Harris corner detector Scale invariant region selechon AutomaHc scale selechon Difference- of- Gaussian (DoG) detector SIFT: an image region descriptor Some background reading: Rick Szeliski, Chapter ; David Lowe, IJCV 2004 Lecture 6 -! 41 Next lecture (#7)