6.869 Advances in Computer Vision. Prof. Bill Freeman March 1, 2005

Transcription

1 6.869 Advances in Computer Vision Prof. Bill Freeman March

2 2

3 Local Features Matching points across images important for: object identification instance recognition object class recognition pose estimation stereo 3-d shape motion estimate stitching together photographs into a mosaic etc 3

4 Toda nteresting points correspondence. Scale and rotation invariant descriptors [Lowe] 4

5 Correspondence using window matching Points are highl individuall ambiguous More unique matches are possible with small regions of image. 5

6 Correspondence using window matching Left Right scanline Criterion function: error disparit 6

7 7 Sum of Squared Piel Differences Left Right L w R w m m w L L w R R L L L L d = + + = ] [ disparit : function of the intensit difference as a The SSD cost measures } { We define the window function : piels. of windows b are corresponding and W v u R L r m m m m m R L m v d u v u d C v u v u W m m w w

8 8 mage Normalization Even when the cameras are identical models there can be differences in gain and sensitivit. The cameras do not see eactl the same surfaces so their overall light levels can differ. For these reasons and more it is a good idea to normalize the piels in each window: Normalized piel ˆ Window magnitude ] [ Average piel 2 1 W W v u W W v u W m m m m m v u v u = = =

9 mages as Vectors Left Right Unwrap image to form vector using raster scan order w L wr m row 1 m w L m row 2 m Each window is a vector in an m 2 dimensional vector space. Normalization makes them unit length. w L row 3 m 9

10 mage windows as vectors 10

11 Possible metrics Distance? w R d w L Angle? 11

12 12 mage Metrics Normalized Sum of Squared Differences 2 2 SSD ] ˆ ˆ [ d w w v d u v u d C R L W v u R L m = = w L w R d Normalized Correlation cosθ ˆ ˆ NC = = = d w w v d u v u d C R L W v u R L m arg ma arg min 2 * d w w d w w d R L d R L d = =

13 Local Features Not all points are equall good for matching 13

14 14

15 15

16 Aperture Problem and Normal Flow 16

17 Aperture Problem and Normal Flow 17

18 Aperture Problem and Normal Flow 18

19 Aperture Problem and Normal Flow 19

20 Aperture Problem and Normal Flow 20

21 Aperture Problem and Normal Flow 21

22 Review Differential approach: Optical flow constraint equation Brightness should sta constant as ou track motion 1 st order Talor series valid for small δt + uδt + vδt t + δt = t t + uδt + vδt + δt = t t Constraint equation u + v + t = 0 BCCE - Brightness Change Constraint Equation 22

23 23 Aperture Problem and Normal Flow 0 0 = = + + U v u t r The gradient constraint: Defines a line in the uv space u v u t = Normal Flow:

24 Combining Local Constraints v 1 1 U = t 2 3 U = U = 2 t 3 t u etc. 24

25 25 Lucas-Kanade: ntegrate gradients over a Patch Ω + + = t v u v u E 2 Assume a single velocit for all piels within an image patch Solve with: = t t v u 2 2 On the LHS: sum of the 22 outer product tensor of the gradient vector t T U = r

26 Local Patch Analsis 26

27 Selecting Good Features What s a good feature? Satisfies brightness constanc Has sufficient teture variation Does not have too much teture variation Corresponds to a real surface patch Does not deform too much over time 27

28 28 Good Features to Track = t t v u 2 2 A u = b When is This Solvable? A should be invertible A should not be too small due to noise eigenvalues λ 1 and λ 2 of A should not be too small A should be well-conditioned λ 1 / λ 2 should not be too large λ 1 = larger eigenvalue Both conditions satisfied when minλ 1 λ 2 > c

29 29 Harris detector W k k W k k k k W k k k k W k k k k k k k k k k 2 2 Auto-correlation matri Auto-correlation matri captures the structure of the local neighborhood measure based on eigenvalues of this matri 2 strong eigenvalues => interest point 1 strong eigenvalue => contour 0 eigenvalue => uniform region nterest point detection threshold on the eigenvalues local maimum for localization

30 Selecting Good Features λ 1 and λ 2 are large 30

31 Selecting Good Features large λ 1 small λ 2 31

32 Selecting Good Features small λ 1 small λ 2 32

33 Toda nteresting points correspondence. Scale and rotation invariant descriptors [Lowe] 33

34 CVPR 2003 Tutorial Recognition and Matching Based on Local nvariant Features David Lowe Computer Science Department Universit of British Columbia

35 nvariant Local Features mage content is transformed into local feature coordinates that are invariant to translation rotation scale and other imaging parameters SFT Features 35

36 Advantages of invariant local features Localit: features are local so robust to occlusion and clutter no prior segmentation Distinctiveness: individual features can be matched to a large database of objects Quantit: man features can be generated for even small objects Efficienc: close to real-time performance Etensibilit: can easil be etended to wide range of differing feature tpes with each adding robustness

37 Res ample Blur Subtra ct Scale invariance Requires a method to repeatabl select points in location and scale: The onl reasonable scale-space kernel is a Gaussian Koenderink 1984; Lindeberg 1994 An efficient choice is to detect peaks in the difference of Gaussian pramid Burt & Adelson 1983; Crowle & Parker 1984 but eamining more scales Difference-of-Gaussian with constant ratio of scales is a close approimation to Lindeberg s scale-normalized Laplacian can be shown from the heat diffusion equation 37

38 Scale space processed one octave at a time 38

39 Res ample Blur Subtra ct Ke point localization Detect maima and minima of difference-of-gaussian in scale space Fit a quadratic to surrounding values for sub-piel and sub-scale interpolation Brown & Lowe 2002 Talor epansion around point: Offset of etremum use finite differences for derivatives: 39

40 Select canonical orientation Create histogram of local gradient directions computed at selected scale Assign canonical orientation at peak of smoothed histogram Each ke specifies stable 2D coordinates scale orientation 0 2π 40

41 Eample of kepoint detection Threshold on value at DOG peak and on ratio of principle curvatures Harris approach a image b 832 DOG etrema c 729 left after peak value threshold d 536 left after testing ratio of principle curvatures 41

42 SFT vector formation Thresholded image gradients are sampled over 1616 arra of locations in scale space Create arra of orientation histograms 8 orientations 44 histogram arra = 128 dimensions 42

43 Feature stabilit to noise Match features after random change in image scale & orientation with differing levels of image noise Find nearest neighbor in database of features 43

44 Feature stabilit to affine change Match features after random change in image scale & orientation with 2% image noise and affine distortion Find nearest neighbor in database of features 44

45 Distinctiveness of features Var size of database of features with 30 degree affine change 2% image noise Measure % correct for single nearest neighbor match 45

46 46

47 47

48 A good SFT features tutorial B Estrada Jepson and Fleet. 48

49 An application of SFT features in m own research 49

50 The couch potato project: Learning from looking at images. Bill Freeman MT Joint work with: Josef Sivic Andrew Zisserman Oford; Bran Russell MT Alosha Efros CMU. December

51 What can ou learn about object categories b simpl looking at images? 51

52 Labelled training databases Labelling object classes in images is tedious and can introduce biases. 52

53 Find words Form histograms Discover topics 53

54 54

55 SFT scale invariant feature transforms David Lowe JCV

56 Visual words Vector quantize SFT descriptors to a vocabular of 2237 visual words. Heuristic design of descriptors makes these words somewhat invariant to: Lighting 2-d Orientation 3-d Viewpoint 56

57 Eamples of visual words 57

58 More visual words 58

59 Polsem the same word with different meanings 59

60 Eperiment E 60

61 Observation matri eperiment E Visual word # Frame # 13.8 % non-zero entries 61

62 Binarized observation matri Visual word # eperiment E Frame # 13.8 % non-zero entries 62

63 63

64 Eample segmentations Faces Motorbikes Airplanes Cars Background Background Background Original images Segmentations All detected visual words

65 65

66 66

67 67

68 68

69 69

70 Faces Motorbikes Airplanes Cars Background Background Background 70

71 71

72 72

73 Faces Motorbikes Airplanes Cars Background Background Background 73

74 74