Perception III: Filtering, Edges, and Point-features

Transcription

1 Perception : Filtering, Edges, and Point-features Davide Scaramuzza Universit of Zurich Margarita Chli, Paul Furgale, Marco Hutter, Roland Siegwart 1

2 Toda s outline mage filtering Smoothing Edge detection Point-feature etraction Harris corners SFT features ntroduction

3 mage filtering The word filter comes from signal processing, where filtering refers to the process of accepting or rejecting certain frequenc components Low-pass filtering: retains lowfrequenc components (smoothing) High-pass filtering: retains high-frequenc components (edge detection)

4 Smoothing - Motivation: noise reduction Common tpes of noise Salt and pepper noise: random occurrences of black and white piels mpulse noise: random occurrences of white piels Gaussian noise: variations in intensit drawn from a Gaussian normal distribution

5 Gaussian noise This image shows the noise values added to the raw intensities of an image using σ = 16 Add Gaussian independent and identicall disributed (white) noise: N(μ, σ) out = in + N(μ, σ) How can we reduce the noise?

6 Moving average Let s replace each piel with an average of all the values in its neighborhood Assumptions: Epect piels to be like their neighbors Epect noise processes to be independent from piel to piel

7 Moving average Let s replace each piel with an average of all the values in its neighborhood Moving Average in 1D

8 Weighted Moving Average Can add weights to our moving average Weights: [1, 1, 1, 1, 1] / 5

9 Weighted Moving Average Can add weights to our moving average Non-uniform weights: [1, 4, 6, 4, 1] / 16

10 Moving Average n D F(, ) G(, )

11 Moving Average in D F(, ) G(, )

12 Moving Average n D F(, ) G(, )

13 Moving Average n D F(, ) G(, )

14 Moving Average n D F(, ) G(, )

15 Moving Average n D F(, ) G(, )

16 Correlation The averaging window size is (k + 1) (k + 1): This is called cross-correlation, denoted b: Filtering an image: replace each piel with a linear combination of its neighbors. H The filter kernel or mask H[u, v] specifies the weights of the linear combination. F

17 Averaging filter What values belong in the kernel H for the moving average eample? = ? bo filter

18 Smoothing b averaging Bo filter (black = 0) original filtered

19 Gaussian filter What if we want nearest neighboring piels to have the most influence on the output? This kernel is an approimation of a Gaussian function:

20 Smoothing with a Gaussian Compare with moving average

21 Sample Matlab code >> hsize = 10; >> sigma = 5; >> h = fspecial( gaussian hsize, sigma); >> mesh(h); >> imagesc(h); >> im = imread( panda.jpg ); >> outim = imfilter(im, h); >> imshow(outim); im outim

22 Separable filters A filter is separable when it can be written as the product of two 1D vectors. This reduces the costs of convolution the image the filtered image can be obtained as a cascade of the two 1D convolution operators! Eample: constant averaging filter Eample : Gaussian filter Consider an m m image and an n n separable filter D convolution number of multiplications per piel = n 1D Correlation no. multiplications per piel = n H separable filters H

23 Boundar issues What about near the edge? the filter window falls off the edge of the image need to pad the image borders methods: zero padding (black) wrap around cop edge reflect across edge

24 Convolution Flip the filter in both dimensions (bottom to top, right to left) Then appl cross-correlation G = H F H Notation for convolution operator 180 deg turn F

25 Properties of convolution Linear & shift invariant Commutative: f h = h f Associative (f g) h = f (g h) dentit: unit impulse e = [, 0, 0, 1, 0, 0, ]. f e = f Differentiation: h f = h f

26 Edge Detection Ultimate goal of edge detection: an idealized line drawing. Edge contours in the image correspond to important scene contours.

27 Edge = intensit discontinuit in one direction Edges correspond to sharp changes of intensit How to detect an edge?

28 Edge = intensit discontinuit in one direction Edges correspond to sharp changes of intensit How to detect an edge? Change is measured b 1 st order derivative in 1D Big intensit change magnitude of derivative is large Or nd order derivative is zero.

29 1D Edge Detection Effect of Noise Consider a single row or column of the image () d d () Where is the edge? image noise cannot be ignored

30 Solution: smooth first () () G s( ) ( ) G ( ) d s( ) s( ) d Where is the edge? Edges occur at maima/minima of s()

31 Derivative Theorem of Convolution d d ) s( ) G ( ) ( ) G ( ) ( This saves us one operation: () G d ( ) G ( ) d s( ) G ( ) ( ) Edges occur at maima/minima of s()

32 Zero-crossings Locations of Maima/minima in s() are equivalent to zero-crossings in s () () G d ( ) d G ( ) : Laplacian of Gaussian operator s ( ) G ( ) ( ) Edges occur at zero-crossings of s ( )

33 D Edge Detection Find gradient of smoothed image in both directions Discard piels with (i.e. edge strength), below a certain below a certain threshold Non-maima suppression: identif local maima of detected edges G G G G G S S S

34 D Edge detection: Eample : original image (Lena image)

35 D Edge detection: Eample S G G G S : Edge strength

36 D Edge detection: Eample Thresholding S

37 D Edge detection: Eample Thinning: non-maimal suppression edge image

38 Partial derivatives of an image (, ) (, )

39 Popular 1 st -Derivative Filters Sample Matlab code >> im = imread( lion.jpg ) >> M = fspecial( sobel ); >> outim = imfilter(double(im), M); >> imagesc(outim); >> colormap gra;

40 Derivative of Gaussian filter ( g) h ( g h)

41 Derivative of Gaussian filters -direction -direction

42 D edge detection filters Laplacian of Gaussian Gaussian derivative of Gaussian is the Laplacian operator: h = h h +

43 Summar on filters Smoothing Values positive Sum to 1 constant regions same as input Amount of smoothing proportional to mask size Remove high-frequenc components; low-pass filter Derivatives Opposite signs used to get high response in regions of high contrast Sum to 0 no response in constant regions High absolute value at points of high contrast

44 Point Features This panorama was generated using AUTOSTTCH (freeware) Build our own:

45 Point Features Harris corners SFT features and more recent algorithms from the state of the art

46 Applications Point features are widel used in: Robot navigation Object recognition 3D reconstruction Visual odometr ndeing and database retrieval Google mages mage stitching: see eample

47 Applications Point features are widel used in: Robot navigation Object recognition 3D reconstruction Visual odometr ndeing and database retrieval Google mages mage stitching: see eample

48 How to build a panorama? We need to match (align) images

49 How to build a panorama? Detect feature points in both images

50 How to build a panorama? Detect feature points in both images Find corresponding pairs

51 How to build a panorama? Detect feature points in both images Find corresponding pairs Use these pairs to align images

52 Matching with Features Problem 1: Detect the same points independentl in both images, if the are in the field of view no chance to match! We need a repeatable feature detector

53 Matching with Features Problem : For each point, identif its correct correspondence in the other image(s)? We need a reliable and distinctive feature descriptor

54 Template matching Find locations in an image that are similar to a template f we look at filters as templates, we can use correlation to detect these locations Detected template Correlation map

55 Similarit measures Sum of Squared Differences (SSD) Sum of Absolute Differences (SAD) (used in optical mice) k k u k k v v u F v u H SSD ), ( ), ( k k u k k v v u F v u H SAD ), ( ), (

56 Similarit measures For slight invariance to intensit changes, the Zero-mean Normalized Cross Correlation (ZNCC) is widel used k k u k k v F k k u k k v H k k u k k v F H v u F v u H v u F v u H ZNCC ), ( ), ( ), ( ), ( 1 ), ( 1 ), ( N v u F N v u H k k u k k v F k k u k k v H

57 Correlation as an inner product Considering the filter H and the portion of the image F as vectors their correlation is: H, F H F cos H F Can ou think of a similar interpretation for the SSD? n ZNCC we consider the unit vectors of H and F, hence we measure their similarit based on the angle. Alternativel, ZNCC maimizes cosθ cos H, H F F k ukvk ukvk k k k H ( u, v) F ( u, v) k H ( u, v) F( u, v) H H k ukvk F F

58 Harris Corner Detector C.Harris and M.Stephens. A Combined Corner and Edge Detector. 1988

59 What is a distinctive feature? mage pairs with etracted patches below Notice how some patches can be localized or matched with higher accurac than others Ke propert: in the region around a corner, image gradient has two or more dominant directions Corners are repeatable and distinctive

60 dentifing Corners How do we identif corners? Shifting a window in an direction should give a large change in intensit in at least directions flat region: no intensit change edge : no change along the edge direction corner : significant change in at least directions

61 How do we implement this? Two image patches of size P one centered at and one centered at The Sum of Squared Differences between them is: Let and. Approimating with a 1 st order Talor epansion: This produces the approimation ), ( P SSD, ), ( ), ( ), ( ), ( ), ( ), ( ), ( ), ( ), ( ), ( P SSD, ) ), ( ), ( ), (

62 How do we implement this? This can be written in a matri form as P SSD, ) ), ( ), ( ), ( M SSD ), ( SSD ), (, P M

63 How do we implement this? This can be written in a matri form as P SSD, ) ), ( ), ( ), ( M SSD ), ( SSD ), ( Alternative wa to write this matri, P M

64 How do we implement this? This can be written in a matri form as P SSD, ) ), ( ), ( ), ( M SSD ), ( ] [, P M SSD ), ( Alternative was to write this matri nd moment matri

65 What does this matri reveal? First, consider an ais-aligned corner: This means dominant gradient directions align with or ais f either λ 1 or λ is close to 0, then this is not a corner: What if we have a corner that is not aligned with the image aes? M M M Corner Edge Flat region

66 General Case 0 R Since M is smmetric, it can alwas be decomposed into R M const We can visualize M as an ellipse with ais lengths determined b the eigenvalues and the two aes orientations determined b R (i.e., the eigenvectors of M) The two eigenvectors identif the directions of largest and smallest changes of SSD direction of the largest change of SSD ( ma ) -1/ ( min) -1/ direction of the smallest change of SSD

67 nterpreting the eigenvalues Classification of image points using eigenvalues of M A corner can then be identified b checking whether the minimum of the two eigenvalues of M is larger than a certain user-defined threshold R = min(, ) > threshold R is called cornerness function The corner detector using this criterion is called «Shi-Tomasi» detector Edge >> 1 Corner 1 and are large, R = min(, ) > threshold J. Shi and C. Tomasi (June 1994). "Good Features to Track,". EEE Conference on Computer Vision and Pattern Recognition SSD increases in all directions 1 and are small; SSD is almost constant in all directions Flat region Edge 1 >> 1

68 nterpreting the eigenvalues Computation of λ 1 and λ is epensive Harris & Stephens suggested using a different cornerness function: R 1 k( 1 ) det( M) k trace ( M) k is constant in the range (0.04 to 0.15) Edge >> 1 Corner 1 and are large, R = min(, ) > threshold 1 and are small; SSD is almost constant in all directions SSD increases in all directions Flat Edge region 1 >> 1

69 Harris Detector: Workflow

70 Harris Detector: Workflow Compute corner response C

71 Harris Detector: Workflow Find points with large corner response: C > threshold

72 Harris Detector: Workflow Take onl the points of local maima of thresholded C

73 Harris Detector: Workflow

74 Harris detector: properties Repeatabilit: How does the Harris detector behave to common image transformations? Will we be able to re-detect the same image patches (Harris corners) when the image ehibits changes in Rotation, View-point, Zoom, llumination? dentif properties of detector & adapt accordingl

75 Harris Detector: Some Properties Rotation invariance mage 1 mage Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response C is invariant to image rotation

76 Harris Detector: Some Properties But: non-invariant to image scale! mage 1 mage All points will be classified as edges Corner!

77 Harris Detector: Some Properties Qualit of Harris detector for different scale changes Repeatabilit= # correspondences detected # correspondences present Scaling the image b ~0% of correspondences get detected

78 Summar on Harris properties Harris detector: probabl the most widel used and known corner detector The detection is invariant to Rotation Linear intensit changes note: to make the matching invariant to these we need a suitable descriptor and matching criterion (e.g. SSD on patches is not rotation- or affine- invariant) The detection is NOT invariant to Scale changes Geometric affine changes: an image transformation which distorts the neighborhood of the corner, can distort its cornerness response

79 Scale nvariant Detection Consider regions (e.g. discs) of different sizes around a point Aim: corresponding regions look the same in image space, when the appropriate scale-change is applied Choose corresponding regions (discs) independentl in each image

80 Scale nvariant Detection Approach: design a function to appl on the region (disc), which is scale invariant (i.e. remains constant for corresponding regions, even if the are at different scales) eample: average image intensit over corresponding regions (at different scales) should remain constant Average intensit value enclosed in each disc, as a function of the disc-radius: mage 1 scale = 1/ mage region size region size

81 Scale nvariant Detection dentif the local maimum in each response these occur at corresponding region sizes The corresponding scale-invariant region size is found in each image independentl! mage 1 scale = 1/ mage region size s 1 s region size

82 Scale nvariant Detection A good function for scale detection has one clear, sharp peak bad bad Good! region size region size region size Sharp, local intensit changes in an image, are good regions to monitor for identifing relative scale in usual images. look for blobs or corners

83 Scale nvariant Detection Functions for determining scale: convolve image with kernel to identif sharp intensit discontinuities f Kernel mage Laplacian of Gaussian kernel: G(, LoG G(, ) ) ) Approimation: Difference of Gaussians (DoG) kernel SFT detector [Lowe et.al., JCV04] DoG Gk (, ) G (, ) G(, Note: These kernels are invariant to scale and rotation

84 LoG for Scale invariant detection Response of LoG for corresponding regions Harris-Laplacian multi-scale detector

85 Scale nvariant Detectors Eperimental evaluation of detectors w.r.t. scale change Repeatabilit= # correspondences detected # correspondences present

86 SFT features SFT: Scale nvariant Feature Transform an approach for detecting and describing regions of interest in an image (developed b D. Lowe in 004) SFT features are reasonabl invariant to changes in: rotation, scaling, small changes in viewpoint, illumination Ver powerful in capturing + describing distinctive structure, but also computationall demanding Main SFT stages: 1. Etract kepoints + scale. Assign kepoint orientation 3. Generate kepoint descriptor

87 Subsample SFT detector (kepoint location + scale) 1. Scale-space pramid: subsample and blur original image. Difference of Gaussians (DoG) pramid: subtract successive smoothed images Blur 3. Kepoints: local etrema in the DoG pramid DoG:

88 SFT: assign kepoint orientation Define orientation of kepoint to achieve rotation invariance Sample intensities around the kepoint Compute a histogram of orientations of intensit gradients Peaks in histogram: dominant orientations Kepoint orientation = histogram peak f there are multiple candidate peaks, construct a different kepoint for each such orientation 0

89 SFT descriptor Descriptor : identit card of kepoint Simplest descriptor: matri of intensit values around a kepoint (image patch) deall, a descriptor should be highl distinctive tolerant/invariant to common image transformations SFT descriptor: 18-element vector Describe all gradient orientations relative to the kepoint orientation Divide kepoint neighborhood in 4 4 regions & compute orientation histograms along 8 directions SFT descriptor: concatenation of all (=18) values Descriptor Matching: L -distance between these descriptor vectors

90 SFT kepoints Final SFT kepoints with detected orientation & scale

91 Feature stabilit to view point change

92 SFT for Planar recognition Planar surfaces can be reliabl recognized at a rotation of 60 awa from the camera Onl 3 points are needed for recognition But objects need to have enough teture Recognition under occlusion

93 Code and Demos SFT feature detector Demo: for Matlab, Win, and Linu (freeware) Make our own panorama with AUTOSTTCH (freeware):

94 More recent features from SOTA suitable for Robotics applications

95 FAST detector [Rosten et al., PAM 010] FAST: Features from Accelerated Segment Test Studies intensit of piels on circle around candidate piel C C is a FAST corner if a set of N contiguous piels on circle are: all brighter than intensit_of(c)+theshold, or all darker than intensit_of(c)+theshold C Tpical FAST mask: test for 1 contiguous piels in a 16-piel circle Ver fast detector - in the order of 100 Mega-piel/second Based on slide b S. Leutenegger

96 BREF descriptor [Calonder et. al, ECCV 010] Binar Robust ndependent Elementar Features Goal: high speed (in description and matching) Binar descriptor formation: Smooth image for each detected kepoint (e.g. FAST), sample all intensit pairs ( 1, ) (tpicall 56 pairs) according to pattern around the kepoint for each pair p if 1 < then set bit p of descriptor to 1 else set bit p of descriptor to 0 Not scale/rotation invariant (etensions eist ) Allows ver fast Hamming Distance matching: count the number of bits that are different in the descriptors matched Pattern for intensit pair samples generated randoml Based on slide b S. Leutenegger

97 BRSK descriptor Detector based on FAST Binar descriptor, formed b pairwise intensit comparisons (like BREF) Pattern defines intensit comparisons in the kepoint neighborhood Red circles: size of the smoothing kernel applied Blue circles: smoothed piel value used Compare short- and long-distance pairs for orientation assignment and descriptor formation Detection and descriptor speed: 10 times faster than SURF (and even faster than SFT) Slower than BREF, but scale- and rotation- invariant BRSK sampling pattern Based on slide b S. Leutenegger

98 BRSK in action Open-source code for FAST, BREF, BRSK and man more, available at the OpenCV librar