Image Analysis. Feature extraction: corners and blobs. Motivation: Panorama Stitching (cont.) Motivation: Panorama Stitching (cont.

Transcription

1 Image Analsis Corners and Blobs Feaure eracion: corners and blobs Chrisophoros Nikou Images aken from: Compuer Vision course b Svelana Lazebnik, Universi of Norh Carolina a Chapel Hill (hp:// D. Forsh and J. Ponce. Compuer Vision: A Modern Approach, Prenice Hall, 003. M. Nion and A. Aguado. Feaure Eracion and Image Processing. Academic Press 010. Universi of Ioannina - Deparmen of Compuer Science 3 Moivaion: Panorama Siching 4 Moivaion: Panorama Siching (con.) Sep 1: feaure eracion Sep : feaure maching 5 Moivaion: Panorama Siching (con.) 6 Characerisics of good feaures Sep 1: feaure eracion Sep : feaure maching Sep 3: image alignmen Repeaabili The same feaure can be found in several images despie geomeric and phoomeric ransformaions. Salienc Each feaure has a disincive descripion. Compacness and efficienc Man fewer feaures han image piels. Locali A feaure occupies a relaivel small area of he image; robus o cluer and occlusion. 1

2 7 Applicaions Feaure poins are used for: Moion racking. Image alignmen. 3D reconsrucion. Objec recogniion. Image indeing and rerieval. Robo navigaion. 8 Image Curvaure Eends he noion of edges. Rae of change in edge direcion. Poins where he edge direcion changes rapidl are characerized as corners. We need some elemens from differenial geomer. Parameric form of a planar curve: C () = [ (), ()] Describes he poins in a coninuous curve as he endpoins of a posiion vecor. 9 Image Curvaure (con.) The angen vecor describes changes in he posiion vecor: d d T () = C () = [ (), ()] =, d d Inuiive meaning hink of he race of he curve as he moion of a poin a ime. The angen vecor describes he insananeous moion. 10 Image Curvaure (con.) A an ime momen, he poin moves wih veloci magniude: in he direcion: C () = () + () 1 () ϕ() = an () 11 Image Curvaure (con.) 1 Image Curvaure (con.) The curvaure a a poin C() describes he changes in he direcion of he angen wih respec o changes in arc lengh (consan displacemens along he curve): dϕ() κ () = ds The curvaure is given wih respec o he arc lengh because a curve parameerized b he arc lengh mainains a consan veloci. Gradien direcion N() Edge curve C() dϕ() κ () = ds φ() Tangen T()

3 13 Image Curvaure (con.) 14 Image Curvaure (con.) Parameerizaion of a planar curve b he arc lengh is he lengh of he curve from 0 o : dc() s () = d 0 d This implies: ds() d dc() dc() = d () () d d = = + 0 d d Parameerizaion b he arc lengh is no unique bu i has he proper: dc() dc() dc() dc() dc() d d d = = d = = = 1 ds d ds ds ds dc() d d d 15 Image Curvaure (con.) 16 Image Curvaure (con.) Back o he definiion of curvaure: dϕ() dϕ() d κ () = = ds d ds d 1 1 = an d () + () () () () () κ() = 3/ () + () Some useful relaions from differenial geomer of planar curves (easil deduced) are: dc() = T () ds dc () = κ() N () ds The curvaure is he magniude of he second derivaive of he curve wih respec o he arc lengh: κ () = dc () ds 17 Image Curvaure (con.) 18 Image Curvaure (con.) I is also useful o epress he normal vecor a a poin o a curve in Caresian coordinaes: () () N () =, T () = [ (), ()] () + () () + () There are hree main approaches o compue curvaure Direc compuaion on an edge image. Derive he measure of curvaure from image inensiies. Correlaion. () () () () κ() = 3/ () + () 3

4 19 Image Curvaure (con.) 0 Image Curvaure (con.) Direc compuaion from edges. Difference in edge direcion. wih κ() = ϕ( + 1) ϕ( 1) 1 ( + 1) ( 1) ϕ() = an ( + 1) ( 1) Conneced edges are needed (hseresis). Direc compuaion from edges. Smoohing is generall required b considering more han wo piels: n 1 κ () = ϕ ( i ) ϕ ( + i 1) n i= 1 No ver reliable resuls. Reformulaion of a firs order edge deecion scheme. Quanizaion errors in angle measuremen. Threshold o deec corners. 1 Image Curvaure (con.) Image Curvaure (con.) Direc compuaion from edges. Objec silhouee Thresholded curvaure The resul depends srongl on he hreshold. Compuaion from image inensi. I should be compued along he curve (normal o he image gradien) for each piel. Caresian coordinaes for he angle of he angen φ(,). Measure of angular changes in he image wih respec o locaion. The curve a an image poin ma be approimaed b: ( ϕ ) ( ϕ ) () = ( 1) + cos (, ) () = ( 1) + sin (, ) 3 Image Curvaure (con.) 4 Image Curvaure (con.) Compuaion from image inensi. The curvaure is given b: ϕ(, ) ϕ(, ) ( ) ϕ(, ) ( ) κϕ (, ) = = + wih () () = cos ( ϕ(, ) ), = sin ( ϕ(, ) ), 1 M ϕ(, ) = an M Normal o he curve. Recall ha his is associaed wih he C ss (). Compuaion from image inensi. Subsiuing he all he erms: κ ϕ 1 M M M M (, ) = M 3/ MM + M MM ( M + M ) Alernaivel, b differeniaing backwards: 1 M M M M κ ϕ (, ) = M 3/ MM M + MM ( M + M ) 4

5 5 Image Curvaure (con.) 6 Image Curvaure (con.) Compuaion from image inensi. Oher measures differeniae along he normal o he curve. The idea is ha curves ma be hicker han one piel wide. Differeniaing along he normal measures he difference beween inernal and eernal gradien angles. Theoreicall, hese are equal. However, in pracice he differ due o image discreizaion. The more he edge is ben, he larger he difference (Kass e al. IJCV 1988). Compuaion from image inensi. The measures are: 1 M M M M κ ϕ (, ) = M 3/ MM MM + MM ( M + M ) 1 M M M M κ ϕ (, ) = M 3/ + MM MM + M ( M + M ) 7 Image Curvaure (con.) Compuaion from image inensi. 8 Finding Corners Image Curvaure b Correlaion κϕ κ ϕ κ ϕ κ ϕ Ke proper: in he region around a corner, image gradien has wo or more dominan direcions. Corners are repeaable and disincive. Beer han direc compuaion bu he resuls are no consisen. C.Harris and M.Sephens. "A Combined Corner and Edge Deecor. Proceedings of he 4h Alve Vision Conference: pages The Basic Idea We should easil recognize he poin b looking hrough a small window. Shifing a window in an direcion should give a large change in inensi. 30 Harris Corner Deecor Change in appearance for he shif [u,v]:, [ ] E( uv, ) = w (, ) I( + u, + v) I(, ) Window funcion Shifed hf inensi Inensi fla region: no change in all direcions edge : no change along he edge direcion corner : significan change in all direcions Window funcion w(,) 1 in window, 0 ouside or Gaussian Source: A. Efros Source: R. Szeliski 5

6 31 Harris Deecor (con.) Change in appearance for he displacemen [u,v]:, [ ] E( uv, ) = w (, ) I( + u, + v) I(, ) Second-order Talor epansion of E(u,v) around (0,0): Eu (0,0) 1 Euu (0,0) Euv (0,0) u E( u, v) E(0,0) + [ u v] + [ u v] Ev (0,0) Euv (0,0) Evv(0,0) v 3 Harris Deecor (con.), [ ] E( uv, ) = w (, ) I( + u, + v) I(, ) As E(0,0)= 0, he higher order erms ield: E I( + u, + v) E u = = w (, ) [ I ( + u, + v ) I (, ) ] u u,,, I ( + u, + v) ( + u) = w (, )[ I( + u, + v) I(, )] = ( + u) u [ ] = w (, ) I( + u, + v) I(, ) I( + u, + v) A he origin: E u (0,0) = 0 33 Harris Deecor (con.) 34 Harris Deecor (con.) E Eu = = w (, )[ I ( + u, + v) I (, )] I( + u, + v) u, The second order erm is: E E Euu = = u u u = wi (, ) ( + u, + vi ) ( + u, + v), [ ] + I ( + u, + v) w(, ) I( + u, + v) I(, ) A he origin: E w I uu (0,0) = (, ) (, ) B he same reasoning we obain he res of he derivaives: E w I uu (0,0) = (, ) (, ) E w I vv(0,0) = (, ) (, ) E (0,0) = E (0,0) = w (, ) I (, ) I (, ) uv vu And he final epression becomes: 1 Euu (0,0) Euv (0,0) u Euv (, ) [ u v] Euv(0,0) Evv (0,0) v I II u Euv (, ) [ u v] w (, ), II I v 35 Harris Deecor (con.) 36 Harris Deecor (con.) The bilinear approimaion simplifies o u E ( u, v) [ u v] M v where M is a mari compued from image derivaives: I II M = w(, ), II I M The surface E(u,v) is locall approimaed b a quadraic form. Le s r o undersand is shape. E ( u, v ) [ u v ] M I I I M = I I I u v 6

7 37 Harris Deecor (con.) 38 Harris Deecor (con.) Firs, consider he ais-aligned case where gradiens are eiher horizonal or verical. M = I I I I I λ 0 = 1 I 0 λ If eiher λ is close o 0, hen his is no a corner, so look for locaions where boh are large. Since M is smmeric, i can be wrien as: M R λ1 0 0 R λ = 1 We can visualize M as an ellipse wih aes lenghs deermined b he eigenvalues and orienaion deermined b he roaion mari R. Ellipse equaion: u [ u v] M = cons v direcion of he fases change (λ ma ) -1/ (λ min ) -1/ direcion of he slowes change 39 Visualizaion of second momen marices 40 Visualizaion of second momen marices (con.) 41 Window size maers! 4 Inerpreing he eigenvalues Classificaion of image poins using eigenvalues of M : λ Edge λ >> λ 1 Corner λ 1 and λ are large, λ 1 ~ λ ; E increases in all direcions λ 1 and λ are small; E is almos consan in all direcions Fla region Edge λ 1 >> λ λ 1 7

8 43 Corner response funcion R = de( M ) M α: consan (0.04 o 0.06) α race( ) = λ1λ α( λ1 + Edge R < 0 Fla region R small Corner R > 0 λ ) Edge R < 0 44 Harris deecor: Seps 1. Compue Gaussian derivaives a each piel.. Compue second momen mari M in a Gaussian window around each piel. 3. Compue corner response funcion R. 4. Threshold R. 5. Find local maima of response funcion (nonmaimum suppression). 45 Harris Deecor: Seps (con.) 46 Harris Deecor: Seps (con.) Compue corner response R 47 Harris Deecor: Seps (con.) 48 Harris Deecor: Seps (con.) Find poins wih large corner response: R>hreshold Take onl he poins of local maima of R 8

9 49 Harris Deecor: Seps (con.) 50 Invariance Feaures should be deeced despie geomeric or phoomeric changes in he image: if we have wo ransformed versions of he same image, feaures should be deeced in corresponding locaions. 51 Models of Image Transformaion 5 Harris Deecor: Invariance Properies Geomeric Roaion Roaion Scale Affine valid for: orhographic camera, locall planar objec Phoomeric Affine inensi change (I a I + b) The ellipse roaes bu is shape (i.e. eigenvalues) remains he same. Corner response R is invarian o image roaion 53 Harris Deecor: Invariance Properies (con.) Affine inensi change 54 Scaling Harris Deecor: Invariance Properies (con.) Onl derivaives are used => invariance o inensi shif I I + b Inensi scale: I a I R hreshold (image coordinae) R (image coordinae) Corner All poins will be classified as edges Pariall invarian o affine inensi change No invarian o scaling 9

10 55 Harris Deecor: Invariance Properies (con.) Harris corners are no invarian o scaling. This is due o he Gaussian derivaives compued a a specific scale. If he image differs in scale he corners will be differen. For scale invariance, i is necessar o deec feaures ha can be reliabl eraced under scale changes. 56 Scale-invarian feaure deecion Goal: independenl deec corresponding regions in scaled versions of he same image. Need scale selecion mechanism for finding characerisic region size ha is covarian wih he image ransformaion. Idea: Given a ke poin in wo images deermine if he surrounding neighborhoods conain he same srucure up o scale. We could do his b sampling each image a a range of scales and perform comparisons a each piel o find a mach bu i is impracical. 57 Scale-invarian feaure deecion (con.) Evaluae a signaure funcion and plo he resul as a funcion of he scale. The shape should be similar in differen scales. 58 Scale-invarian feaure deecion (con.) The onl operaor fulfilling hese requiremens is a scale-normalized Gaussian. T. Lindeberg. Scale space heor: a basic ool for analzing srucures a differen scales. Journal of Applied Saisics, 1(), pp. 4 70, Scale-invarian feaure deecion (con.) Based on he above idea, Lindeberg (1998) proposed a deecor for blob-like feaures ha searches for scale space erema of a scalenormalized LoG. 60 Scale-invarian feaure deecion (con.) T. Lindeberg. Feaure deecion wih auomaic scale selecion. Inernaional Journal of Compuer Vision, 1(), pp. 4 70,

11 61 Recall: Edge Deecion 6 Recall: Edge Deecion (con.) f Edge f Edge d g d Derivaive of Gaussian Second derivaive d g of Gaussian d (Laplacian) d f d g Edge = maimum of derivaive d g d f Edge = zero crossing of second derivaive Source: S. Seiz Source: S. Seiz 63 From edges o blobs Edge = ripple. Blob = superposiion of wo edges (wo ripples). 64 Scale selecion We wan o find he characerisic scale of he blob b convolving i wih Laplacians a several scales and looking for he maimum response. However, Laplacian response decas as scale increases: maimum Spaial selecion: he magniude of he Laplacian response will achieve a maimum a he cener of he blob, provided he scale of he Laplacian is mached o he scale of he blob. original signal (radius=8) increasing σ Wh does his happen? 65 Scale normalizaion The response of a derivaive of Gaussian filer o a perfec sep edge decreases as σ increases. 1 σ π 66 Scale normalizaion (con.) The response of a derivaive of Gaussian filer o a perfec sep edge decreases as σ increases. To keep he response he same (scaleinvarian), we mus mulipl he Gaussian derivaive b σ. The Laplacian is he second derivaive of he Gaussian, so i mus be muliplied b σ. 11

12 67 Effec of scale normalizaion 68 Blob deecion in D Original signal Unnormalized Laplacian response Laplacian: Circularl smmeric operaor for blob deecion in D. Scale-normalized Laplacian response g g g = + maimum 69 Blob deecion in D (con.) Laplacian: Circularl smmeric operaor for blob deecion in D. 70 Scale selecion A wha scale does he Laplacian achieve a maimum response for a binar circle of radius r? r Scale-normalized: g g g = σ + norm image Laplacian 71 Scale selecion (con.) The D LoG is given (up o scale) b ( + σ ) e ( + )/ σ For a binar circle of radius r, he LoG achieves a maimum a σ = r / 7 Characerisic scale We define he characerisic scale as he scale ha produces a peak of he LoG response. r image Laplacian response r / scale (σ) characerisic scale T. Lindeberg (1998). "Feaure deecion wih auomaic scale selecion." Inernaional Journal of Compuer Vision, 30 (): pp

13 73 Scale-space blob deecor 74 Scale-space blob deecor: Eample 1. Convolve he image wih scalenormalized LoG a several scales.. Find he maima of squared LoG response in scale-space. 75 Scale-space blob deecor: Eample 76 Scale-space blob deecor: Eample 77 Efficien implemenaion 78 Efficien implemenaion Approimaing he LoG wih a difference of Gaussians: ( (,, ) (,, )) L= G + G (Laplacian) σ σ σ DoG = G(,, kσ ) G(,, σ ) (Difference of Gaussians) We have sudied his opic in edge deecion. Divide each ocave ino an equal number K of inervals such ha: 1/ k = K, σ n n = k σ0 n= 1,..., K. Implemenaion b a Gaussian pramid. D. G. Lowe. "Disincive image feaures from scale-invarian kepoins. Inernaional Journal of Compuer Vision 60 (), pp ,

14 79 The Harris-Laplace deecor I combines he Harris operaor for corner-like srucures wih he scale space selecion mechanism of DoG. Two scale spaces are buil: one for he Harris corners and one for he blob deecor (DoG). A ke poin is a Harris corner wih a simulaneousl maimun DoG a he same scale. I provides fewer ke poins wih respec o DoG due o he consrain. K. Mikolajczk and C. Schmid, Scale and Affine invarian ineres poin deecors, Inernaional Journal of Compuer Vision, 60(1):63-86, From scale invariance o affine invariance For man problems, i is imporan o find feaures ha are invarian under large viewpoin changes. The projecive disorion ma no be correced locall due o he small number of piels. A local affine approimaion is usuall sufficien. 81 From scale invariance o affine invariance (con.) 8 Affine Adapaion Recall: M = I I I λ 1 1 w(, ) = R I I I, 0 0 R λ We can visualize M as an ellipse wih ais lenghs deermined b he eigenvalues and orienaion deermined b R. Affine adapaion of scale invarian deecors. Find local regions where an ellipse can be reliabl and repeaedl eraced purel from local image properies. Ellipse equaion: u [ u v] M = cons v direcion of he fases change (λ ma ) -1/ (λ min ) -1/ direcion of he slowes change 83 Affine adapaion eample 84 Affine adapaion eample Scale-invarian regions (blobs) Affine-adaped blobs 14

15 85 Affine adapaion The covaring of Harris corner deecor ellipse ma be viewed as he characerisic shape of a region. We can normalize he region b ransforming he ellipse ino a uni circle. 86 Affine adapaion (con.) Problem: he second momen window deermined b weighs w(,) mus mach he characerisic shape of he region. Soluion: ieraive approach Use a circular window o compue he second momen mari. Based on he eigenvalues, perform affine adapaion o find an ellipse-shaped window. Recompue second momen mari in he ellipse and ierae. The normalized regions ma be deeced under an affine ransformaion. 87 Ieraive affine adapaion 88 Orienaion ambigui There is no unique ransformaion from an ellipse o a uni circle We can roae or flip a uni circle, and i sill sas a uni circle. K. Mikolajczk and C. Schmid, Scale and Affine invarian ineres poin deecors, Inernaional Journal of Compuer Vision, 60(1):63-86, 004. hp:// 89 Orienaion ambigui (con.) 90 Summar: Feaure eracion We have o assign a unique orienaion o he kepoins in he circle: Creae he hisogram of local gradien direcions in he pach. Assign ho he pach he orienaion of he peak of he smoohed hisogram. Erac affine regions Normalize regions Eliminae roaional ambigui Compue appearance descripors SIFT (Lowe 04) 0 π 15

16 91 Invariance vs. covariance Invariance: feaures(ransform(image)) = feaures(image) Covariance: feaures(ransform(image)) = ransform(feaures(image)) Covarian deecion => invarian descripion 16