Image Analysis. Active contour models (snakes)

Transcription

1 Image Analyss Actve contour models (snakes) Chrstophoros Nkou Images taken from: Computer Vson course by Krsten Grauman, Unversty of Texas at Austn. Unversty of Ioannna - Department of Computer Scence and Engneerng

2 Deformable contours a.k.a. actve contours, snakes Gven: ntal contour (model) near desred object [Snakes: Actve contour models, Kass, Wtkn, & Terzopoulos, ICCV987] Fgure credt: Yur Boykov

3 3 Deformable contours (cont.) Gven: ntal contour (model) near desred object Goal: evolve the contour to ft exact object boundary Man dea: elastc band s teratvely adjusted so as to be near mage postons wth hgh gradents, and satsfy shape preferences or contour prors [Snakes: Actve contour models, Kass, Wtkn, & Terzopoulos, ICCV987]

4 4 Deformable contours (cont.) Image from

5 5 Deformable contours vs Hough Both are useful for shape fttng but they dffer sgnffcanlty: ntal ntermedate fnal Hough Rgd model shape Sngle votng pass can detect multple nstances Deformable contours Pror on shape types, but shape teratvely adjusted (deforms) Requres ntalzaton nearby One optmzaton pass to ft a sngle contour

6 6 Deformable contours (why?) Some objects have smlar basc form but some varety n the contour shape.

7 7 Deformable contours (why?) Non-rgd, deformable objects can change ther shape over tme, e.g. lps, hands

8 8 Deformable contours (why?) Non-rgd, deformable objects can change ther shape over tme, e.g. lps, hands

9 9 Deformable contours (why?) Non-rgd, deformable objects can change ther shape over tme.

10 0 Aspects to consder Representaton of the contours Defnng the energy functons External Internal Mnmzng the energy functon Extensons: Trackng Interactve segmentaton

11 Aspects to consder Representaton of the contours Defnng the energy functons External Internal Mnmzng the energy functon Extensons: Trackng Interactve segmentaton

12 Representaton We ll consder a dscrete representaton of the contour, consstng of a lst of D pont postons ( vertces ). ( x 0, y0 ) ( x, y 9 9) ( x, y ), for 0,,, n At each teraton, we ll have the opton to move each vertex to another nearby locaton ( state ).

13 3 Fttng deformable contours How should we adjust the current contour to form the new contour at each teraton? Defne a cost functon ( energy functon) that says how good a canddate confguraton s. Seek next confguraton that mnmzes that cost functon. ntal ntermedate fnal

14 4 Aspects to consder Representaton of the contours Defnng the energy functons External Internal Mnmzng the energy functon Extensons: Trackng Interactve segmentaton

15 5 Energy functon The total energy (cost) of the current snake s defned as: E total E nternal E external Internal energy: encourages pror shape preferences: e.g., smoothness, elastcty, partcular known shape. External energy ( mage energy): encourages contour to ft on places where mage structures exst, e.g., edges. A good ft between the current deformable contour and the target shape n the mage wll yeld a low value for ths cost functon.

16 6 External energy It measures how well the curve matches the mage data It Attracts the curve toward dfferent mage features Edges, lnes, texture gradent, etc.

17 7 External energy (cont.) How do edges affect the shape of the rubber band? Thnk of external energy from mage as gravtatonal pull towards areas of hgh contrast Magntude of gradent - (Magntude of gradent) G ( I) G ( I) x y G ( I) G ( I) x y

18 8 External energy (cont.) G x ( x, y) ( x, y) Gradent mages and G y External energy at a pont on the curve s: E external ( ) ( G x ( ) G y ( ) ) External energy for the whole curve: E external n Gx( x, y ) Gy ( x, y 0 )

19 9 Dstance transform External mage can nstead be taken from the dstance transform of the edge mage. orgnal - gradent edges Dstance transform Value at (x,y) ndcates how far that poston s from the nearest edge pont

20 0 Internal energy: ntuton What are the underlyng boundares n ths fragmented edge mage? And n ths one?

21 Internal energy: ntuton (cont.) A pror, we want to favor smooth shapes, contours wth low curvature, contours smlar to a known shape, etc. to balance what s actually observed (.e., n the gradent mage).

22 Internal energy For a contnuous curve, a common nternal energy term s the bendng energy. At some pont v(s) on the curve, ths s: d d E nternal ( ( s)) ds d s Tenson, Elastcty Stffness, Curvature

23 3 For our dscrete representaton, Internal energy for the whole curve: 0 ), ( n y x v ds d ) ( ) ( ds d 0 n nternal E Note these are dervatves relatve to poston---not spatal mage gradents. Why do these reflect tenson and curvature? Internal energy

24 4 Example (curvature) E curvature( v ) (,5) (,) ( x x x ) ( y y y ) (,) (3,) (,) (3,)

25 5 Penalze elastcty Current elastc energy defnton uses a dscrete estmate of the dervatve: Eelastc n 0 n ( x x ) ( y y ) 0 What s the possble problem wth ths defnton?

26 6 Penalze elastcty (cont.) Current elastc energy defnton uses a dscrete estmate of the dervatve: Eelastc n 0 Instead, we may employ: n 0 ( x x ) ( y y d ) where d s the average dstance between pars of ponts updated at each teraton.

27 7 Mssng data The preferences for low-curvature, smoothness help deal wth mssng data: Illusory contours found!

28 8 Extend the nternal energy by a shape pror If the object s a smooth varaton of a known shape, we can use a term that wll penalze devaton from that shape: Enternal n 0 ( ˆ ) ˆ } { where are the ponts of the known shape.

29 9 external nternal total E E E 0 n nternal d E 0 ), ( ), ( y n x external y x G y x G E Total energy

30 30 Total energy (cont.) e.g., weght controls the penalty for nternal elastcty large medum small Fg from Y. Boykov

31 3 Recap A smple elastc snake s defned by: A set of n ponts, An nternal energy term (tenson, bendng, plus optonal shape pror) An external energy term (gradentbased) To segment an object: Intalze n the vcnty of the object Modfy the ponts to mnmze the total energy

32 3 Aspects to consder Representaton of the contours Defnng the energy functons External Internal Mnmzng the energy functon Extensons: Trackng Interactve segmentaton

33 33 Energy mnmzaton Several algorthms have been proposed to ft deformable contours. We ll look at: Greedy search Dynamc programmng (for D snakes) Calculus of varatons

34 34 Energy mnmzaton (greedy) For each pont, defne a search wndow around t and move to where energy functon s mnmal Typcal wndow sze, e.g., 5 x 5 pxels Stop when a predefned number of ponts have not changed n last teraton, or after a maxmum number of teratons Note: Convergence not guaranteed Need decent ntalzaton

35 35 Energy mnmzaton (dynamc programmng) v 3 v v v 4 v 5 v 6 The Vterb algorthm. Iterate untl optmal poston for each pont s the center of the box,.e., the snake s optmal n the local search space constraned by boxes. [Amn, Weymouth, Jan, 990]

36 36 Dynamc programmng Possble because snake energy can be rewrtten as a sum of par-wse nteracton potentals: E total n,, n) E (, ) ( Or sum of trple-nteracton potentals. E total n,, n) E (,, ) (

37 37 ), ( ), ( ),,,,, ( y n x n n total y x G y x G y y x x E ) ( ) ( n y y x x ) ( ),, ( n n total G E n ), (... ), ( ), ( ),, ( 3 n n n n total v v E v v E v v E E ) ( ), ( G E where Re-wrtng the above wth : y x v, Snake energy: par-wse nteractons

38 38 Vterb algorthm Man dea: determne the optmal poston (state) of predecessor, for each possble poston of self. Then backtrack from best state for last vertex. E total E ( v, v) E( v, v3)... En ( vn, v n ) E( v, v) E( v, v3) E3( v3, v4) v v v3 4 E4( v4, v5 ) v v5

39 vertces 39 Vterb algorthm (cont.) E total E ( v, v) E( v, v3)... En ( vn, v n ) states m E () E () E (3) E( v, v) E( v, v3) E3( v3, v4) v v v3 v E ( m) 0 Complexty: E () E () E (3) E ( m) O( nm ) E 3 () E 3 () E 3 (3) E ( ) E 4 ( m) 3 m E 4 () E 4 () E 4 (3) E ( v 4, v 4 n vs. brute force search? ) v n E n () E n () E n (3) E n (m)

40 40 Vterb algorthm (cont.) DP can be appled to optmze an open ended snake E ( v, v) E( v, v3)... En ( vn, v n ) n For a closed snake, a loop s ntroduced nto the total energy. E ( v, v) E( v, v3)... En ( vn, vn) En( vn, ) v n n 4 3 Work around: ) Fx v and solve for rest. ) Fx an ntermedate node at ts poston found n (), solve for rest.

41 4 Calculus of varatons Gven a D functon ux ( ) :[0,], the basc problem s to mnmze a gven energy (functonal) w.r.t u(x): E( u) F( x, u, u) dx 0 wth boundary condtons: (0), () u a u b and F : gven by the problem.

42 4 Calculus of varatons (cont.) Frst varaton: E E 0 0 E( u v) E( u) 0, u v :[0,], v(0) a, v() b v beng a small ncrement satsfyng the boundary condtons. E( u v) E( u) F( x, u v, u v) dx F( x, u, u) dx 0 0 [ F( x, u v, u v ) F( x, u, u )] dx 0

43 43 Calculus of varatons (cont.) E 0 [ F( x, u v, u v) F( x, u, u)] dx 0 0 Usng Taylor seres approxmaton: F F F( x, u v, u v) F( x, u, u) v v... u u the energy functon becomes: F F E 0 v v dx 0 0 u u

44 44 Calculus of varatons (cont.) F F E 0 v dx v dx 0 0 u 0 u Integratng the second ntegral by parts: F F d F F d F v dx v v dx 0 v dx 0 u u dx u u dx u Snce ths holds for all v, the Euler equaton s: F d F u dx u 0

45 45 Calculus of varatons (cont.) In a smlar form, for the energy: we obtan the Euler equaton: E( u) F( x, u, u, u ) dx 0 u dx u dx u F d F d F 0

46 46 Calculus of varatons (cont.) Analogous s the D problem: u u u u E( u) F( x, y,,,, ) dxdy 0 0 x y x y whose Euler equaton s F d F d F d F d F 0 u dx ux dy u y dx uxx dy u yy

47 47 Back to the snake energy dv d v E( v) Enternal ( ) Emage ( ) a b cg( s) ds ds ds 0 The mage force s of the form: gs () and the Euler equaton s: 4 d v d v a b c g ds ds 4 0 Is ( )

48 48 Iteratve scheme There s no closed form soluton. We consder that we have an ntal soluton. An teratve Newton-type scheme s appled untl stablty s acheved: 4 dv d v d v a b cg 4 dt ds ds

49 49 Iteratve scheme (cont.) Temporal dscretzaton: t t t t t t dv v v x x y y, dt dv v v ds x y Spatal dscretzaton:, x x x v 4 x, v, v x x x x x 4 4x 6x 4x 6x y y y v 4 y, v, v y y y y y 4 4y 6y 4y 6y T T

50 50 Iteratve scheme (cont.) Pluggng the prevous results nto the man equaton, the problem s solved ndependently n each spatal drecton: t t T I A c x x x g t t T I A c y y y g T T T T t t t t t t t t t t t t t t x x,... x, y y,... y, g g ( x, y )..., g g ( x, y )... x y 0 N 0 N x 0 0 y 0 0 A crculant 6b a (4b a) b b (4b a) N N

51 5 Aspects to consder Representaton of the contours Defnng the energy functons External Internal Mnmzng the energy functon Extensons: Trackng Interactve segmentaton

52 5 Trackng by deformable contours. Use fnal contour/model extracted at frame t as an ntal soluton for frame t+. Evolve ntal contour to ft exact object boundary at frame t+ 3. Repeat, ntalzng wth most recent frame. Trackng Heart Ventrcles (multple frames)

53 53 Trackng by deformable contours (cont.) Vsual Dynamcs Group, Dept. Engneerng Scence, Unversty of Oxford. Applcatons: Traffc montorng Human-computer nteracton Anmaton Survellance Computer asssted dagnoss n medcal magng

54 54 3D actve contours

55 55 Lmtatons May over-smooth the boundary It cannot follow topologcal changes of objects

56 56 Lmtatons (cont.) External energy: snake does not really see object boundares n the mage unless t gets very close to t. mage gradents I are large only drectly on the boundary Soluton: Balloon force or gradent vector flow snake

57 57 Gradent vector flow Dffuson of the edge force. Compute a new force feld u(x,y), v(x,y) mnmzng the energy: E( u, v) ( ux uy vx vy ) ( gx g y ) ( u gx) ( u gx) dxdy Euler equatons: u ( g g )( u g ) 0 x y x v ( g g )( v g ) 0 x y y

58 58 Tradtonal external force feld v/s GVF feld Tradtonal force GVF force (Dagrams courtesy Snakes, shapes, gradent vector flow, Xu, Prnce)

59 59 Dffusng Image Gradents mage gradents dffused va Gradent Vector Flow (GVF) Chenyang Xu and Jerry Prnce, 98 f/ 5-59

60 60 More examples

61 6 3D surfaces

62 6 Pros and cons Pros: Useful to track and ft non-rgd shapes Contour remans connected Possble to fll n subjectve contours Flexblty n how energy functon s defned, weghted. Cons: Must have decent ntalzaton near true boundary, may get stuck n local mnmum Parameters of energy functon must be set well based on pror nformaton