Object Detection and Segmentation from Joint Embedding of Parts and Pixels

Transcription

1 Object Detection and Segmentation from Joint Embedding of Parts and Pixels Michael Maire 1, Stella X. Yu 2, Pietro Perona 1 1 California Institute of Technology - Pasadena, CA Boston College - Chestnut Hill, MA 02467

2 Segmentation Detection

3 Segmentation Detection }{{} Perceptual Grouping Framework

4 Ingredients Plug in state-of-the-art components:

5 Ingredients Plug in state-of-the-art components: low-level cues: color, texture, edges [Arbeláez, Maire, Fowlkes, Malik, PAMI 2011]

6 Ingredients Plug in state-of-the-art components: low-level cues: color, texture, edges top-down parts: poselets for person detection [Arbeláez, Maire, Fowlkes, Malik, PAMI 2011] [Bourdev, Maji, Brox, Malik, ECCV 2010]

7 Ingredients Plug in state-of-the-art components: PASCAL VOC 2010 Person Category: Improved Detection and Segmentation low-level cues: color, texture, edges top-down parts: poselets for person detection [Arbeláez, Maire, Fowlkes, Malik, PAMI 2011] [Bourdev, Maji, Brox, Malik, ECCV 2010]

8 Grouping Relationships

9 Grouping Relationships

10 Pixel Affinity: Color, Texture Similarity

11 Pixel Affinity: Color, Texture Similarity

12 Pixel Affinity: Color, Texture Similarity

13 Part Affinity: Geometric Compatibility

14 Part Affinity: Geometric Compatibility

15 pixels

16 parts pixels

17 parts surround pixels

18 parts surround pixels

19 parts surround pixels

20 parts surround figure/ground prior pixels

21 parts surround figure/ground prior pixels Angular Embedding objects figure/ground segmentation

22 Angular Embedding

23 Angular Embedding q p

24 Angular Embedding q p

25 Angular Embedding q Given: Relative ordering Θ(, ) Confidence on relationships C(, ) p

26 Angular Embedding p q Given: Relative ordering Θ(, ) Confidence on relationships C(, ) Compute: Global ordering θ( ) Embed into unit circle: p z(p) = e iθ(p) -p -q θ

27 Angular Embedding p q Given: Relative ordering Θ(, ) Confidence on relationships C(, ) Compute: Global ordering θ( ) Embed into unit circle: p z(p) = e iθ(p) -p -q θ Subject to: Linear constraints on embedding solution in columns of U

28 i z(p) z(r) z(q) 1 minimize: ε = p 0 1 q C(p,q) p,q C(p,q) z(p) z(p) 2 [Yu, PAMI 2011]

29 z(r)e iθ(p,r) i z(p) z(q)e iθ(p,q) z(r) C(p, r) Θ(p, r) C(p, q) Θ(p, q) z(q) 1 minimize: ε = p 0 1 q C(p,q) p,q C(p,q) z(p) z(p) 2 [Yu, PAMI 2011]

30 z(r)e iθ(p,r) i z(p) z(r) C(p, r) Θ(p, r) z(p) C(p, q) Θ(p, q) z(q)e iθ(p,q) z(q) 1 minimize: ε = p 0 1 q C(p,q) p,q C(p,q) z(p) z(p) 2 [Yu, PAMI 2011]

31 C q (C f, Θ f ) (C s, Θ s ) U C p

32 C = Θ = Σ 1 pixels {}}{ parts prior {}}{ {}}{ surround {}}{ C p α C q β C s γ C f 0 β Cs T γ Cf T Θ s Θ f 0 Θ T s Θ T f 0 0

33 Angular Embedding Relax to generalized eigenproblem QPQz = λz: P = D 1 W Q = I D 1 U(U T D 1 U) 1 U T with D and W defined as: D = Diag(C1 n ) W = C e iθ Eigenvectors {z 0, z 1,..., z m 1 } embed pixels and parts into C m

34 Angular Embedding z 0 encodes global ordering z 1, z 2,..., z m 1 encode grouping

35 Angular Embedding z 0 encodes global ordering z 1, z 2,..., z m 1 encode grouping if Θ = 0 Normalized Cuts (grouping without ordering)

36 Decoding Eigenvectors: Object Detection I(z0) I(z1) I(z2) R(z 2 ) R(z 0 ) R(z 1 ) Ordering Grouping

37 Decoding Eigenvectors: Object Detection I(z0) I(z1) I(z2) R(z 2 ) R(z 0 ) R(z 1 ) Ordering Grouping

38 Decoding Eigenvectors: Object Detection I(z0) I(z1) I(z2) R(z 2 ) R(z 0 ) R(z 1 ) Ordering Grouping

39 Decoding Eigenvectors: Object Detection I(z0) I(z1) I(z2) R(z 2 ) R(z 0 ) R(z 1 ) Ordering Grouping

40 Decoding Eigenvectors: Object Detection I(z0) I(z1) I(z2) R(z 2 ) R(z 0 ) R(z 1 ) Ordering Grouping

41 Decoding Eigenvectors: Object Detection I(z0) I(z1) I(z2) R(z 2 ) R(z 0 ) R(z 1 ) Ordering Grouping

42 Decoding Eigenvectors: Object Detection I(z0) I(z1) I(z2) R(z 2 ) R(z 0 ) R(z 1 ) Ordering Grouping

43 Decoding Eigenvectors: Object Detection I(z0) I(z1) I(z2) R(z 2 ) R(z 0 ) R(z 1 ) Ordering Grouping

44 Decoding Eigenvectors: Figure/Ground I(z) R(z) z 0 z 1 z 2 z 3 z 4

45 Decoding Eigenvectors: Figure/Ground I(z) R(z) z 0 z 1 z 2 z 3 z 4 IR(z) z 0 z 1 z 2 z 3 z 4

46 Decoding Eigenvectors: Segmentation IR(z) z 0 z 1 z 2 z 3 z 4 Figure/Ground }{{} Hierarchical Segmentation [Arbeláez, Maire, Fowlkes, Malik, PAMI 2011]

47 Decoding Eigenvectors: Object Segmentation Assign pixels pk to objects Qi via parts qj : pk argmin min {Dist(pk, qj )} Qi qj Qi

48 Decoding Eigenvectors: Object Segmentation Assign pixels pk to objects Qi via parts qj : pk argmin min {Dist(pk, qj )} Qi qj Qi

49 Decoding Eigenvectors

50 Results: PASCAL 2010 Person Category Detections Poselet Mask F/G Mask Segmentation

51 Results: PASCAL 2010 Person Category Detections Poselet Mask F/G Mask Segmentation

52 Results: PASCAL 2010 Person Category I Segmentation task score: 41.1 (35.5 for poselet baseline)

53 Results: PASCAL 2010 Person Category I Segmentation task score: 41.1 (35.5 for poselet baseline) I 11% relative improvement due to better detection

54 Summary Simultaneous segmentation and detection: Part detectors figure pop-out, object grouping Color, texture pixel grouping

55 Summary Simultaneous segmentation and detection: Part detectors figure pop-out, object grouping Color, texture pixel grouping Graph: Parts and pixels as nodes Links encode multiple relationship types

56 Summary Simultaneous segmentation and detection: Part detectors figure pop-out, object grouping Color, texture pixel grouping Graph: Parts and pixels as nodes Links encode multiple relationship types Embedding: graph nodes C m

57 Summary Simultaneous segmentation and detection: Part detectors figure pop-out, object grouping Color, texture pixel grouping Graph: Parts and pixels as nodes Links encode multiple relationship types Embedding: graph nodes C m Decode: Figure/ground Image segmentation Detected objects Segmentation of each object instance

58 Summary Simultaneous segmentation and detection: Part detectors figure pop-out, object grouping Color, texture pixel grouping Graph: Parts and pixels as nodes Links encode multiple relationship types Embedding: graph nodes C m Decode: Figure/ground Image segmentation Detected objects Segmentation of each object instance Better person detection and segmentation on PASCAL

59 Thank You