Pushmeet Kohli Microsoft Research

Transcription

1 Pushmeet Kohli Microsoft Research

2 E(x) x in {0,1} n Image (D) [Boykov and Jolly 01] [Blake et al. 04]

3 E(x) = c i x i Pixel Colour x in {0,1} n Unary Cost (c i ) Dark (Bg) Bright (Fg) x* = arg min E(x) [Boykov and Jolly 01] [Blake et al. 04]

4 E(x) = c i x i + d ij x i -x j x in {0,1} n Pixel Colour Smoothness Prior Unary Cost (c i ) Dark (Bg) Bright (Fg) [Boykov and Jolly 01] [Blake et al. 04]

5 E(x) = c i x i + d ij x i -x j x in {0,1} n Pixel Colour Smoothness Prior How to minimize E(x)? x* = arg min E(x) Old Solution [Boykov and Jolly 01] [Blake et al. 04]

6 NP-Hard MAXCUT CSP Space of Problems

7 Perfect Graphs, Low-tree width, Submodular functions, Structured decomposable functions NP-Hard MAXCUT CSP Tractable Problems March 2014 Space of Problems

8 Tractability Properties NP-Hard MAXCUT CSP Structural Tractability Tree Structured Space of Problems

9 Tractability Properties Constraints on the terms of your energy functions d ij x i -x j Language or Form Tractability NP-Hard CSP Tree Structured MAXCUT Pair-wise O(n3 ) Submodular Functions O(n 6 ) Space of Problems

10 S st-mincut E(x) Pairwise Submodular T Solution [Hammer, 1965] [Kolmogorov and Zabih, 2002]

11 Demo

12 .. So what are the challenges?

13 Modelling Challenges

14 Does not enforce connectivity Short boundary bias Cannot enforce priors on label counts

15 Taskar et al. 02 associative potentials Roth & Black 05 field of experts Kohli Kumar Torr. 07 segment consistency Kohli Ladicky Torr 08 segment consistency Woodford et al. 08 planarity constraint Vicente et al. 08 connectivity constraint Nowozin & Lampert 09 connectivity constraint Ladický et al. 09 consistency over several scales Woodford et al. 09 marginal probability Delong et al. 10 label occurrence costs Ladicky et al. 10 label set co-occurrence costs Jegelka and Bilmes 11 label set co-occurrence costs many others

16 Transformation schemes [Kolmogorov 02] [Kohli et al.07, 08,09] Constrained Inference using Parametric Mincuts [Kolmogorov 07] [Lim et al.08, 14] Decomposition Techniques [Woodford et al. 09] [Komodakis 09].. Iterative refinement of constraints (Connectivity and Bounding Box Potentials) [Nowozin and Lampert 08, Lempitsky et al. 09].. Special purpose message computation [Gupta and Sarawagi 07, 08] [Tarlow et al. 09].. Learning to preserve higher-order statistics [Pletscher and Kohli 12 ]

17 Transformation schemes [Kolmogorov 02] [Kohli et al.07, 08,09] Constrained Inference using Parametric Mincuts [Kolmogorov 07] [Lim et al.08, 14] Decomposition Techniques [Woodford et al. 09] [Komodakis 09].. Iterative refinement of constraints (Connectivity and Bounding Box Potentials) [Nowozin and Lampert 08, Lempitsky et al. 09].. Special purpose message computation [Gupta and Sarawagi 07, 08] [Tarlow et al. 09].. Learning to preserve higher-order statistics [Pletscher and Kohli 12 ]

18 Higher Order Function Pairwise Submodular Function Ensure tractability of the transformed problem

19

20 Unary Potentials [Shotton et al. ECCV 2006] Colour, Location & Texture Higher Order Energy Sky Building Grass Tree + Pairwise Smoothness Potentials + Pixels belonging to a group should take the same label p Higher Order Potentials (Defined using multiple Segmentations) h(x p ) = { 0 if x i =L, I ϵ p C otherwise

21 Unary Potentials [Shotton et al. ECCV 2006] Colour, Location & Texture Higher Order Energy Sky Tree Building Grass + Pairwise Smoothness Potentials + Higher Order Potentials (Defined using multiple Segmentations)

22 Unary Potentials [Shotton et al. ECCV 2006] Colour, Location & Texture Higher Order Energy Sky Tree Building Grass + Pairwise Smoothness Potentials + Energy Minimization Segmentation Solution Higher Order Potentials (Defined using multiple Segmentations)

23 h(x p ) = { 0 if all x i = Tree (0), I ϵ p 1 otherwise Pixels belonging to the group p should take the same label tree p

24 x i { 0 if all x i = 0 f(x)= 1 otherwise x i

25 x i { 0 if all x i = 0 f(x)= 1 otherwise x i

26 min f(x) x = Higher Order Submodular Function min x,a ϵ {0,1} a + ā x i Quadratic Submodular Function x i f(x)= { 0 if all x i = 0 1 otherwise 1 a=0 a=1 1 x i 2 3

27 Image (MSRC-21) Pairwise CRF Higher order CRF Ground Truth Grass Sheep [Runner-Up, PASCAL VOC 2008 ]

28

29 E(X) = c + i x i d ij x i -x j Encourages short boundaries Penalize types of boundaries not the actual number of boundaries! Image Segmentation Cooperative Cuts [Jegelka and Bilmes, CVPR 2011]

30 E(X) = c + i x i d ij x i -x j

31 E(X) = c + i x i d ij x i -x j + h g (X p ) g in G Divide edges into different types Incorporate a higher order consistency potential over the edges h g (X p ) = F ( x i -x j ) ij in g F x i -x j ij in g

32 E(X) = c + i x i d ij x i -x j + h g (X p ) g in G Divide edges into different types Incorporate a higher order consistency potential over the edges h g (X p ) = F ( x i -x j ) ij in g F x i -x j ij in g

33 E(X) = c + i x i d ij x i -x j + h g (X p ) g in G Divide edges into different types Incorporate a higher order consistency potential over the edges h g (X p ) = F ( x i -x j ) ij in g F x i -x j ij in g Needs Special Purpose Minimization x

34 f1 f2 f1 f2 Transformation f1 f2

35

36

37 Image Ground Truth Pairwise Model Higher-order Model

38 Image Ground Truth Pairwise Model Deep Model

39

40 min f(x) x = Higher Order Submodular Function min f 1 (x)a + f 2 (x) (1-a) x,a ϵ {0,1} Quadratic Submodular Function f 2 (x) a=0 f 1 (x) a=1 Lower envelop of concave functions is concave x i [Kohli et al. 08]

41 min f(x) x = x ϵ {0,1} Higher Order Submodular Function min max f 1 (x)a + f 2 (x) (1-a) a ϵ {0,1} Quadratic Submodular Function f 2 (x) a=0 f 1 (x) a=1 Upper envelope of linear functions is convex Very Hard x i Problem!!!! [Kohli and Kumar, CVPR 2010]

42 SILHOUETTE CONSTRAINTS SIZE/VOLUME PRIORS [Sinha et al. 05, Cremers et al. 08] [Woodford et al. 2009] 3D RECONSTRUCTION BINARY SEGMENTATION Rays must touch silhouette at least once Object Background Prior on size of object (say n/2) [Kohli and Kumar, CVPR 2010]

43 x* = Such that: arg min x E(x) x has area A = A x has boundary length B = B

44 Parametric Maxflow/st-Mincut λ = + λ = x v =0 x v =n x* = arg min

45 Parametric Maxflow/st-Mincut λ = + λ = x v =0 x v =n x* = arg min [Gallo et al. 1986] [Kolmogorov et al. ICCV 2007]

46 Generalized (Multi-dimensional) Parametric Maxflow

47

48 x y Can we find parameters that lead to solutions consistent with higher-order statistics? [Pletscher and Kohli, AISTATS 2012 ]

49 x y [AISTATS 2012]

50 x y Requires solution of [AISTATS 2012]

51 [AISTATS 2012]

52 Adaptive data-driven representations for Higher order Potentials Global potentials that encode topology constraints Efficiency