Submodularity beyond submodular energies: Coupling edges in graph cuts

Transcription

1 Submodularity beyond submodular energies: Coupling edges in graph cuts Stefanie Jegelka and Jeff Bilmes Max Planck Institute for Intelligent Systems Tübingen, Germany University of Washington Seattle, USA 1 / 18

2 local pairwise random fields... 2 / 18

3 Graph Cuts Cooperative Cuts Optimization Applications 3 / 18

4 Graph Cuts Cooperative Cuts Optimization Random Walker Curvature reg. Applications Graph Cut 3 / 18

5 Graph Cuts Cooperative Cuts Optimization Random Walker Curvature reg. Applications Graph Cut 3 / 18

6 Markov Random Fields and Energies p(x z) exp( E Ψ (x; z)) MAP x = arg min x E Ψ (x; z) s t 4 / 18

7 Markov Random Fields and Energies p(x z) exp( E Ψ (x; z)) MAP x = arg min x E Ψ (x; z) s E(x; z) = i Ψ i (x i ) + (i,j) N Ψ ij (x i, x j ) t 4 / 18

8 Markov Random Fields and Energies p(x z) exp( E Ψ (x; z)) MAP x = arg min x E Ψ (x; z) s E(x; z) = Ψ i (x i ) i E(x; z) = + Ψ ij (x i, x j ) (i,j) N e Γx E t w e + e Γx E n w e t 4 / 18

9 Markov Random Fields and Energies p(x z) exp( E Ψ (x; z)) MAP x = arg min x E Ψ (x; z) E(x; z) = Ψ i (x i ) + Ψ ij (x i, x j ) i (i,j) N E(x; z) = w e + w e e Γx E t e Γx E n 1 s t 4 / 18

10 s t 5 / 18

11 s t 5 / 18

12 s t 5 / 18

13 s t Couple edges globally 5 / 18

14 Richer Cuts: Cooperative Cuts E(x) = e Γx w(e) s = w(γx) t 6 / 18

15 Richer Cuts: Cooperative Cuts E(x) = e Γx w(e) s = w(γx) t E f (x) = f (Γx) submodular function on edges 6 / 18

16 Richer Cuts: Cooperative Cuts E(x) = e Γx w(e) s = w(γx) t E f (x) = f (Γx) submodular function on edges non-submodular & global energy 6 / 18

17 Coupling via Submodularity s t 7 / 18

18 Coupling via Submodularity e e A B A B f (A e) f (A) f (A B e) f (A B) t Graph Cuts: LHS = RHS it does not matter which other edges are cut s 8 / 18

19 Coupling via Submodularity e e A B A B f (A e) f (A) f (A B e) f (A B) s t Graph Cuts: LHS = RHS it does not matter which other edges are cut submodularity: reward co-occurrence structure 8 / 18

20 Generality Special cases of cooperative cuts: s labels t features... boundary (robust) P n potentials (Kohli et al. 07, 09) label costs (Delong et al. 11) discrete versions of norm-based cuts (Sinop & Grady 07)... 9 / 18

21 Optimization? 10 / 18

22 Optimization? (s, t)-cut Γ E with min cost f (Γ). Theorem Minimum Cooperative Cut is NP-hard. 10 / 18

23 Optimization Γ 0 = ; repeat compute upper bound ˆf i f based on Γ i 1 ; until convergence ; ˆf i (Γ i 1 ) = f (Γ i 1 ) 11 / 18

24 Optimization Γ 0 = ; repeat compute upper bound ˆf i f based on Γ i 1 ; Γ i argmin{ ˆf i (Γ) Γ a cut } ; i = i + 1; until convergence ; // Min-cut! ˆf i (Γ i 1 ) = f (Γ i 1 ) 11 / 18

25 Optimization Γ 0 = ; repeat compute upper bound ˆf i f based on Γ i 1 ; Γ i argmin{ ˆf i (Γ) Γ a cut } ; i = i + 1; until convergence ; // Min-cut! Worst-case approximation bound: E f (x) Γ 1+( Γ 1)ν E f (x ) for ν = min e Γ ρe(e\e) max e C f (e) 11 / 18

26 Image Segmentation Random Walker Curvature reg. Graph Cut 12 / 18

27 Image Segmentation Random Walker Curvature reg. Graph Cut prefer congruous boundaries 12 / 18

28 Selective Discount for Congruous Boundaries s t E w (x) = w e +λ e Γ E t E f (x) = w e +λ f (Γ E n ) e Γ E t e Γ E n w e 13 / 18

29 Selective Discount for Congruous Boundaries s t E w (x) = w e +λ e Γ E t E f (x) = w e +λ f (Γ E n ) e Γ E t e Γ E n w e discount for co-occurring similar edges no discount for dissimilar edges 13 / 18

30 Structured Discounts groups S i of edges f (Γ) = i f i(γ S i ) / 18

31 Structured Discounts groups S i of edges f (Γ) = i f i(γ S i ) / 18

32 Structured Discounts groups S i of edges f (Γ) = i f i(γ S i ) / 18

33 Graph Cuts Cooperative Cuts Optimization Applications Some Results: Shading Graph Cut CoopCut 7.39% 2.23% 7.65% 3.50% 15 / 18

34 Some Results: Shading gray color high-freq Graph Cut: no discount CoopCut (1 group): discount CoopCut (15 groups): structured discount Graph Cut CoopCut 5.08% 0.64% 16 / 18

35 Shrinking bias GC twig CoopC twig GC total CoopC total ψ i (x i ) + λ w e e Γx i total error (%) twig error (%) λ Graph Cut 17 / 18

36 Shrinking bias GC twig CoopC twig GC total CoopC total ψ i (x i ) + λ w e e Γx i total error (%) twig error (%) λ Graph Cut 17 / 18

37 Shrinking bias GC twig CoopC twig GC total CoopC total ψ i (x i ) + λ f (Γx) i total error (%) twig error (%) CoopCut λ Graph Cut 17 / 18

38 Shrinking bias GC twig CoopC twig GC total CoopC total ψ i (x i ) + λ f (Γx) i total error (%) twig error (%) CoopCut λ Graph Cut 17 / 18

39 Summary: Coupling Edges in Graph Cuts global, non-submodular family of energies NP-hard, but... graph structure indirect submodularity efficient approximation algorithm applications guide segmentations via edge coupling 18 / 18