MAP Estimation Algorithms in Computer Vision - Part II

Transcription

1 MAP Estimation Algorithms in Comuter Vision - Part II M. Pawan Kumar, University of Oford Pushmeet Kohli, Microsoft Research

2 Eamle: Image Segmentation E() = c i i + c ij i (1- j ) i i,j E: {0,1} n R 0 fg 1 bg n = number of iels Image (D)

3 Eamle: Image Segmentation E() = c i i + c ij i (1- j ) i i,j E: {0,1} n R 0 fg 1 bg n = number of iels Unary Cost (c i ) Dark (negative) Bright (ositive)

4 Eamle: Image Segmentation E() = c i i + c ij i (1- j ) i i,j E: {0,1} n R 0 fg 1 bg n = number of iels Discontinuity Cost (c ij )

5 Eamle: Image Segmentation E() = c i i + c ij i (1- j ) i i,j E: {0,1} n R 0 fg 1 bg n = number of iels * = arg min E() Global Minimum (*) How to minimize E()?

6 Outline of the Tutorial The st-mincut roblem Connection between st-mincut and energy minimization? What roblems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Oen Problems

7 Outline of the Tutorial The st-mincut roblem Connection between st-mincut and energy minimization? What roblems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Oen Problems

8 The st-mincut Problem Source v 1 v Grah (V, E, C) Vertices V = {v 1, v 2... v n } Edges E = {(v 1, v 2 )...} Costs C = {c (1, 2)...} Sink

9 The st-mincut Problem What is a st-cut? Source v 1 v Sink

10 The st-mincut Problem What is a st-cut? Source An st-cut (S,T) divides the nodes between source and sink v 1 v What is the cost of a st-cut? Sum of cost of all edges going from S to T Sink = 16

11 The st-mincut Problem What is a st-cut? Source v 1 v Sink = 7 An st-cut (S,T) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T What is the st-mincut? st-cut with the minimum cost

12 How to comute the st-mincut? Source v 1 v Sink 4 Solve the dual maimum flow roblem Comute the maimum flow between Source and Sink Constraints Edges: Flow < Caacity Nodes: Flow in = Flow out Min-cut\Ma-flow Theorem In every network, the maimum flow equals the cost of the st-mincut

13 Maflow Algorithms Source v 1 v 2 5 Flow = Sink 4 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

14 Maflow Algorithms Source v 1 v 2 5 Flow = Sink 4 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

15 Maflow Algorithms Source v 1 v Flow = Sink 4 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

16 Maflow Algorithms Source v 1 v 2 3 Flow = 2 2 Sink 4 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

17 Maflow Algorithms Source v 1 v 2 3 Flow = 2 2 Sink 4 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

18 Maflow Algorithms Source v 1 v 2 3 Flow = 2 2 Sink 4 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

19 Maflow Algorithms Source v 1 v 2 3 Flow = Sink 0 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

20 Maflow Algorithms Source v 1 v 2 3 Flow = 6 2 Sink 0 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

21 Maflow Algorithms Source v 1 v 2 3 Flow = 6 2 Sink 0 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

22 Maflow Algorithms Source v 1 v 2 2 Flow = Sink 0 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

23 Maflow Algorithms Source v 1 v 2 2 Flow = 7 3 Sink 0 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

24 Maflow Algorithms Source v 1 v 2 2 Flow = 7 3 Sink 0 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

25 History of Maflow Algorithms Augmenting Path and Push-Relabel n: #nodes m: #edges U: maimum edge weight Algorithms assume nonnegative edge weights [Slide credit: Andrew Goldberg]

26 History of Maflow Algorithms Augmenting Path and Push-Relabel n: #nodes m: #edges U: maimum edge weight Algorithms assume nonnegative edge weights [Slide credit: Andrew Goldberg]

27 Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting ath Source a 1 a Sink

28 Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting ath Source a 1 a Sink Bad Augmenting Paths

29 Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting ath Source a 1 a Sink Bad Augmenting Path

30 Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting ath Source a 1 a Sink

31 Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting ath n: #nodes m: #edges Source a 1 a Sink We will have to erform 2000 augmentations! Worst case comleity: O (m Total_Flow) (Pseudo-olynomial bound: deends on flow)

32 Augmenting Path based Algorithms Dinic: Choose shortest augmenting ath n: #nodes m: #edges Source a 1 a Sink Worst case Comleity: O (m n 2 )

33 Maflow in Comuter Vision Secialized algorithms for vision roblems Grid grahs Low connectivity (m ~ O(n)) Dual search tree augmenting ath algorithm [Boykov and Kolmogorov PAMI 2004] Finds aroimate shortest augmenting aths efficiently High worst-case time comleity Emirically outerforms other algorithms on vision roblems

34 Maflow in Comuter Vision Secialized algorithms for vision roblems Grid grahs Low connectivity (m ~ O(n)) Dual search tree augmenting ath algorithm [Boykov and Kolmogorov PAMI 2004] Finds aroimate shortest augmenting aths efficiently High worst-case time comleity Emirically outerforms other algorithms on vision roblems Efficient code available on the web htt://

35 Outline of the Tutorial The st-mincut roblem Connection between st-mincut and energy minimization? What roblems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Oen Problems

36 St-mincut and Energy Minimization S st-mincut T Minimizing a Qudratic Pseudoboolean function E() Functions of boolean variables Pseudoboolean? E: {0,1} n R E() = c i i + c ij i (1- j ) i i,j c ij 0 Polynomial time st-mincut algorithms require non-negative edge weights

37 So how does this work? Construct a grah such that: 1. Any st-cut corresonds to an assignment of 2. The cost of the cut is equal to the energy of : E() S st-mincut E() T Solution

38 Grah Construction E(a 1,a 2 ) Source (0) a 1 a 2 Sink (1)

39 Grah Construction E(a 1,a 2 ) = 2a 1 2 Source (0) a 1 a 2 Sink (1)

40 Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 2 Source (0) a 1 a 2 5 Sink (1)

41 Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 Source (0) 2 9 a 1 a Sink (1)

42 Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 Source (0) 2 9 a 1 a Sink (1)

43 Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a Source (0) a 1 a Sink (1)

44 Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a Source (0) a 1 a Sink (1)

45 Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Source (0) 2 9 Cost of cut = 11 1 a 1 a 2 a 1 = 1 a 2 = E (1,1) = 11 Sink (1)

46 Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Source (0) a 1 a 2 st-mincut cost = 8 a 1 = 1 a 2 = E (1,0) = 8 Sink (1)

47 Energy Function Rearameterization Two functions E 1 and E 2 are rearameterizations if E 1 () = E 2 () for all For instance: E 1 (a 1 ) = 1+ 2a 1 + 3ā 1 E 2 (a 1 ) = 3 + ā 1 a 1 ā a 1 + 3ā ā

48 Flow and Rearametrization E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a Source (0) a 1 a Sink (1)

49 Flow and Rearametrization E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Source (0) a 1 a a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1 Sink (1)

50 Flow and Rearametrization E(a 1,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Source (0) a 1 a a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1 Sink (1)

51 Flow and Rearametrization E(a 1,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Source (1) a 1 a a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2 Sink (0)

52 Flow and Rearametrization E(a 1,a 2 ) = 2 + 3ā 1 + 5a a 1 ā 2 + ā 1 a 2 Source (1) a 1 a a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2 Sink (0)

53 Flow and Rearametrization E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 Source (1) a 1 a Sink (0)

54 Flow and Rearametrization E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 Source (1) a 1 a Sink (0)

55 Flow and Rearametrization E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 3ā 1 + 5a 2 + 2a 1 ā 2 Source (1) a 1 a Sink (0) = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2 a 1 a 2 F1 F

56 Flow and Rearametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 3ā 1 + 5a 2 + 2a 1 ā 2 Source (1) a 1 a Sink (0) = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2 a 1 a 2 F1 F

57 Flow and Rearametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Source (1) a 1 a No more augmenting aths ossible Sink (0)

58 Flow and Rearametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Residual Grah (ositive coefficients) Total Flow bound on the otimal solution Source (1) a 1 a Sink (0) Inference of the otimal solution becomes trivial because the bound is tight

59 Flow and Rearametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Residual Grah (ositive coefficients) Total Flow bound on the otimal solution Source (1) a 1 a 2 st-mincut cost = 8 a 1 = 1 a 2 = E (1,0) = 8 Sink (0) Inference of the otimal solution becomes trivial because the bound is tight

60 Eamle: Image Segmentation E() = c i i + c ij i (1- j ) i i,j E: {0,1} n R 0 fg 1 bg * = arg min E() Global Minimum (*) How to minimize E()?

61 How does the code look like? Grah *g; For all iels /* Add a node to the grah */ nodeid() = g->add_node(); Source (0) /* Set cost of terminal edges */ set_weights(nodeid(), fgcost(), bgcost()); end for all adjacent iels,q add_weights(nodeid(), nodeid(q), cost); end g->comute_maflow(); label_ = g->is_connected_to_source(nodeid()); Sink (1) // is the label of iel (0 or 1)

62 How does the code look like? Grah *g; For all iels end /* Add a node to the grah */ nodeid() = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeid(), fgcost(), bgcost()); for all adjacent iels,q add_weights(nodeid(), nodeid(q), cost); end g->comute_maflow(); label_ = g->is_connected_to_source(nodeid()); Source (0) bgcost(a 1 ) bgcost(a 2 ) a 1 a 2 fgcost(a 1 ) fgcost(a 2 ) Sink (1) // is the label of iel (0 or 1)

63 How does the code look like? Grah *g; For all iels end /* Add a node to the grah */ nodeid() = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeid(), fgcost(), bgcost()); for all adjacent iels,q add_weights(nodeid(), nodeid(q), cost(,q)); end g->comute_maflow(); label_ = g->is_connected_to_source(nodeid()); Source (0) bgcost(a 1 ) bgcost(a 2 ) cost(,q) a 1 a 2 fgcost(a 1 ) fgcost(a 2 ) Sink (1) // is the label of iel (0 or 1)

64 How does the code look like? Grah *g; For all iels end /* Add a node to the grah */ nodeid() = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeid(), fgcost(), bgcost()); for all adjacent iels,q add_weights(nodeid(), nodeid(q), cost(,q)); end g->comute_maflow(); label_ = g->is_connected_to_source(nodeid()); // is the label of iel (0 or 1) Source (0) bgcost(a 1 ) bgcost(a 2 ) cost(,q) a 1 a 2 fgcost(a 1 ) fgcost(a 2 ) Sink (1) a 1 = bg a 2 = fg

65 Image Segmentation in Video n-links s = 0 st-cut E() * w q t = 1 Image Flow Global Otimum

66 Image Segmentation in Video Image Flow Global Otimum

67 Boykov & Jolly ICCV 01, Kohli & Torr (ICCV05, PAMI07) Dynamic Energy Minimization E A minimize S A Can we do better? Recycling Solutions E B comutationally eensive oeration S B

68 Dynamic Energy Minimization E A minimize S A Reuse flow A and B similar differences between A and B E B* Simler energy cheaer oeration E B comutationally eensive oeration S B Rearametrization time seedu! Boykov & Jolly ICCV 01, Kohli && Torr Torr (ICCV05, PAMI07)

69 Dynamic Energy Minimization Original Energy E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Rearametrized Energy E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 New Energy E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 7a 1 ā 2 + ā 1 a 2 New Rearametrized Energy E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 + 5a 1 ā 2 Kohli & Torr (ICCV05, PAMI07) Boykov & Jolly ICCV 01, Kohli & Torr (ICCV05, PAMI07)

70 Outline of the Tutorial The st-mincut roblem Connection between st-mincut and energy minimization? What roblems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Oen Problems

71 Minimizing Energy Functions General Energy Functions NP-hard to minimize Only aroimate minimization ossible Easy energy functions Solvable in olynomial time Submodular ~ O(n 6 ) MAXCUT NP-Hard Submodular Functions Functions defined on trees Sace of Function Minimization Problems

72 Submodular Set Functions Let E= {a 1,a 2,... a n } be a set 2 E = #subsets of E Set function f 2 E R

73 Submodular Set Functions Let E= {a 1,a 2,... a n } be a set 2 E = #subsets of E Set function f 2 E R is submodular if f(a) + f(b) f(ab) + f(ab) for all A,B E E A B Imortant Proerty Sum of two submodular functions is submodular

74 Minimizing Submodular Functions Minimizing general submodular functions O(n 5 Q + n 6 ) where Q is function evaluation time [Orlin, IPCO 2007] Symmetric submodular functions E () = E (1 - ) O(n 3 ) [Queyranne 1998] Quadratic seudoboolean Can be transformed to st-mincut One node er variable [ O(n 3 ) comleity] Very low emirical running time

75 Submodular Pseudoboolean Functions Function defined over boolean vectors = { 1, 2,... n } Definition: All functions for one boolean variable (f: {0,1} -> R) are submodular A function of two boolean variables (f: {0,1} 2 -> R) is submodular if f(0,1) + f(1,0) f(0,0) + f(1,1) A general seudoboolean function f 2 n R is submodular if all its rojections f are submodular i.e. f (0,1) + f (1,0) f (0,0) + f (1,1)

76 Quadratic Submodular Pseudoboolean Functions E() = θ i ( i ) + θ ij ( i, j ) i i,j For all ij θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1)

77 Quadratic Submodular Pseudoboolean Functions E() = θ i ( i ) + θ ij ( i, j ) i i,j For all ij θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1) Equivalent (transformable) E() = c i i + c ij i (1- j ) i i,j c ij 0 i.e. All submodular QPBFs are st-mincut solvable

78 How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) i 0 1 j 0 1 A B C D = A C-A C-A D-C D-C B+C -A-D 0 0 if 1 =1 add C-A if 2 = 1 add D-C θ ij ( i, j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) i + (θ ij (1,0)-θ ij (0,0)) j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1- i ) j B+C-A-D 0 is true from the submodularity of θ ij

79 How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) i 0 1 j 0 1 A B C D = A C-A C-A D-C D-C B+C -A-D 0 0 if 1 =1 add C-A if 2 = 1 add D-C θ ij ( i, j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) i + (θ ij (1,0)-θ ij (0,0)) j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1- i ) j B+C-A-D 0 is true from the submodularity of θ ij

80 How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) i 0 1 j 0 1 A B C D = A C-A C-A D-C D-C B+C -A-D 0 0 if 1 =1 add C-A if 2 = 1 add D-C θ ij ( i, j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) i + (θ ij (1,0)-θ ij (0,0)) j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1- i ) j B+C-A-D 0 is true from the submodularity of θ ij

81 How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) i 0 1 j 0 1 A B C D = A C-A C-A D-C D-C B+C -A-D 0 0 if 1 =1 add C-A if 2 = 1 add D-C θ ij ( i, j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) i + (θ ij (1,0)-θ ij (0,0)) j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1- i ) j B+C-A-D 0 is true from the submodularity of θ ij

82 How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) i 0 1 j 0 1 A B C D = A C-A C-A D-C D-C B+C -A-D 0 0 if 1 =1 add C-A if 2 = 1 add D-C θ ij ( i, j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) i + (θ ij (1,0)-θ ij (0,0)) j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1- i ) j B+C-A-D 0 is true from the submodularity of θ ij

83 Quadratic Submodular Pseudoboolean Functions E() = θ i ( i ) + θ ij ( i, j ) i i,j in {0,1} n For all ij θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1) Equivalent (transformable) S st-mincut T

84 Minimizing Non-Submodular Functions E() = θ i ( i ) + θ ij ( i, j ) i i,j θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1) for some ij Minimizing general non-submodular functions is NP-hard. Commonly used method is to solve a relaation of the roblem [Slide credit: Carsten Rother]

85 Minimization using Roof-dual Relaation E({ }) ( ) unary ~ q q ( (,, q q ) ) ~ q ( 0,0) q(1,1) q(0,1) q(1,0) airwise submodular q ~ (0,0) q (1,1) ~ ~ (0,1) airwise nonsubmodular q q (1,0) [Slide credit: Carsten Rother]

86 2 ), (1 ~ ),1 ( ~ 2 ),1 (1 ), ( 2 ) (1 ) ( }) },{ '({ q q q q q q q q E Double number of variables: ), ( ~ ), ( ) ( }) ({ q q q q E, Minimization using Roof-dual Relaation ) 1 ( [Slide credit: Carsten Rother]

87 2 ), (1 ~ ),1 ( ~ 2 ),1 (1 ), ( 2 ) (1 ) ( }) },{ '({ q q q q q q q q E Double number of variables: ), ( ~ ), ( ) ( }) ({ q q q q E, Minimization using Roof-dual Relaation ),1 ( ~ ), ( q q q q f (1,0) ~ (0,1) ~ (1,1) ~ (0,0) ~ q q q q (1,1) (0,0) (1,0) 0,1) ( q q q q f f f f Non- submodular Submodular ) 1 (

88 2 ), (1 ~ ),1 ( ~ 2 ),1 (1 ), ( 2 ) (1 ) ( }) },{ '({ q q q q q q q q E Double number of variables: ), ( ~ ), ( ) ( }) ({ q q q q E, Minimization using Roof-dual Relaation ) 1 ( is submodular! Ignore (solvable using st-mincut) Proerty of the roblem: 1

89 1 is the otimal label Proerty of the solution: 2 ), (1 ~ ),1 ( ~ 2 ),1 (1 ), ( 2 ) (1 ) ( }) },{ '({ q q q q q q q q E Double number of variables: ), ( ~ ), ( ) ( }) ({ q q q q E, ) 1 ( Minimization using Roof-dual Relaation

90 Reca Eact minimization of Submodular QBFs using grah cuts. Obtaining artially otimal solutions of nonsubmodular QBFs using grah cuts.

91 But... Need higher order energy functions to model image structure Field of eerts [Roth and Black] Many roblems in comuter vision involve multile labels E() = θ i ( i ) + θ ij ( i, j ) + θ c ( c ) i i,j c ϵ Labels L = {l 1, l 2,, l k } Clique c V

92 Transforming roblems in QBFs Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions Multi-label Functions Pseudoboolean Functions

93 Transforming roblems in QBFs Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions Multi-label Functions Pseudoboolean Functions

94 Higher order to Quadratic Simle Eamle using Auiliary variables f() = { 0 if all i = 0 C 1 otherwise ϵ L = {0,1} n min f() =,a ϵ {0,1} Higher Order Submodular Function min C 1 a + C 1 ā i Quadratic Submodular Function i = 0 a=0 (ā=1) f() = 0 i 1 a=1 (ā=0) f() = C 1

95 Higher order to Quadratic min f() = Higher Order Submodular Function min C 1 a + C 1 ā i,a ϵ {0,1} Quadratic Submodular Function C 1 i C i

96 Higher order to Quadratic min f() = Higher Order Submodular Function min C 1 a + C 1 ā i,a ϵ {0,1} Quadratic Submodular Function C 1 i C 1 a=0 a=1 Lower envelo of concave functions is concave i

97 Higher order to Quadratic min f() = Higher Order Submodular Function min f 1 ()a + f 2 ()ā,a ϵ {0,1} Quadratic Submodular Function f 2 () f 1 () a=1 Lower envelo of concave functions is concave i

98 Higher order to Quadratic min f() = Higher Order Submodular Function min f 1 ()a + f 2 ()ā,a ϵ {0,1} Quadratic Submodular Function f 2 () a=0 f 1 () a=1 Lower envelo of concave functions is concave i

99 Transforming roblems in QBFs Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions Multi-label Functions Pseudoboolean Functions

100 Multi-label to Pseudo-boolean So what is the roblem? E m (y 1,y 2,..., y n ) E b ( 1, 2,..., m ) y i ϵ L = {l 1, l 2,, l k } i ϵ L = {0,1} Multi-label Problem Binary label Problem such that: Let Y and X be the set of feasible solutions, then 1. For each binary solution ϵ X with finite energy there eists eactly one multi-label solution y ϵ Y -> One-One encoding function T:X->Y 2. arg min E m (y) = T(arg min E b ())

101 Multi-label to Pseudo-boolean Poular encoding scheme [Roy and Co 98, Ishikawa 03, Schlesinger & Flach 06]

102 Multi-label to Pseudo-boolean Poular encoding scheme [Roy and Co 98, Ishikawa 03, Schlesinger & Flach 06] Ishikawa s result: E(y) = θ i (y i ) + θ ij (y i,y j ) i i,j y ϵ Labels L = {l 1, l 2,, l k } θ ij (y i,y j ) = g( y i -y j ) Conve Function g( y i -y j ) y i -y j

103 Multi-label to Pseudo-boolean Poular encoding scheme [Roy and Co 98, Ishikawa 03, Schlesinger & Flach 06] Schlesinger & Flach 06: E(y) = θ i (y i ) + θ ij (y i,y j ) i i,j y ϵ Labels L = {l 1, l 2,, l k } θ ij (l i+1,l j ) + θ ij (l i,l j+1 ) θ ij (l i,l j ) + θ ij (l i+1,l j+1 ) Covers all Submodular multi-label functions More general than Ishikawa

104 Multi-label to Pseudo-boolean Problems Alicability Only solves restricted class of energy functions Cannot handle Potts model otentials Comutational Cost Very high comutational cost Problem size = Variables Labels Gray level image denoising (1 Miel image) (~ grah nodes)

105 Outline of the Tutorial The st-mincut roblem Connection between st-mincut and energy minimization? What roblems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Oen Problems

106 St-mincut based Move algorithms E() = θ i ( i ) + θ ij ( i, j ) i i,j ϵ Labels L = {l 1, l 2,, l k } Commonly used for solving non-submodular multi-label roblems Etremely efficient and roduce good solutions Not Eact: Produce local otima

107 Energy Move Making Algorithms Solution Sace

108 Energy Move Making Algorithms Current Solution Search Neighbourhood Otimal Move Solution Sace

109 Energy Comuting the Otimal Move Current Solution Search Neighbourhood (t) c Otimal Move Key Proerty Move Sace Solution Sace Bigger move sace Better solutions Finding the otimal move hard

110 Moves using Grah Cuts Eansion and Swa move algorithms [Boykov Veksler and Zabih, PAMI 2001] Makes a series of changes to the solution (moves) Each move results in a solution with smaller energy Current Solution Search Neighbourhood Move Sace (t) : 2 N Sace of Solutions () : L N N L Number of Variables Number of Labels

111 Moves using Grah Cuts Eansion and Swa move algorithms [Boykov Veksler and Zabih, PAMI 2001] Makes a series of changes to the solution (moves) Each move results in a solution with smaller energy Current Solution Move to new solution Construct a move function Minimize move function to get otimal move How to minimize move functions?

112 General Binary Moves = t 1 + (1- t) 2 New solution Current Solution Second solution E m (t) = E(t 1 + (1- t) 2 ) Minimize over move variables t to get the otimal move Move energy is a submodular QPBF (Eact Minimization Possible) Boykov, Veksler and Zabih, PAMI 2001

113 Swa Move Variables labeled α, β can swa their labels [Boykov, Veksler, Zabih]

114 Swa Move Variables labeled α, β can swa their labels Swa Sky, House Tree Ground House Sky [Boykov, Veksler, Zabih]

115 Swa Move Variables labeled α, β can swa their labels Move energy is submodular if: Unary Potentials: Arbitrary Pairwise otentials: Semimetric θ ij (l a,l b ) 0 θ ij (l a,l b ) = 0 a = b Eamles: Potts model, Truncated Conve [Boykov, Veksler, Zabih]

116 Eansion Move Variables take label a or retain current label [Boykov, Veksler, Zabih]

117 Eansion Move Variables take label a or retain current label Status: Initialize Eand Ground House Sky with Tree Tree Ground House Sky [Boykov, Veksler, Zabih]

118 Eansion Move Variables take label a or retain current label Move energy is submodular if: Unary Potentials: Arbitrary Pairwise otentials: Metric θ ij (l a,l b ) + θ ij (l b,l c ) θ ij (l a,l c ) Semi metric + Triangle Inequality Eamles: Potts model, Truncated linear Cannot solve truncated quadratic [Boykov, Veksler, Zabih]

119 General Binary Moves = t 1 + (1-t) 2 New solution First solution Second solution Minimize over move variables t Move Tye First Solution Second Solution Guarantee Eansion Old solution All alha Metric Fusion Any solution Any solution Move functions can be non-submodular!!

120 Solving Continuous Problems using Fusion Move = t 1 + (1-t) 2 1, 2 can be continuous Otical Flow Eamle Solution from Method 2 2 Solution from Method 1 1 F Final Solution (Lemitsky et al. CVPR08, Woodford et al. CVPR08)

121 Range Moves Move variables can be multi-label = (t ==1) 1 + (t==2) 2 + +(t==k) k Otimal move found out by using the Ishikawa Useful for minimizing energies with truncated conve airwise otentials θ ij (y i,y j ) = min( y i -y j,t) θ ij (y i,y j ) T y i -y j O. Veksler, CVPR 2007

122 Move Algorithms for Solving Higher Order Energies E() = θ i ( i ) + θ ij ( i, j ) + θ c ( c ) i i,j c ϵ Labels L = {l 1, l 2,, l k } Clique c V Higher order functions give rise to higher order move energies Move energies for certain classes of higher order energies can be transformed to QPBFs. [Kohli, Kumar and Torr, CVPR07] [Kohli, Ladicky and Torr, CVPR08]

123 Outline of the Tutorial The st-mincut roblem Connection between st-mincut and energy minimization? What roblems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Oen Problems

124 Solving Mied Programming Problems binary image segmentation ( i {0,1}) ω non-local arameter (lives in some large set Ω) E(,ω) = C(ω) + θ i (ω, i ) + θ ij (ω, i, j ) i i,j constant unary otentials airwise otentials 0 ω Pose Stickman Model Rough Shae Prior θ i (ω, i ) Shae Prior

125 Oen Problems Characterization of Problems Solvable using st-mincut What functions can be transformed to submodular QBFs? Submodular Functions st-mincut Equivalent

126 Minimizing General Higher Order Functions We saw how simle higher order otentials can be solved How more sohisticated higher order otentials can be solved?

127 Summary Labelling Problem Eact Transformation (global otimum) Or Relaed transformation (artially otimal) Submodular Quadratic Pseudoboolean Function S st-mincut Sub-roblem T Move making algorithms

128 Thanks. Questions?

129 Stereo - Woodford et al. CVPR 2008 Use of Higher order Potentials E( 1, 2, 3 ) = θ 12 ( 1, 2 ) + θ 23 ( 2, 3 ) θ ij ( i, j ) = { 0 if i = j C otherwise Disarity Labels E(6,6,6) = = 0 P 1 P 2 P 3 Piels

130 Stereo - Woodford et al. CVPR 2008 Use of Higher order Potentials E( 1, 2, 3 ) = θ 12 ( 1, 2 ) + θ 23 ( 2, 3 ) θ ij ( i, j ) = { 0 if i = j C otherwise Disarity Labels E(6,6,6) = = 0 E(6,7,7) = = 1 P 1 P 2 P 3 Piels

131 Stereo - Woodford et al. CVPR 2008 Use of Higher order Potentials E( 1, 2, 3 ) = θ 12 ( 1, 2 ) + θ 23 ( 2, 3 ) θ ij ( i, j ) = { 0 if i = j C otherwise Disarity Labels E(6,6,6) = = 0 E(6,7,7) = = 1 E(6,7,8) = = 2 P 1 P 2 P 3 Piels Pairwise otential enalize slanted lanar surfaces

132 Comuting the Otimal Move Current Solution Search Neighbourhood E() (t) c T Otimal Move Transformation function T( c, t) = n = c + t

133 Comuting the Otimal Move Current Solution Search Neighbourhood E() (t) c T E m Otimal Move Transformation function Move Energy T( c, t) = n = c + t E m (t) = E(T( c, t))

134 Comuting the Otimal Move Current Solution Search Neighbourhood E() (t) c T E m Otimal Move Transformation function Move Energy T( c, t) = n = c + t E m (t) = E(T( c, t)) minimize t* Otimal Move