MAP Estimation Algorithms in Computer Vision - Part II

MAP Estimation Algorithms in Comuter Vision - Part II M. Pawan Kumar, University of Oford Pushmeet Kohli, Microsoft Research

Eamle: Image Segmentation E() = c i i + c ij i (1- j ) i i,j E: {0,1} n R 0 fg 1 bg n = number of iels Image (D)

Eamle: Image Segmentation E() = c i i + c ij i (1- j ) i i,j E: {0,1} n R 0 fg 1 bg n = number of iels Unary Cost (c i ) Dark (negative) Bright (ositive)

Eamle: Image Segmentation E() = c i i + c ij i (1- j ) i i,j E: {0,1} n R 0 fg 1 bg n = number of iels Discontinuity Cost (c ij )

Eamle: Image Segmentation E() = c i i + c ij i (1- j ) i i,j E: {0,1} n R 0 fg 1 bg n = number of iels * = arg min E() Global Minimum (*) How to minimize E()?

Outline of the Tutorial The st-mincut roblem Connection between st-mincut and energy minimization? What roblems can we solve using st-mincut? st-mincut based Move algorithms Recent Advances and Oen Problems

The st-mincut Problem Source 2 9 1 v 1 v 2 5 2 4 Grah (V, E, C) Vertices V = {v 1, v 2... v n } Edges E = {(v 1, v 2 )...} Costs C = {c (1, 2)...} Sink

The st-mincut Problem What is a st-cut? Source 2 1 9 v 1 v 2 5 2 4 Sink

The st-mincut Problem What is a st-cut? Source An st-cut (S,T) divides the nodes between source and sink. 2 9 1 v 1 v 2 5 2 4 What is the cost of a st-cut? Sum of cost of all edges going from S to T Sink 5 + 2 + 9 = 16

The st-mincut Problem What is a st-cut? Source v 1 v 2 5 2 9 1 2 Sink 4 2 + 1 + 4 = 7 An st-cut (S,T) divides the nodes between source and sink. What is the cost of a st-cut? Sum of cost of all edges going from S to T What is the st-mincut? st-cut with the minimum cost

How to comute the st-mincut? Source 2 9 1 v 1 v 2 5 2 Sink 4 Solve the dual maimum flow roblem Comute the maimum flow between Source and Sink Constraints Edges: Flow < Caacity Nodes: Flow in = Flow out Min-cut\Ma-flow Theorem In every network, the maimum flow equals the cost of the st-mincut

Maflow Algorithms Source v 1 v 2 5 Flow = 0 2 9 1 2 Sink 4 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

Maflow Algorithms Source 2-2 9 1 v 1 v 2 5-2 Flow = 0 + 2 2 Sink 4 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

Maflow Algorithms Source 0 9 1 v 1 v 2 3 Flow = 2 2 Sink 4 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

Maflow Algorithms Source 0 5 1 v 1 v 2 3 Flow = 2 + 4 2 Sink 0 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

Maflow Algorithms Source 0 5 1 v 1 v 2 3 Flow = 6 2 Sink 0 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

Maflow Algorithms Source 0 4 1-1 v 1 v 2 2 Flow = 6 + 1 2+1 Sink 0 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

Maflow Algorithms Source 0 4 0 v 1 v 2 2 Flow = 7 3 Sink 0 Augmenting Path Based Algorithms 1. Find ath from source to sink with ositive caacity 2. Push maimum ossible flow through this ath 3. Reeat until no ath can be found Algorithms assume non-negative caacity

History of Maflow Algorithms Augmenting Path and Push-Relabel n: #nodes m: #edges U: maimum edge weight Algorithms assume nonnegative edge weights [Slide credit: Andrew Goldberg]

Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting ath Source 1000 1 1000 a 1 a 2 0 1000 1000 Sink

Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting ath Source 1000 1 1000 a 1 a 2 0 1000 1000 Sink Bad Augmenting Paths

Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting ath Source 1000 1 1000 a 1 a 2 0 1000 1000 Sink Bad Augmenting Path

Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting ath Source 999 0 1000 a 1 a 2 1 1000 999 Sink

Augmenting Path based Algorithms Ford Fulkerson: Choose any augmenting ath n: #nodes m: #edges Source 999 0 1000 a 1 a 2 1 1000 999 Sink We will have to erform 2000 augmentations! Worst case comleity: O (m Total_Flow) (Pseudo-olynomial bound: deends on flow)

Augmenting Path based Algorithms Dinic: Choose shortest augmenting ath n: #nodes m: #edges Source 1000 1 1000 a 1 a 2 0 1000 1000 Sink Worst case Comleity: O (m n 2 )

Maflow in Comuter Vision Secialized algorithms for vision roblems Grid grahs Low connectivity (m ~ O(n)) Dual search tree augmenting ath algorithm [Boykov and Kolmogorov PAMI 2004] Finds aroimate shortest augmenting aths efficiently High worst-case time comleity Emirically outerforms other algorithms on vision roblems

St-mincut and Energy Minimization S st-mincut T Minimizing a Qudratic Pseudoboolean function E() Functions of boolean variables Pseudoboolean? E: {0,1} n R E() = c i i + c ij i (1- j ) i i,j c ij 0 Polynomial time st-mincut algorithms require non-negative edge weights

So how does this work? Construct a grah such that: 1. Any st-cut corresonds to an assignment of 2. The cost of the cut is equal to the energy of : E() S st-mincut E() T Solution

Grah Construction E(a 1,a 2 ) Source (0) a 1 a 2 Sink (1)

Grah Construction E(a 1,a 2 ) = 2a 1 2 Source (0) a 1 a 2 Sink (1)

Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 2 Source (0) a 1 a 2 5 Sink (1)

Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 Source (0) 2 9 a 1 a 2 5 4 Sink (1)

Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 Source (0) 2 9 a 1 a 2 5 2 4 Sink (1)

Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 9 1 Source (0) a 1 a 2 5 2 4 Sink (1)

Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Source (0) 2 9 Cost of cut = 11 1 a 1 a 2 a 1 = 1 a 2 = 1 5 2 4 E (1,1) = 11 Sink (1)

Grah Construction E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Source (0) 2 9 1 a 1 a 2 st-mincut cost = 8 a 1 = 1 a 2 = 0 5 2 4 E (1,0) = 8 Sink (1)

Energy Function Rearameterization Two functions E 1 and E 2 are rearameterizations if E 1 () = E 2 () for all For instance: E 1 (a 1 ) = 1+ 2a 1 + 3ā 1 E 2 (a 1 ) = 3 + ā 1 a 1 ā 1 1+ 2a 1 + 3ā 1 3 + ā 1 0 1 4 4 1 0 3 3

Flow and Rearametrization E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 2 9 1 Source (0) a 1 a 2 5 2 4 Sink (1)

Flow and Rearametrization E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Source (0) 2 9 1 a 1 a 2 2 5 4 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1 Sink (1)

Flow and Rearametrization E(a 1,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Source (0) 0 9 1 a 1 a 2 2 3 4 2a 1 + 5ā 1 = 2(a 1 +ā 1 ) + 3ā 1 = 2 + 3ā 1 Sink (1)

Flow and Rearametrization E(a 1,a 2 ) = 2 + 3ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Source (1) 0 9 1 a 1 a 2 2 3 4 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2 Sink (0)

Flow and Rearametrization E(a 1,a 2 ) = 2 + 3ā 1 + 5a 2 + 4 + 2a 1 ā 2 + ā 1 a 2 Source (1) 0 5 1 a 1 a 2 2 3 0 9a 2 + 4ā 2 = 4(a 2 +ā 2 ) + 5ā 2 = 4 + 5ā 2 Sink (0)

Flow and Rearametrization E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 Source (1) 0 5 1 a 1 a 2 3 2 0 Sink (0)

Flow and Rearametrization E(a 1,a 2 ) = 6 + 3ā 1 + 5a 2 + 2a 1 ā 2 + ā 1 a 2 3ā 1 + 5a 2 + 2a 1 ā 2 Source (1) 0 5 1 a 1 a 2 2 3 0 Sink (0) = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2 a 1 a 2 F1 F2 0 0 1 1 0 1 2 2 1 0 1 1 1 1 1 1

Flow and Rearametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 3ā 1 + 5a 2 + 2a 1 ā 2 Source (1) 0 3 3 a 1 a 2 0 1 0 Sink (0) = 2(ā 1 +a 2 +a 1 ā 2 ) +ā 1 +3a 2 = 2(1+ā 1 a 2 ) +ā 1 +3a 2 F1 = ā 1 +a 2 +a 1 ā 2 F2 = 1+ā 1 a 2 a 1 a 2 F1 F2 0 0 1 1 0 1 2 2 1 0 1 1 1 1 1 1

Flow and Rearametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Source (1) 0 3 3 a 1 a 2 0 1 0 No more augmenting aths ossible Sink (0)

Flow and Rearametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Residual Grah (ositive coefficients) Total Flow bound on the otimal solution Source (1) 0 3 3 a 1 a 2 1 0 0 Sink (0) Inference of the otimal solution becomes trivial because the bound is tight

Flow and Rearametrization E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 Residual Grah (ositive coefficients) Total Flow bound on the otimal solution Source (1) 0 3 3 a 1 a 2 st-mincut cost = 8 a 1 = 1 a 2 = 0 1 0 0 E (1,0) = 8 Sink (0) Inference of the otimal solution becomes trivial because the bound is tight

Eamle: Image Segmentation E() = c i i + c ij i (1- j ) i i,j E: {0,1} n R 0 fg 1 bg * = arg min E() Global Minimum (*) How to minimize E()?

How does the code look like? Grah *g; For all iels /* Add a node to the grah */ nodeid() = g->add_node(); Source (0) /* Set cost of terminal edges */ set_weights(nodeid(), fgcost(), bgcost()); end for all adjacent iels,q add_weights(nodeid(), nodeid(q), cost); end g->comute_maflow(); label_ = g->is_connected_to_source(nodeid()); Sink (1) // is the label of iel (0 or 1)

How does the code look like? Grah *g; For all iels end /* Add a node to the grah */ nodeid() = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeid(), fgcost(), bgcost()); for all adjacent iels,q add_weights(nodeid(), nodeid(q), cost); end g->comute_maflow(); label_ = g->is_connected_to_source(nodeid()); Source (0) bgcost(a 1 ) bgcost(a 2 ) a 1 a 2 fgcost(a 1 ) fgcost(a 2 ) Sink (1) // is the label of iel (0 or 1)

How does the code look like? Grah *g; For all iels end /* Add a node to the grah */ nodeid() = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeid(), fgcost(), bgcost()); for all adjacent iels,q add_weights(nodeid(), nodeid(q), cost(,q)); end g->comute_maflow(); label_ = g->is_connected_to_source(nodeid()); Source (0) bgcost(a 1 ) bgcost(a 2 ) cost(,q) a 1 a 2 fgcost(a 1 ) fgcost(a 2 ) Sink (1) // is the label of iel (0 or 1)

How does the code look like? Grah *g; For all iels end /* Add a node to the grah */ nodeid() = g->add_node(); /* Set cost of terminal edges */ set_weights(nodeid(), fgcost(), bgcost()); for all adjacent iels,q add_weights(nodeid(), nodeid(q), cost(,q)); end g->comute_maflow(); label_ = g->is_connected_to_source(nodeid()); // is the label of iel (0 or 1) Source (0) bgcost(a 1 ) bgcost(a 2 ) cost(,q) a 1 a 2 fgcost(a 1 ) fgcost(a 2 ) Sink (1) a 1 = bg a 2 = fg

Image Segmentation in Video n-links s = 0 st-cut E() * w q t = 1 Image Flow Global Otimum

Image Segmentation in Video Image Flow Global Otimum

Boykov & Jolly ICCV 01, Kohli & Torr (ICCV05, PAMI07) Dynamic Energy Minimization E A minimize S A Can we do better? Recycling Solutions E B comutationally eensive oeration S B

Dynamic Energy Minimization E A minimize S A Reuse flow A and B similar differences between A and B E B* Simler energy cheaer oeration E B comutationally eensive oeration S B Rearametrization 3 100000 time seedu! Boykov & Jolly ICCV 01, Kohli && Torr Torr (ICCV05, PAMI07)

Dynamic Energy Minimization Original Energy E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 2a 1 ā 2 + ā 1 a 2 Rearametrized Energy E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 New Energy E(a 1,a 2 ) = 2a 1 + 5ā 1 + 9a 2 + 4ā 2 + 7a 1 ā 2 + ā 1 a 2 New Rearametrized Energy E(a 1,a 2 ) = 8 + ā 1 + 3a 2 + 3ā 1 a 2 + 5a 1 ā 2 Kohli & Torr (ICCV05, PAMI07) Boykov & Jolly ICCV 01, Kohli & Torr (ICCV05, PAMI07)

Minimizing Energy Functions General Energy Functions NP-hard to minimize Only aroimate minimization ossible Easy energy functions Solvable in olynomial time Submodular ~ O(n 6 ) MAXCUT NP-Hard Submodular Functions Functions defined on trees Sace of Function Minimization Problems

Submodular Set Functions Let E= {a 1,a 2,... a n } be a set 2 E = #subsets of E Set function f 2 E R

Submodular Set Functions Let E= {a 1,a 2,... a n } be a set 2 E = #subsets of E Set function f 2 E R is submodular if f(a) + f(b) f(ab) + f(ab) for all A,B E E A B Imortant Proerty Sum of two submodular functions is submodular

Minimizing Submodular Functions Minimizing general submodular functions O(n 5 Q + n 6 ) where Q is function evaluation time [Orlin, IPCO 2007] Symmetric submodular functions E () = E (1 - ) O(n 3 ) [Queyranne 1998] Quadratic seudoboolean Can be transformed to st-mincut One node er variable [ O(n 3 ) comleity] Very low emirical running time

Submodular Pseudoboolean Functions Function defined over boolean vectors = { 1, 2,... n } Definition: All functions for one boolean variable (f: {0,1} -> R) are submodular A function of two boolean variables (f: {0,1} 2 -> R) is submodular if f(0,1) + f(1,0) f(0,0) + f(1,1) A general seudoboolean function f 2 n R is submodular if all its rojections f are submodular i.e. f (0,1) + f (1,0) f (0,0) + f (1,1)

Quadratic Submodular Pseudoboolean Functions E() = θ i ( i ) + θ ij ( i, j ) i i,j For all ij θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1)

Quadratic Submodular Pseudoboolean Functions E() = θ i ( i ) + θ ij ( i, j ) i i,j For all ij θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1) Equivalent (transformable) E() = c i i + c ij i (1- j ) i i,j c ij 0 i.e. All submodular QPBFs are st-mincut solvable

How are they equivalent? A = θ ij (0,0) B = θ ij (0,1) C = θ ij (1,0) D = θ ij (1,1) i 0 1 j 0 1 A B C D = A + 0 1 0 1 0 0 C-A C-A 0 1 0 1 0 D-C + + 0 D-C 0 1 0 1 0 B+C -A-D 0 0 if 1 =1 add C-A if 2 = 1 add D-C θ ij ( i, j ) = θ ij (0,0) + (θ ij (1,0)-θ ij (0,0)) i + (θ ij (1,0)-θ ij (0,0)) j + (θ ij (1,0) + θ ij (0,1) - θ ij (0,0) - θ ij (1,1)) (1- i ) j B+C-A-D 0 is true from the submodularity of θ ij

Quadratic Submodular Pseudoboolean Functions E() = θ i ( i ) + θ ij ( i, j ) i i,j in {0,1} n For all ij θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1) Equivalent (transformable) S st-mincut T

Minimizing Non-Submodular Functions E() = θ i ( i ) + θ ij ( i, j ) i i,j θ ij (0,1) + θ ij (1,0) θ ij (0,0) + θ ij (1,1) for some ij Minimizing general non-submodular functions is NP-hard. Commonly used method is to solve a relaation of the roblem [Slide credit: Carsten Rother]

Minimization using Roof-dual Relaation E({ }) ( ) unary ~ q q ( (,, q q ) ) ~ q ( 0,0) q(1,1) q(0,1) q(1,0) airwise submodular q ~ (0,0) q (1,1) ~ ~ (0,1) airwise nonsubmodular q q (1,0) [Slide credit: Carsten Rother]

2 ), (1 ~ ),1 ( ~ 2 ),1 (1 ), ( 2 ) (1 ) ( }) },{ '({ q q q q q q q q E Double number of variables: ), ( ~ ), ( ) ( }) ({ q q q q E, Minimization using Roof-dual Relaation ) 1 ( [Slide credit: Carsten Rother]

2 ), (1 ~ ),1 ( ~ 2 ),1 (1 ), ( 2 ) (1 ) ( }) },{ '({ q q q q q q q q E Double number of variables: ), ( ~ ), ( ) ( }) ({ q q q q E, Minimization using Roof-dual Relaation ),1 ( ~ ), ( q q q q f (1,0) ~ (0,1) ~ (1,1) ~ (0,0) ~ q q q q (1,1) (0,0) (1,0) 0,1) ( q q q q f f f f Non- submodular Submodular ) 1 (

2 ), (1 ~ ),1 ( ~ 2 ),1 (1 ), ( 2 ) (1 ) ( }) },{ '({ q q q q q q q q E Double number of variables: ), ( ~ ), ( ) ( }) ({ q q q q E, Minimization using Roof-dual Relaation ) 1 ( is submodular! Ignore (solvable using st-mincut) Proerty of the roblem: 1

1 is the otimal label Proerty of the solution: 2 ), (1 ~ ),1 ( ~ 2 ),1 (1 ), ( 2 ) (1 ) ( }) },{ '({ q q q q q q q q E Double number of variables: ), ( ~ ), ( ) ( }) ({ q q q q E, ) 1 ( Minimization using Roof-dual Relaation

Reca Eact minimization of Submodular QBFs using grah cuts. Obtaining artially otimal solutions of nonsubmodular QBFs using grah cuts.

But... Need higher order energy functions to model image structure Field of eerts [Roth and Black] Many roblems in comuter vision involve multile labels E() = θ i ( i ) + θ ij ( i, j ) + θ c ( c ) i i,j c ϵ Labels L = {l 1, l 2,, l k } Clique c V

Transforming roblems in QBFs Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions Multi-label Functions Pseudoboolean Functions

Higher order to Quadratic Simle Eamle using Auiliary variables f() = { 0 if all i = 0 C 1 otherwise ϵ L = {0,1} n min f() =,a ϵ {0,1} Higher Order Submodular Function min C 1 a + C 1 ā i Quadratic Submodular Function i = 0 a=0 (ā=1) f() = 0 i 1 a=1 (ā=0) f() = C 1

Higher order to Quadratic min f() = Higher Order Submodular Function min C 1 a + C 1 ā i,a ϵ {0,1} Quadratic Submodular Function C 1 i C 1 1 2 3 i

Higher order to Quadratic min f() = Higher Order Submodular Function min C 1 a + C 1 ā i,a ϵ {0,1} Quadratic Submodular Function C 1 i C 1 a=0 a=1 Lower envelo of concave functions is concave 1 2 3 i

Higher order to Quadratic min f() = Higher Order Submodular Function min f 1 ()a + f 2 ()ā,a ϵ {0,1} Quadratic Submodular Function f 2 () f 1 () a=1 Lower envelo of concave functions is concave 1 2 3 i

Higher order to Quadratic min f() = Higher Order Submodular Function min f 1 ()a + f 2 ()ā,a ϵ {0,1} Quadratic Submodular Function f 2 () a=0 f 1 () a=1 Lower envelo of concave functions is concave 1 2 3 i

Transforming roblems in QBFs Higher order Pseudoboolean Functions Quadratic Pseudoboolean Functions Multi-label Functions Pseudoboolean Functions

Multi-label to Pseudo-boolean So what is the roblem? E m (y 1,y 2,..., y n ) E b ( 1, 2,..., m ) y i ϵ L = {l 1, l 2,, l k } i ϵ L = {0,1} Multi-label Problem Binary label Problem such that: Let Y and X be the set of feasible solutions, then 1. For each binary solution ϵ X with finite energy there eists eactly one multi-label solution y ϵ Y -> One-One encoding function T:X->Y 2. arg min E m (y) = T(arg min E b ())

Multi-label to Pseudo-boolean Poular encoding scheme [Roy and Co 98, Ishikawa 03, Schlesinger & Flach 06]

Multi-label to Pseudo-boolean Poular encoding scheme [Roy and Co 98, Ishikawa 03, Schlesinger & Flach 06] Ishikawa s result: E(y) = θ i (y i ) + θ ij (y i,y j ) i i,j y ϵ Labels L = {l 1, l 2,, l k } θ ij (y i,y j ) = g( y i -y j ) Conve Function g( y i -y j ) y i -y j

Multi-label to Pseudo-boolean Poular encoding scheme [Roy and Co 98, Ishikawa 03, Schlesinger & Flach 06] Schlesinger & Flach 06: E(y) = θ i (y i ) + θ ij (y i,y j ) i i,j y ϵ Labels L = {l 1, l 2,, l k } θ ij (l i+1,l j ) + θ ij (l i,l j+1 ) θ ij (l i,l j ) + θ ij (l i+1,l j+1 ) Covers all Submodular multi-label functions More general than Ishikawa

Multi-label to Pseudo-boolean Problems Alicability Only solves restricted class of energy functions Cannot handle Potts model otentials Comutational Cost Very high comutational cost Problem size = Variables Labels Gray level image denoising (1 Miel image) (~2.5 10 8 grah nodes)

St-mincut based Move algorithms E() = θ i ( i ) + θ ij ( i, j ) i i,j ϵ Labels L = {l 1, l 2,, l k } Commonly used for solving non-submodular multi-label roblems Etremely efficient and roduce good solutions Not Eact: Produce local otima

Energy Move Making Algorithms Solution Sace

Energy Move Making Algorithms Current Solution Search Neighbourhood Otimal Move Solution Sace

Energy Comuting the Otimal Move Current Solution Search Neighbourhood (t) c Otimal Move Key Proerty Move Sace Solution Sace Bigger move sace Better solutions Finding the otimal move hard

Moves using Grah Cuts Eansion and Swa move algorithms [Boykov Veksler and Zabih, PAMI 2001] Makes a series of changes to the solution (moves) Each move results in a solution with smaller energy Current Solution Search Neighbourhood Move Sace (t) : 2 N Sace of Solutions () : L N N L Number of Variables Number of Labels

Moves using Grah Cuts Eansion and Swa move algorithms [Boykov Veksler and Zabih, PAMI 2001] Makes a series of changes to the solution (moves) Each move results in a solution with smaller energy Current Solution Move to new solution Construct a move function Minimize move function to get otimal move How to minimize move functions?

General Binary Moves = t 1 + (1- t) 2 New solution Current Solution Second solution E m (t) = E(t 1 + (1- t) 2 ) Minimize over move variables t to get the otimal move Move energy is a submodular QPBF (Eact Minimization Possible) Boykov, Veksler and Zabih, PAMI 2001

Swa Move Variables labeled α, β can swa their labels [Boykov, Veksler, Zabih]

Swa Move Variables labeled α, β can swa their labels Swa Sky, House Tree Ground House Sky [Boykov, Veksler, Zabih]

Swa Move Variables labeled α, β can swa their labels Move energy is submodular if: Unary Potentials: Arbitrary Pairwise otentials: Semimetric θ ij (l a,l b ) 0 θ ij (l a,l b ) = 0 a = b Eamles: Potts model, Truncated Conve [Boykov, Veksler, Zabih]

Eansion Move Variables take label a or retain current label [Boykov, Veksler, Zabih]

Eansion Move Variables take label a or retain current label Status: Initialize Eand Ground House Sky with Tree Tree Ground House Sky [Boykov, Veksler, Zabih]

Eansion Move Variables take label a or retain current label Move energy is submodular if: Unary Potentials: Arbitrary Pairwise otentials: Metric θ ij (l a,l b ) + θ ij (l b,l c ) θ ij (l a,l c ) Semi metric + Triangle Inequality Eamles: Potts model, Truncated linear Cannot solve truncated quadratic [Boykov, Veksler, Zabih]

General Binary Moves = t 1 + (1-t) 2 New solution First solution Second solution Minimize over move variables t Move Tye First Solution Second Solution Guarantee Eansion Old solution All alha Metric Fusion Any solution Any solution Move functions can be non-submodular!!

Solving Continuous Problems using Fusion Move = t 1 + (1-t) 2 1, 2 can be continuous Otical Flow Eamle Solution from Method 2 2 Solution from Method 1 1 F Final Solution (Lemitsky et al. CVPR08, Woodford et al. CVPR08)

Range Moves Move variables can be multi-label = (t ==1) 1 + (t==2) 2 + +(t==k) k Otimal move found out by using the Ishikawa Useful for minimizing energies with truncated conve airwise otentials θ ij (y i,y j ) = min( y i -y j,t) θ ij (y i,y j ) T y i -y j O. Veksler, CVPR 2007

Move Algorithms for Solving Higher Order Energies E() = θ i ( i ) + θ ij ( i, j ) + θ c ( c ) i i,j c ϵ Labels L = {l 1, l 2,, l k } Clique c V Higher order functions give rise to higher order move energies Move energies for certain classes of higher order energies can be transformed to QPBFs. [Kohli, Kumar and Torr, CVPR07] [Kohli, Ladicky and Torr, CVPR08]

Solving Mied Programming Problems binary image segmentation ( i {0,1}) ω non-local arameter (lives in some large set Ω) E(,ω) = C(ω) + θ i (ω, i ) + θ ij (ω, i, j ) i i,j constant unary otentials airwise otentials 0 ω Pose Stickman Model Rough Shae Prior θ i (ω, i ) Shae Prior

Oen Problems Characterization of Problems Solvable using st-mincut What functions can be transformed to submodular QBFs? Submodular Functions st-mincut Equivalent

Minimizing General Higher Order Functions We saw how simle higher order otentials can be solved How more sohisticated higher order otentials can be solved?

Summary Labelling Problem Eact Transformation (global otimum) Or Relaed transformation (artially otimal) Submodular Quadratic Pseudoboolean Function S st-mincut Sub-roblem T Move making algorithms

Thanks. Questions?

Stereo - Woodford et al. CVPR 2008 Use of Higher order Potentials E( 1, 2, 3 ) = θ 12 ( 1, 2 ) + θ 23 ( 2, 3 ) θ ij ( i, j ) = { 0 if i = j C otherwise Disarity Labels 8 7 6 5 E(6,6,6) = 0 + 0 = 0 P 1 P 2 P 3 Piels

Stereo - Woodford et al. CVPR 2008 Use of Higher order Potentials E( 1, 2, 3 ) = θ 12 ( 1, 2 ) + θ 23 ( 2, 3 ) θ ij ( i, j ) = { 0 if i = j C otherwise Disarity Labels 8 7 6 5 E(6,6,6) = 0 + 0 = 0 E(6,7,7) = 1 + 0 = 1 P 1 P 2 P 3 Piels

Comuting the Otimal Move Current Solution Search Neighbourhood E() (t) c T Otimal Move Transformation function T( c, t) = n = c + t

Comuting the Otimal Move Current Solution Search Neighbourhood E() (t) c T E m Otimal Move Transformation function Move Energy T( c, t) = n = c + t E m (t) = E(T( c, t))

Comuting the Otimal Move Current Solution Search Neighbourhood E() (t) c T E m Otimal Move Transformation function Move Energy T( c, t) = n = c + t E m (t) = E(T( c, t)) minimize t* Otimal Move