Energy Minimization via Graph Cuts

Energy Minimization via Graph Cuts Xiaowei Zhou, June 11, 2010, Journal Club Presentation 1

outline Introduction MAP formulation for vision problems Min-cut and Max-flow Problem Energy Minimization via Graph Cuts Binary labeling problem Multiple labeling problem Connecting Graph-cuts with Active Contour Discussion 2

MAP Framework (MRF) Many vision problems require estimating some spatially varying quantity from noisy measurements. Markov Random Field (Geman and Geman, 1984) A Graphic Model to describe the posterior probability of the quantity to be estimated given the measurements. 4

MAP Framework (MRF) fˆ fˆ arg max Pr ( f O) f arg max Pr( O f ) Pr( f ) Likelihood function (data term or unary term) f arg max exp ln g p( O fp) f p ( p, q Prior (MRF model or interaction term) ) V ( p, q) ( fp, fq) The interaction term makes the optimization of MRF very hard! 5

Min-cut Problem Find the minimum s-t cut of a graph An s-t cut C = (S,T) of a graph G = (V, E) is a cut of G such that s S and t T, where s and t are the source and the sink of N respectively. Cost of an s-t cut: C uv, (u,v) E,uS,v T Source Min-Cut = Max-Flow Polynomial-time solution Augmenting Path: O( E 2 V ) Push-Relabel: O( E V 2 ) 2 9 2 v 1 v 2 5 1 4 6 Sink

Intuition Image Graph n-links t s Segmentation Cut 7

outline Overview MAP formulation for vision problems Min-cut and Max-flow Problem Energy Minimization via Graph Cuts Binary labeling problem Multiple labeling problem Connecting Graph-cuts with Active Contour Discussion 8

Questions How can the graph-based algorithms help to solve the energy minimization problem? What kind of energy can be minimized via Graph Cuts? What about multi-labeling problem? 9

Main idea The main idea of graph-cut based MRF optimization is to construct a graph such that 1. there is a one-to-one correspondence between cuts of graph and configurations of the MRF 2. the total cost of the cut is exactly the same as the total energy of the configuration. 10

Example: Generalized Potts Model V Clique potential ( fp, fq) u{ p, q} ( f ( p, q) p q f ) Penalty Weight for discontinuity at (p,q) E f D f u f f ( ) p( p) { p, q} ( p q) p { p, q} 11

Solve the Potts model via Graph Cuts E f D f u f f ( ) p( p) { p, q} ( p q) p { p, q} D p (s) n-links t w pq s t-links a cut D p (t) n-links f p { s, t} A s-t cut corresponds to one configuration of binary labeling C( S, T) E( f ) 12

What about other energy functions? Let E be a function of n binary variables with the following form: E( L) D ( f ) V ( f, f ) p p p p q p, qn Then, E is graph-representable if and only if [3]: V(0,0) V(1,1) V(0,1) V(1,0) Functions satisfying above condition are called regular 13

Constructive Proof [3] To prove: given a regular function, a graph can be constructed to represent the function. Additivity Theorem: The sum of two graph-representable functions is graphrepresentable. Graph Construction: Dp( fp) is easy to represent. How to represent V ( f p, fq)? 14

Graph Construction Decompose the original function into: Part 1 Part 2 Part 3 Part 1 and Part 2 must be Graph-representable Part 3 depends on the sign of B+C-A-D If B+C-A-D >= 0, it s ok Else, impossible S C-A B+C-A-D p t q C-D 15

Alternative understanding Equations describing correspondence p S x1 2 x 3 x 5 x 4 t Conditions of existence of solution: x x q x 6 x5 x6 A k x1 x2 D k x2 x3 x5 B k x1 x4 x6 C k xi 0 3 x4 B C A D 0 To construct the graph: Minimize Cardinality of X to reduce the number of edges, subject to above constraints 16

Multi-label energy minimization break multi-way cut into a sequence of binary s-t cuts a-expansion move: a other labels 1. Start with any initial solution 2. For each label a in any order 1. Compute optimal a- expansion move (s-t graph cuts) 2. Decline the move if there is no energy decrease 3. Stop when no expansion move would decrease energy 17

How to construct graph? How to construct the graph for a-expansion move? Energy function for each a-expansion move: Define a mapping: f( x ) p xp 0 0 fp xp 1 We can get an binary-label energy function, and it is consistent with the original energy function: E( x) D ( f ( x )) V ( f ( x ), f ( x )) p p p p q p, qn s-t cuts can be used to minimize the above energy 18

More about a-expansion Conditions: Graph-representable: V ( f (0), f (0)) V ( f (1), f (1)) V ( f (0), f (1)) V ( f (1), f (0)) V (, ) V (, ) V (, ) V (, ) Optimality: Guaranteed approximation within a constant factor from the global minima (2 for the Potts model). Detailed proof is given in [1]. 19

Geodesic Active Contour The shortest curve between two points is a geodesic distance map B distance map B A A Euclidian metric (constant) Riemannian metric (space varying, tensor D(p)) E C g I C s C s ds ' geodesic ( ) ( ( ( )) ) ( ) C Image gradient based Riemannian metric 21

Another View of Curve Length C C e ec Curve length can be measured by how many edges are cut by the curve. How to determine the edge weight? 22

Curve length in integral geometry Cauchy-Crofton formula: C 2 a subset of lines L intersecting contour C a set of all lines L L C 0 Euclidean length of C : C 1 2 n L d d the number of times line L intersects C 23

Discrete Approximation Edges of any regular neighborhood system generate families of lines C C Euclidean length 1 2 k n k the number of edges of family k intersecting C k k C gc graph cut cost for edge weights: w k k k 2 24

Riemannian metric Riemannian length of C : det D C n d d R 1 2 T 3/2 L 2( uldul) w k det D k k T 3/2 2( uldul) 2 D is the local Riemannian metric u L is the unit vector along the direction of curve 25

Region-based Active Contour E C I u d I u d length C 2 2 CV ( ) ( ) ( ) ( ) outside( C) in X outside( C) out X {( I u ) ( f ' in') ( I u ) ( f ' out ')} dx length( C) 2 2 in X out X Boundary term (prior term) Region term (data term) E( L) D ( f ) w ( f f ) p p p pq p q pqn Both are formulated in MAP framework Continuous representation vs. Discrete representation Variational Optimization vs. Discrete Optimization 26

Benefits of Graph-cuts based segmentation Global optimal solution via combinatorial optimization Numerically stable Convenient to add hard constraints Interactive segmentation e.g. Grab-cut [5]: Commercial application in Office 2010 28

Limitations of Graph-cuts Discrete artifacts Geometric artifacts Can t reach sub-pixel accuracy Higher requirements on energy function design Global optimal solution of the energy function is not guaranteed to be what we what. Model bias Model inaccuracy Parameter setting 29

Other Issues Non-regular Energy Minimization QPBO [6] Other optimization methods ICM Message Passing Higher Order Potential Transform to sum of first order potentials [7] Comparison to other discrete optimization methods http://vision.middlebury.edu/mrf/results/ 30

Key References [1] Y. Boykov, O. Veksler and R. Zabih, "Fast approximate energy minimization via graph cuts," IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, pp. 1222-1239, 2001. [2] Y. Boykov and V. Kolmogorov, "Computing geodesics and minimal surfaces via graph cuts," in Proceedings of the Ninth IEEE International Conference on Computer Vision- Volume 2, 2003, pp. 26. [3] V. Kolmogorov and R. Zabih, "What Energy Functions Can Be Minimized via Graph Cuts?" IEEE Trans. Pattern Anal. Mach. Intell., vol. 26, pp. 147, 2004. [4] Y. Boykov and G. Funka-Lea, "Graph cuts and efficient n-d image segmentation," International Journal of Computer Vision, vol. 70, pp. 109-131, 2006. [5] C. Rother, V. Kolmogorov and A. Blake, "Grabcut: Interactive foreground extraction using iterated graph cuts," in ACM SIGGRAPH 2004 Papers, 2004, pp. 314. [6] V. Kolmogorov and C. Rother, "Minimizing nonsubmodular functions with graph cutsa review," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, pp. 1274-1279, 2007. [7] Hiroshi Ishikawa, "Transformation of General Binary MRF Minimization to the First Order Case," IEEE Transactions on Pattern Analysis and Machine Intelligence, 31 Mar. 2010. 31