Fast, Approximately Optimal Solutions for Single and Dynamic MRFs

Transcription

1 Fast, Aroximately Otimal Solutions for Single and Dynami MRFs Nikos Komodakis, Georgios Tziritas University of Crete, Comuter Siene Deartment Nikos Paragios MAS, Eole Centrale de Paris Abstrat A new effiient MRF otimization algorithm, alled Fast- PD, is roosed, whih generalizes α-exansion. One of its main advantages is that it offers a substantial seedu over that method, e.g. it an be at least 3-9 times faster than α-exansion. Its effiieny is a result of the fat that Fast-PD exloits information oming not only from the original MRF roblem, but also from a dual roblem. Furthermore, besides stati MRFs, it an also be used for boosting the erformane of dynami MRFs, i.e. MRFs varying over time. On to of that, Fast-PD makes no omromise about the otimality of its solutions: it an omute exatly the same answer as α-exansion, but, unlike that method, it an also guarantee an almost otimal solution for a muh wider lass of NP-hard MRF roblems. Results on stati and dynami MRFs demonstrate the algorithm s effiieny and ower. E.g., Fast-PD has been able to omute disarity for stereosoi sequenes in real time, with the resulting disarity oiniding with that of α-exansion.. Introdution Disrete MRFs are ubiquitous in omuter vision, and thus otimizing them is a roblem of fundamental imortane. Aording to it, given a weighted grah G (with nodes V, edges E and weights w q, one seeks to assign a label x (from a disrete set of labels L to ea V, so that the following ost is minimized: + w qd, x q. ( V (,q E Here, (, d(, determine the singleton and airwise MRF otential funtions resetively. U to now, grah-ut based methods, like α-exansion [3], have been very effetive in MRF otimization, generating solutions with good otimality roerties [8]. However, besides solutions otimality, another imortant issue is that of omutational effiieny. In fat, this issue has reently been looked at for the seial ase of dynami MRFs [5, 4], i.e. MRFs varying over time. Thus, trying to onentrate on both of these issues here, we raise the following questions: an there be a grah-ut based method, whih will be more effiient, but equally (or even more owerful, than α-exansion, for the ase of single MRFs? Furthermore, This work was artially suorted from the Frenh ANR-Blan grant SURF (5-8 and Platon (6-7. an that method also offer a omutational advantage for the ase of dynami MRFs? With reset to the questions raised above, this work makes the following ontributions. Effiieny for single MRFs: α-exansion works by solving a series of max-flow roblems. Its effiieny is thus largely determined from the effiieny of these max-flow roblems, whih, in turn, deends on the number of augmenting aths er max-flow. Here, we build uon reent work of [6], and roose a new rimal-dual MRF otimization method, alled Fast-PD. This method, like [6] or α-exansion, also ends u solving a max-flow roblem for a series of grahs. However, unlike these tehniques, the grahs onstruted by Fast-PD ensure that the number of er max-flow dereases dramatially over time, thus boosting the effiieny of MRF inferene. To show this, we rove a generalized relationshi between the number of and the so-alled rimal-dual ga assoiated with the original MRF roblem and its dual. Furthermore, to fully exloit the above roerty, new extensions are also roosed: an adated max-flow algorithm, as well as an inremental grah onstrution method. Otimality roerties: Desite its effiieny, our method also makes no omromise regarding the otimality of its solutions. So, if d(, is a metri, Fast-PD is as owerful as α-exansion, i.e. it omutes exatly the same solution, but with a substantial seedu. Moreover, it alies to a muh wider lass of MRFs, e.g. even with a non-metri d(,, while still guaranteeing an almost otimal solution. Effiieny for dynami MRFs: Furthermore, our method an also be used for boosting the effiieny of dynami MRFs (introdued to omuter vision in [5]. Two works have been roosed in this regard reently [5, 4]. These methods an be alied to dynami MRFs that are binary or have onvex riors. On the ontrary, Fast-PD naturally handles a muh wider lass of dynami MRFs, and an do so by also exloiting information from a roblem, whih is dual to the original MRF roblem. Fast-PD an thus be thought of as a generalization of revious tehniques. The rest of the aer is organized as follows. In se., we briefly review the work of [6] about using the rimaldual shema for MRF otimization. The Fast-PD algorithm is then desribed in se. 3. Its effiieny for otimizing Fast-PD requires only d(a, b, d(a, b= a=b

2 : [x,y] INIT DUALS PRIMALS( ; x old x : for eah label in L do 3: y PREEDIT DUALS(,x,y; 4: [x,y ] UPDATE DUALS PRIMALS(,x,y; 5: y POSTEDIT DUALS(,x,y ; 6: x x ; y y ; 7: end for 8: if x x old then 9: x old x; goto ; : end if Fig. : The rimal dual shema for MRF otimization. single MRFs is further analyzed in se. 4, where related results and some imortant extensions of Fast-PD are resented as well. Se. 5 exlains how Fast-PD an boost the erformane of dynami MRFs, and also ontains more exerimental results. Finally, we onlude in setion 6.. Primal-dual MRF otimization algorithms In this setion, we review very briefly the work of [6]. Consider the rimal-dual air of linear rograms, given by: PRIMAL: min T x DUAL: max b T y s.t. Ax = b,x s.t. A T y One seeks an otimal rimal solution, with the extra onstraint of x being integral. This makes for an NP-hard roblem, and so one an only hoe for finding an aroximate solution. To this end, the following shema an be used: Theorem (Primal-Dual shema. Kee generating airs of integral-rimal, dual solutions (x k,y k, until the elements of the last air, say x,y, are both feasible and have osts that are lose enough, e.g. their ratio is f a : T x f a b T y ( Then x is guaranteed to be an f a -aroximate solution to the otimal integral solution x, i.e. T x f a T x. The above shema has been used in [6], for deriving aroximation algorithms for a very wide lass of MRFs. To this end, MRF otimization was first ast as an equivalent integer rogram and then, as required by the rimal-dual shema, its linear rogramming relaxation and its dual were derived. Based on these LPs, the authors then show that, for Theorem to be true with f a = dmax d min, it suffies that the next (so-alled relaxed omlementary slakness onditions hold true for the resulting rimal and dual variables: = min a L (a, V (3 y q +y q (x q = w q d, x q, q E (4 y q (a+y q (b w q d max, q E, a L, b L (5 In these formulas, the rimal variables, denoted by x = {x } V, determine the labels assigned to nodes (alled ative labels hereafter, e.g. x is the ative label of node. Whereas, the dual variables are divided into balane and height variables. There exist balane variables y q (a, y q (a er edge (, q and label a, as well as height variable (a er node and label a. Variables y q (a, y q (a are also alled onjugate and, for the dual solution to be feasible, these must be set oosite to eah other, i.e.: y q ( y q (. Furthermore, the height variables are always defined in terms of the balane variables as follows: d max max a b d(a, b, d min min a b d(a, b ( ( + q:q E y q(. (6 Note that, due to (6, only the vetor y (of all balane variables is needed for seifying a dual solution. In addition, for simlifying onditions (4,(5, one an also define: load q (a, b y q (a+y q (b. (7 The rimal-dual variables are iteratively udated until all onditions (3-(5 hold true. The basi struture of a rimal-dual algorithm an be seen in Fig.. During an inner -iteration (lines 3-6 in Fig., a label is seleted and a new rimal-dual air of solutions (x,y is generated based on the urrent air (x,y. To this end, among all balane variables y q (., only the balane variables of -labels (i.e. y q ( are udated during a -iteration. L suh iterations (i.e. one -iteration er label in L make u an outer iteration (lines -7 in Fig., and the algorithm terminates if no hange of label takes lae at the urrent. During an inner iteration, the main udate of the rimal and dual variables takes lae inside UP- DATE DUALS PRIMALS, and (as shown in [6] this udate redues to solving a max-flow roblem in an aroriate grah G. Furthermore, the routines PREEDIT DUALS and POSTEDIT DUALS simly aly orretions to the dual variables before and after this main udate, i.e. to variables y and y resetively. Also, for simliity s sake, note that we will hereafter refer to only one of the methods derived in [6], and this will be the so-alled PD3 a method. 3. Fast rimal-dual MRF otimization The omlexity of the PD3 a rimal-dual method largely deends on the omlexity of all max-flow instanes (one instane er, whih, in turn, deends on the number of er max-flow. So, for designing faster rimal-dual algorithms, we first need to understand how the grah G, assoiated with the max-flow roblem at a -iteration of PD3 a, is onstruted. To this end, we also have to reall the following intuitive interretation of the dual variables [6]: for eah node, a searate oy of all labels in L is onsidered, and all these labels are reresented as balls, whih float at ertain heights relative to a referene lane. The role of the height variables ( is then to determine the balls height (see Figure (a. E.g. the height of label a at node is given by (a. Also, exressions like label a at is below/above label b imly (a (b. Furthermore, balls are not stati, but may move in airs through udating airs of onjugate balane variables. E.g., in Figure (a, label at is raised by +δ (due to adding +δ to y q (, and so label at q has to move down by δ (due to adding δ to y q ( so that ondition y q ( = y q ( still holds. Therefore, the role of balane variables is to raise or lower labels. In artiular, the value of balane variable y q (a reresents the artial raise of label a at due to edge q, while (by (6 the total raise of a at equals the sum of artial raises from all edges of G inident to.

3 w q (a (b ( q h q ( ( x=a x +δ -δ f as ( h q (x q s x ( q =a x f t a t a s a t s t f s f t Fig. : (a Dual variables visualization for a simle MRF with nodes {, q} and labels {a, }. A oy of labels {a, } exists for every node, and all these labels are reresented by balls floating at ertain heights. The role of the height variables ( is to seify exatly these heights. Furthermore, balls are not stati, but may move (i.e. hange their heights in airs by udating onjugate balane variables. E.g., here, ball at is ulled u by +δ (due to inreasing y q( by +δ and so ball at q moves down by δ (due to dereasing y q( by δ. Ative labels are drawn with a thiker irle. (b If label at is below x, then (due to (3 we want label to raise and reah x. We thus onnet node to the soure s with an edge s (i.e. is an s-linked node, and flow f s reresents the total raise of (we also set a s = (. ( If label at is above x, then (due to (3 we want label not to go below x. We thus onnet node to the sink t with edge t (i.e. is a t-linked node, and flow f t reresents the total derease in the height of (we also set a t =( so that will still remain above x. Hene, PD3 a tries to iteratively move labels u or down, until all onditions (3-(5 hold true. To this end, it uses the following strategy: it ensures that onditions (4-(5 hold at eah iteration (whih is always easy to do and is just left with the main task of making the labels heights satisfy ondition (3 as well in the end (whih is the most diffiult art, requiring eah ative label x to be the lowest label for. For this urose, labels are moved in grous. In artiular, during a -iteration, only the -labels are allowed to move. Furthermore, it was shown in [6] that the movement of all -labels (i.e. the udate of dual variables y q ( and ( for all, q an be simulated by ushing the maximum flow through a direted grah G (whih is onstruted based on the urrent rimal-dual air (x, y at a -iteration. The nodes of G onsist of all nodes of grah G (the internal nodes, lus external nodes, the soure s and the sink t. In addition, all nodes of G are onneted by two tyes of edges: interior and exterior edges. Interior edges ome in airs q, q (with one suair for every neighbors, q in G, and are resonsible for udating the balane variables. In artiular, the flows f q /f q of these edges reresent the inrease/derease of balane variable y q (, i.e. y q (= y q ( +f q f q. Also, as we shall see, the aaities of interior edges are used together with PREEDIT DUALS, POSTEDIT DUALS to imose onditions (4, (5. But for now, in order to understand how to make a faster rimal-dual method, it is the exterior edges (whih are in harge of the udate of height variables, as well as their aaities (whih are used for imosing the remaining ondition (3, that are of interest to us. The reason is that these edges determine the number of s-linked nodes, whih, in turn, affets the number of augmenting aths er max-flow. In artiular, eah internal node onnets to either the soure s (i.e. it is an s-linked node or to the sink t (i.e. it is a t-linked node through one of these exterior edges, and this is done (with the goal of ensuring (3 as follows: if label at is above x during a -iteration (i.e. ( >, then label should not go below x, or else (3 will be violated for. Node thus onnets to t through direted edge t (i.e. beomes t-linked, and flow f t reresents the total derease in the height of after UPDATE DUALS PRIMALS, i.e. h (= ( f t (see Fig. (. Furthermore, the aaity of t is set so that label will still remain above x, i.e. a t = (. On the other hand, if label at is below ative label x (i.e. ( <, then (due to (3 label should raise so as to reah x, and so onnets to s through edge s (i.e. beomes s-linked, while flow f s reresents the total raise of ball, i.e. h ( = (+f s (see Fig. (b. In this ase, we also set a s = (. This way, by ushing flow through the exterior edges of G, all -labels that are stritly below an ative label try to raise and reah that label during UPDATE DU- ALS PRIMALS 3. Not only that, but the are the -labels below an ative label (i.e. the are the s-linked nodes, the will be the edges onneted to the soure, and thus the less will be the number of ossible augmenting aths. In fat, the algorithm terminates when, for any label, there are no more -labels stritly below an ative label (i.e. no s-linked nodes exist and thus no augmenting aths may be found, in whih ase ondition (3 will finally hold true, as desired. Put another way, UPDATE DUALS PRIMALS tries to ush -labels (whih are at a low height u, so that the number of s-linked nodes is redued and thus augmenting aths may be ossible for the next iteration. However, although UPDATE DUALS PRIMALS tries to redue the number of s-linked nodes (by ushing the maximum amount of flow, PREEDIT DUALS or POSTEDIT DU- ALS very often soil that rogress. As we shall see later, this ours beause, in order to restore ondition (4 (whih is their main goal, these routines are fored to aly orretions to the dual variables (i.e. to the labels height. This is abstratly illustrated in Figure 3, where, as a result of ushing flow, a -label initially managed to reah an ative label x, but it again droed below x, due to some orretion alied by these routines. In fat, as one an show, the only oint where a new s-linked node an be reated is during either PREEDIT DUALS or POSTEDIT DUALS. 3 Equivalently, if -label at annot raise high enough to reah x, UPDATE DUALS PRIMALS then assigns that -label as the new ative label of (i.e. x =, thus effetively making the ative label go down. This hels ondition (3 to beome true, and forms the main rationale behind the udate of the rimal variables x in UPDATE DUALS PRIMALS.

4 ( x a s x f s ( ( x orretion a s (a before max-flow (b after max-flow ( after orretion by PREEDIT_DUALS or POSTEDIT_DUALS Fig. 3: (a Label at is below x, and thus label is allowed to raise itself in order to reah x. This means that will be an s-linked node of grah G, i.e. a s >, and thus a non-zero flow f s (reresenting the total raise of label may ass through edge s. Therefore, in this ase, edge s may beome art of an augmenting ath during max-flow. (b After UPDATE DUALS PRIMALS, label has managed to raise by f s and reah x. Sine it annot go higher than that, no flow an ass through edge s, i.e. a s =, and so no augmenting ath may traverse that edge thereafter. ( However, due to some orretion alied to -label s height, label has droed below x one more and has beome an s-linked node again (i.e. a s >. Edge s an thus be art of an augmenting ath again (as in (a. To fix this roblem, we will redefine PREEDIT DUALS, POSTEDIT DUALS so that they an now ensure ondition (4 by using just a minimum amount of orretions for the dual variables, (e.g. by touhing these variables only rarely. To this end, however, UPDATE DUALS PRIMALS needs to be modified as well. The resulting algorithm, alled Fast-PD, arries the following main differenes over PD3 a during a -iteration (its seudoode aears in Fig. 4: - the new PREEDIT DUALS modifies a air y q (, y q ( of dual variables only when absolutely neessary. So, whereas the revious version modified these variables (thereby hanging the height of a -label whenever x, x q (whih ould haen extremely often, a modifiation is now alied only if load q (, x q > w q d(, x q or load q, > w q d, (whih, in ratie, haens muh more rarely. In this ase, a modifiation is needed (see ode in Fig. 4, beause the above inequalities indiate that ondition (4 will be violated if either (, x q or, beome the new ative labels for, q. On the ontrary, no modifiation is needed if the following inequalities are true: load q (, x q < w q d(, x q, load q, < w q d,, beause then, as we shall see below, the new UP- DATE DUALS PRIMALS an always restore (4 (i.e. even if (, x q or, are the next ative labels - e.g., see (. In fat, the modifiation to y q ( that is oasionally alied by the new PREEDIT DUALS an be shown to be the minimal orretion that restores exatly the above inequalities (assuming, of ourse, this restoration is ossible. - Similarly, the new POSTEDIT DUALS modifies 4 balane variables y q(x (with x = and y q(x q (with x q = only if the inequality load q (x, x q >w qd(x, x q holds, in whih ase POSTEDIT DUALS simly has to 4 We reall that POSTEDIT DUALS may modify only dual solution y. For that solution, we define load q (a, b y q (a+y q (b, as in (7. [x, y] INIT DUALS PRIMALS( : x random labels; y ; q,adjust y q or y q(x q so that load q, x q=w qd, x q y PREEDIT DUALS(, x, y: q,if load q(, x q>w qd(, x q or load q, >w qd, adjust y q( so that load q(, x q=w qd(, x q [x,y ] UPDATE DUALS PRIMALS(, x,y: x x; y y; Construt G andalymax-flowtoomuteallflows f s/f t, f q q, y q ( yq(+fq fq, if an unsaturated ath from s to exists, then x y POSTEDIT DUALS(, x,y : {Wedenote load q (, =y q ( +y q ( } q,if load q (x, x q >wqd(x, x q {Thisimlies x =or x q =} adjust y q (sothat load q (x, x q =wqd(x, x q Fig. 4: Fast-PD s seudoode. redue load q(x, x q for restoring (4. However, this inequality will hold true very rarely (e.g. for a metri d(,, one may show that it an never hold, and so POSTEDIT DU- ALS will modify a -balane variable (thereby hanging the height of a -label only in very seldom oasions. - But, to allow for the above hanges, we also need to modify the onstrution of grah G in UPDATE DU- ALS PRIMALS. In artiular, for x and x q, the aaities of interior edges q, q must now be set as follows: 5 a q = [ w q d(, x q load q (, x q ] +, (8 a q = [ w q d, load q, ] +, (9 where [x] + max(x,. Besides ensuring (5 (by not letting the balane variables inrease too muh, the main rationale behind the above definition of interior aaities is to also ensure that (after max-flow ondition (4 will be met by most airs (, q, even if (, x q or, are the next labels assigned to them (whih is a good thing, sine we will thus manage to avoid the need for a orretion by POSTEDIT DUALS for all but a few, q. For seeing this, the ruial thing to observe is that if, say, (, x q are the next labels for and q, then aaity a q an be shown to reresent the inrease of load q (, x q after max-flow, i.e.: load q(, x q = load q (, x q + a q. ( Hene, if the following inequality is true as well: load q (, x q w q d(, x q, ( then ondition (4 will do remain valid after max-flow, as the following trivial derivation shows: load q (, x q (,(8 = load q (, x q +[w q d(, x q load q (, x q ] + ( = w q d(, x q ( But this means that a orretion may need to be alied by POSTEDIT DUALS only for airs, q violating ( (before max-flow. However, suairs tend to be very rare in ratie (e.g., as one an rove, no suairs exist when d(, is a metri, and thus very few orretions need to take lae. Fig. 5 summarizes how Fast-PD sets the aaities for all edges of G. As already exlained, the interior aaities (with the hel of PREEDIT DUALS, POSTEDIT DUALS 5 If =x or =x q, then a q =a q = as before, i.e. as in PD3 a.

5 in a few ases allow UPDATE DUALS PRIMALS to imose onditions (4,(5, while the exterior aaities allow UP- DATE DUALS PRIMALS to imose ondition (3. As a result, the next theorem holds (see [] for a omlete roof: Theorem. The last rimal-dual air (x, y of Fast-PD satisfies (3-(5, and so x is an f a -aroximate solution. In fat, Fast-PD maintains all good otimality roerties of the PD3 a method. E.g., for a metri d(,, Fast-PD roves to be as owerful as α-exansion (see []: Theorem 3. If d(, is a metri, then the Fast-PD algorithm omutes the best -exansion after any -iteration. 4. Effiieny of Fast-PD for single MRFs But, besides having all these good otimality roerties, a very imortant advantage of Fast-PD over all revious rimal-dual methods, as well as α-exansion, is that it roves to be muh more effiient in ratie. In fat, the omutational effiieny for all methods of this kind is largely determined from the time taken by eah max-flow roblem, whih, in turn, deends on the number of augmenting aths that need to be omuted. For the ase of Fast-PD, the number of er dereases dramatially, as the algorithm rogresses. E.g. Fast-PD has been alied to the roblem of image restoration, and fig. 7 ontains a related result about the denoising of a orruted (with gaussian noise enguin image (56 labels and a trunated quadrati distane d(a, b = min( a b, D - where D = - has been used in this ase. Also, fig. 8(a shows the orresonding number of augmenting aths er outer-iteration (i.e. er grou of L s. Notie that, for both α-exansion, as well as PD3 a, this number remains very high (i.e. almost over 6 aths throughout all iterations. On the ontrary, for the ase of Fast-PD, it dros towards zero very quikly, e.g. only 495 and 7 aths had to be found during the 8 th and last outer-iteration resetively (obviously, as also shown in Fig. 9(a, this diretly affets the total time needed er outer-iteration. In fat, for the ase of Fast-PD, it is very tyial that, after very few s, no more than or augmenting aths need to be omuted er max-flow, whih really boosts the erformane in this ase. This roerty an be exlained by the fat that Fast-PD maintains both a rimal, as well as a dual solution throughout its exeution. Fast-PD then manages to effetively use the dual solutions of revious inner iterations, so as to redue the number of augmenting aths for the next inneriterations. Intuitively, what haens is that Fast-PD ultimately wants to lose the ga between the rimal and the exterior aaities interior aaities a x a q =[w q d(,x q -load q (,x q ] + x = s =[ - (] + a q = a x q a q =[w q d,-load q,] + t =[ (- ] + x q = a q = Fig. 5: Caaities of grah G, as set by Fast-PD. dual dual osts rimal osts ga k dual k- dual k rimal k rimal rimal k- (a High-level view of the Fast-PD algorithm fixed dual ost ga k rimal osts dual rimal k rimal k- rimal (b High-level view of the α-exansion algorithm Fig. 6: (a Fast-PD generates airs of rimal-dual solutions iteratively, with the goal of always reduing the rimal-dual ga (i.e. the ga between the resulting rimal and dual osts. But, for the ase of Fast-PD, this ga an be viewed as a rough estimate for the number of, and so this number is fored to redue over time as well. (b On the ontrary, α-exansion works only in the rimal domain (i.e. it is as if a fixed dual ost is used at the start of eah new iteration and thus the rimal-dual ga an never beome small enough. Therefore, no signifiant redution in the number of takes lae as the algorithm rogresses. dual ost (see Theorem, and, for this, it iteratively generates rimal-dual airs, with the goal of dereasing the size of this ga (see Fig. 6(a. But, for Fast-PD, the ga s size an be thought of as, roughly seaking, an uer-bound for the number of augmenting aths er. Sine, furthermore, Fast-PD manages to redue this ga at any time throughout its exeution, the number of augmenting aths is fored to derease over time as well. On the ontrary, a method like α-exansion, that works only in the rimal domain, ignores dual solutions omletely. It is, roughly seaking, as if α-exansion is resetting the dual solution to zero at the start of eah, thus effetively forgetting that solution thereafter (see Fig. 6(b. For this reason, it fails to redue the rimal-dual ga and thus also fails to ahieve a redution in ath over time, i.e. aross inneriterations. But the PD3 a algorithm as well fails to mimi Fast-PD s behavior (desite being a rimal-dual method. As exlained in se. 3, this haens beause, in this ase, PREEDIT DUAL and POSTEDIT DUAL temorarily destroy the ga just before the start of UPDATE DUALS PRIMALS, i.e. just before max-flow is about to begin omuting the augmenting aths. (Note, of ourse, that this destrution is only temorary, and the ga is restored again after the exeution of UPDATE DUALS PRIMALS. The above mentioned relationshi between rimal-dual ga and number of augmenting aths is formally desribed in the next theorem (see [] for a omlete roof: Theorem 4. For Fast-PD, the rimal-dual ga at the urrent forms an aroximate uer bound for the number of augmenting aths at eah iteration thereafter. Sketh of roof. During a -iteration, it an be shown that dual-ost min((,, whereas rimal-ost=, and so the rimal-dual ga uer-bounds the following quantity: [ (] + = a s.

6 Fig. 7: Left: Tsukuba image and its disarity by Fast-PD. Middle: a SRI tree image and orresonding disarity by Fast-PD. Right: noisy enguin image and its restoration by Fast-PD. But this quantity obviously forms an uer-bound on the maximum flow, whih, in turn, uer-bounds the number of (assuming integral flows. Due to the above mentioned roerty, the time er outer-iteration dereases dramatially over time. This has been verified exerimentally with virtually all roblems that Fast-PD has been tested on. E.g. Fast-PD has been also alied to the roblem of stereo mathing, and fig. 7 ontains the resulting disarity (of size with 6 labels for the well-known Tsukuba stereo air, as well as the resulting disarity (of size with labels for an image air from the well-known SRI tree sequene (in both ases, a trunated linear distane d(a, b=min( a b, D - with D= and D=5 - has been used, while the weights w q were allowed to vary based on the image gradient at. Figures 9(b, 9( ontain the orresonding running times er. Notie how muh faster the outer-iterations of Fast-PD beome as the algorithm rogresses, e.g. the last outer-iteration of Fast-PD (for the SRI-tree examle lasted less than mse (sine, as it turns out, only 4 augmenting aths had to be found during that iteration. Contrast this with the behavior of either the α-exansion or the PD3 a algorithm, whih both require an almost onstant amount of time er outer-iteration, e.g. the last outer-iteration of α-exansion needed more than.4 ses to finish (i.e. it was more than 4 times slower than Fast-PD s iteration!. Similarly, for the Tsukuba examle, α-exansion s last outer-iteration was more than times slower than Fast-PD s iteration. Max-flow algorithm adatation: However, for fully exloiting the dereasing number of ath and redue the running time, we had to roerly adat the max-flow algorithm. To this end, the ruial thing to observe was that the dereasing number of was diretly related to the dereasing number of s-linked nodes, as already exlained in se. 3. E.g. fig. 8(b shows how the number of s-linked nodes varies er outer-iteration for the enguin examle (with a similar behavior being observed for the other examles as well. As an be seen, this number dereases drastially over time. In fat, as No. of.5.5 x 6 PD3 a α exansion x 6 No. of s linked nodes ( (a (b Fig. 8: (a Number of augmenting aths er for the enguin examle (similar results hold for the other examles as well. Only in the ase of Fast-PD, this number dereases dramatially over time. (b This roerty of Fast-PD is diretly related to the dereasing number of s-linked nodes er outer-iteration (this number is shown here for the same examle as in (a. time (ses time (ses PD3 a α exansion (a enguin PD3 a α exansion ( SRI tree time (ses PD3 a α exansion (b Tsukuba total time (ses enguin tsukuba SRI tree F a st-pd a -ex a nsi o n PD 3 a (d Total times Fig. 9: Total time er for the (a enguin, (b Tsukuba and ( SRI tree examles. (d Total running times. For all exeriments of this aer, a.6ghz lato has been used. imlied by ondition (3, no s-linked nodes will finally exist uon the algorithm s termination. Any augmentation-based max-flow algorithm striving for omutational effiieny, should ertainly exloit this roerty when trying to extrat its augmenting aths. The most effiient of these algorithms [] maintains searh trees for the fast extration of these aths, a soure and a sink tree. Here, the soure tree will start growing by exloring non-saturated edges that are adjaent to s-linked nodes, whereas the sink tree will grow starting from all t-linked nodes. Of ourse, the algorithm terminates when no adjaent unsaturated edges an be found any more. However, in our ase, maintaining the sink tree is omletely ineffiient and does not exloit the muh smaller number of s-linked nodes. We thus roose maintaining only the soure tree during max-flow, whih will be a muh heaer thing to do here (e.g., in many inner iterations, there an be than s-linked nodes, but many thousands of t-linked nodes. Moreover, due to the small size of the soure tree, deteting the termination of the max-flow roedure an now be done a lot faster, i.e. with-

7 3 subotimality bound (Tsukuba subotimality bound (enguin inner iteration inner iteration Fig. : Subotimality bounds er inner iteration (for Tsukuba and enguin. These bounds dro to very fast, meaning that the orresonding solutions have beome almost otimal very early. out having to fully exand the large sink tree (whih is a very ostly oeration, thus giving a substantial seedu. In addition to that, for effiiently building the soure tree, we kee trak of all s-linked nodes and don t reomute them from srath eah time. In our ase, this traking an be done without ost, sine, as exlained in se. 3, an s-linked node an be reated only inside the PREEDIT DUALS or the POSTEDIT DUALS routine, and thus an be easily deteted. The above simle strategy has been extremely effetive for boosting the erformane of max-flow, eseially when a small number of were needed. Inremental grah onstrution: But besides the maxflow algorithm adatation, we may also modify the way grah G is onstruted. I.e. instead of onstruting the aaitated grah G from srath eah time, we also roose an inremental way of setting its aaities. The following lemma turns out to be ruial in this regard: Lemma. Let G, Ḡ be the grahs for the urrent and revious -iteration. Let also, q be neighboring MRF nodes. If, during the interval from the revious to the urrent -iteration, no hange of label took lae for and q, then the aaities of the interior edges q, q in G and of the exterior edges s, t, sq, qt in G equal the residual aaities of the orresonding edges in Ḡ. The roof follows diretly from the fat that if no hange of label took lae for, q, then none of the height variables, h q (x q or the balane variables y q, y q (x q ould have hanged. Due to lemma, for building grah G, we an simly reuse the residual grah of Ḡ and only reomute those aaities of G for whih the above lemma does not hold, thus seeding-u the algorithm even further. Combining seed with otimality: Fig. 9(d ontains the running times of Fast-PD for various MRF roblems. As an be seen from that figure, Fast-PD roves to be muh faster than either the α-exansion 6 or the PD3 a method, e.g. Fast-PD has been more than 9 times faster than α-exansion for the ase of the enguin image (7.44 ses vs 73. ses. In fat, this behavior is a tyial one, sine Fast-PD has onsistently rovided at least a 3-9 times seedu for all the roblems it has been tested on. However, besides its effiieny, Fast-PD does not make any omromise regarding the otimality of its solutions. On one hand, this is ensured by theorems, 3. On the other hand, Fast-PD, like 6 Sine α-exansion annot be used if d(, is not a metri, the method roosed in [7] had to be used for the ases of a non-metri d(,. [x, y] INIT DUALS PRIMALS( x, ȳ: x x; y ȳ; q, y q +=w qd, x q w q d(x, x q;, ( += ( ( ; Fig. : Fast-PD s new seudoode for dynami MRFs. any other rimal-dual method, an also tell for free how well it erformed by always roviding a er-instane subotimality bound for its solution. This omes at no extra ost, sine any ratio between the ost of a rimal solution and the ost of a dual solution an form suh a bound. E.g. fig. shows how these ratios vary er for the tsukuba and enguin roblems (with similar results holding for the other roblems as well. As one an notie, these ratios dro to very quikly, meaning that an almost otimal solution has already been estimated even after just a few iterations (and desite the roblem being NP-hard. 5. Dynami MRFs But, besides single MRFs, Fast-PD an be easily adated to also boost the effiieny for dynami MRFs [5], i.e. MRFs varying over time, thus showing the generality and ower of the roosed method. In fat, Fast-PD fits erfetly to this task. The imliit assumtion here is that the hange between suessive MRFs is small, and so, by initializing the urrent MRF with the final (rimal solution of the revious MRF, one exets to seed u inferene. A signifiant advantage of Fast-PD in this regard, however, is that it an exloit not only revious MRF s rimal solution (say x, but also its dual solution (say ȳ. And this, for initializing urrent MRF s botrimal and dual solutions (sayx,y. Obviously, for initializing x, one an simly set x= x. Regarding the initialization of y, however, things are slightly more omliated. For maintaining Fast-PD s otimality roerties, it turns out that, after setting y = ȳ, a slight orretion still needs to be alied to y. In artiular, Fast-PD requires its initial solution y to satisfy ondition (4, i.e. y q + y q (x q = w q d, x q, whereas ȳ satisfies ȳ q + ȳ q (x q = w q d(x, x q, i.e. ondition (4 with w q d(, relaed by the airwise otential w q d(, of the revious MRF. The solution for fixing that is very simle: e.g. we an simly set y q += w q d, x q w q d(x, x q. Finally, for taking into aount the ossibly different singleton otentials between suessive MRFs, the new heights will obviously need to be udated as ( += ( (, where ( are the singleton otentials of the revious MRF. These are the only hanges needed for the ase of dynami MRFs, and thus the new seudoode aears in Fig.. As exeted, for dynami MRFs, the seedu rovided by Fast-PD is even greater than single MRFs. E.g. Fig. (a shows the running times er frame for the SRI tree image sequene. Fast-PD roves to be be more than times faster than α-exansion in this ase (requiring on average. ses er frame, whereas α-exansion required.8 ses on average. Fast-PD an thus run on about 5

8 time (ses No. of x 5 α exansion frame (a Running times er frame for the SRI tree sequene α exansion frame (b Augmenting aths er frame for the SRI tree sequene Fig. : Statistis for the SRI tree sequene. frames/se, i.e. it an do stereo mathing almost in real time for this examle (in fat, if suessive MRFs bear greater similarity, even muh bigger seedus an be ahieved. Furthermore, fig. (b shows the orresonding number of augmenting aths er frame for the SRI tree image sequene (for both α-exansion and Fast-PD. As an be seen from that figure, a substantial redution in the number of augmenting aths is ahieved by Fast-PD, whih hels that algorithm to redue its running time. This same behavior has been observed in all other dynami roblems that Fast-PD has been tested on as well. Intuitively, what haens is illustrated in Fig. 3(a. Fast-PD has already managed to lose the ga between the final rimal-dual osts rimal x, dualȳ of the revious MRF. However, due to the ossibly different singleton (i.e. ( or airwise (i.e. w q d(, otentials of the urrent MRF, these osts need to be erturbed to generate the new initial osts rimal x, dual y. Nevertheless, as only slight erturbations take lae, the new rimal-dual ga (i.e. between rimal x, dual y will still be lose to the revious ga (i.e. between rimal x, dualȳ. As a result, the new ga will remain small. Few augmenting aths will therefore have to be found for the urrent MRF, and thus the algorithm s erformane is boosted. Put otherwise, for the ase of dynami MRFs, Fast-PD manages to boost erformane, i.e. redue number of augmenting aths, aross two different axes. The first axis lies along the different s of the same MRF (e.g. see red arrows in Fig. 3(b, whereas the seond axis extends aross time, i.e. aross different MRFs (e.g. see blue arrow in Fig. 3(b, onneting the last iteration of MRF t to the first iteration of MRF t. dual y dual y ga ga MRF t- rimal x rimal x MRF t (a (b Fig. 3: (a The final osts rimal x, dualȳ of the revious MRF are slightly erturbed to give the initial osts rimal x, dual y of the urrent MRF. Therefore, the initial rimal-dual ga of the urrent MRF will be lose to the final rimal-dual ga of the revious MRF. Sine the latter is small, so will be the former, and thus few augmenting aths will need to be omuted for the urrent MRF. (b Fast-PD redues the number of augmenting aths in ways: internally, i.e. aross iterations of the same MRF (see red arrows, as well as externally, i.e. aross different MRFs (see blue arrow. 6. Conlusions In onlusion, a new grah-ut based method for MRF otimization has been roosed. It generalizes α-exansion, while it also manages to be substantially faster than this state-of-the-art tehnique. Hene, regarding otimization of stati MRFs, this method rovides a signifiant seedu. In addition to that, however, it an also be used for boosting the erformane of dynami MRFs. In both ases, its effiieny omes from the fat that it exloits information not only from the rimal roblem (i.e. the MRF otimization roblem, but also from a dual roblem. Moreover, desite its seed, the roosed method an nevertheless guarantee almost otimal solutions for a very wide lass of NP-hard MRFs. Due to all of the above, and given the ubiquity of MRFs, we strongly believe that Fast-PD an rove to be an extremely useful tool for many roblems in omuter vision in the years to ome. Referenes [] N. Komodakis, G. Tziritas and N. Paragios. Fast Primal-Dual Strategies for MRF Otimization. Tehnial reort, 6. 5 [] Y. Boykov and V. Kolmogorov. An exerimental omarison of min-ut/max-flow algorithms for energy minimization in vision. PAMI, 6(9, 4. 6 [3] Y. Boykov, O. Veksler, and R. Zabih. Fast aroximate energy minimization via grah uts. PAMI, 3(,. [4] O. Juan and Y. Boykov. Ative grah uts. In CVPR, 6. [5] P. Kohli and P. H. Torr. Effiiently solving dynami markov random fields using grah uts. In ICCV, 5., 7 [6] N. Komodakis and G. Tziritas. A new framework for aroximate labeling via grah-uts. In ICCV, 5.,, 3 [7] C. Rother, S. Kumar, V. Kolmogorov, and A. Blake. Digital taestry. In CVPR, 5. 7 [8] R. Szeliski, et al. A omarative study of energy minimization methods for markov random fields. In ECCV, 6.