Key words. closure problem, nonlinear costs, Bayesian estimation, maximum flow, parametric minimum cut, convex optimization

Transcription

1 MINIMIZING A CONVEX COST CLOSURE SET DORIT S. HOCHBAUM AND MAURICE QUEYRANNE Abstract. Many applications in the area of production and statistical estiation are probles of convex optiization subject to ranking constraints that represent a given partial order. This proble which we call the convex cost closure proble, or (CCC) is a generalization of the known axiu (or iniu) closure proble and the isotonic regression proble. For a (CCC) proble on n variables and constraints we describe an algorith that has the coplexity of the iniu cut proble plus the coplexity of finding the inia of up to n convex functions. Since the convex cost closure proble is a generalization of both iniu cut and of iniization of n convex functions this coplexity is fastest possible. For the quadratic proble the coplexity of our algorith is strongly polynoial, O(n log n2 ). For the isotonic regression proble the coplexity is O(n log U), for U the largest range for a variable value. Key words. closure proble, nonlinear costs, Bayesian estiation, axiu flow, paraetric iniu cut, convex optiization AMS subject classifications. 15A15, 15A09, 15A23 1. Introduction. A coon proble in statistical estiation is that observations do not satisfy preset ranking order requireents. In that case the proble is to find an adjustent of the observations that does fit the ranking order constraints and iniizes the total deviation penalty. The deviation penalty is a convex function of the fitted values. Forally, we define the proble for a directed graph G = (V, A) and a convex function f j () associated with each node j V. The forulation of the convex cost closure proble is then, (CCC) Min j V f j(x j ) subject to x i x j 0 (i, j) A l j x j u j integer j V. This proble generalizes the isotonic regression proble in which the graph is a partial order graph for linear order the arcs of A are of the for (i, i + 1). Another well known proble that (CCC) generalizes is the iniu closure proble. That proble is the binary case of (CCC): (Miniu Min j V w j x j Closure) subject to x i x j 0 (i, j) A 0 x j 1 integer j V. Picard established in 1976 [17] that the closure proble is equivalent to a iniu cut proble on a graph associated with G to which we add a source and a sink. This construction is described in Section 3. Solving the (CCC) proble is thus at least as hard as solving the iniu cut proble on the associated graph. When the graph is epty the (CCC) proble reduces to the integer iniization of n convex functions, each in a given interval. Thus (CCC) generalizes the proble of convex functions integer iniization in bounded intervals. Supported by NSF grants Nos. DMI and DMI Departent of Industrial Engineering and Operations Research and Walter A. Haas School of Business, University of California, Berkeley California (hochbau@ieor.berkeley.edu). University of British Colubia Vancouver, British Colubia V6T1W5, Canada. 1

2 2 D. S. HOCHBAUM AND M. QUEYRANNE The challenge of convex optiization proble is that searching for a iniu of a convex function involves an unavoidable factor such as log U in the running tie, for U the length of the interval containing the optial value of the variable. Although one can replace other paraeters that depend on the variability of the functions, the running tie cannot be ade strongly polynoial using the arithetic coplexity odel (see [11] for details on this result). The algorith presented here differs fro previous algoriths in that the search for the inia of the convex functions is disjoint fro the rest of the algorith and conducted as a post-processing step. The ain body of the algorith identifies disjoint intervals that are guaranteed to contain the optial values of each variable and satisfy the partial order constraints. The run tie of our algorith, O(n log n2 + n log U) for U = ax i{u i l i }, is the fastest known for the proble, and it either atches or iproves the coplexity of algoriths devised for special cases of (CCC). In dealing with nonlinear functions it is necessary to specify the coplexity odel used. We assue the unit cost odel and no restriction on the structure of the convex functions: We assue the existence of an oracle returning function values for every polynoial length arguent input in O(1). Since we search for an optial solution aong the integers we will only be interested in integer arguents. Any arithetic operation or coparison involving function values is executed in unit tie in this odel. Derivatives, or rather subgradients, are required as finite differences, f j (x) = f j(x + 1) f j (x). In the next section we overview the literature and applications of the proble. In Section 3 the link of the closure proble to the axiu flow proble is reviewed. Section 4 discusses a linear tie algorith that is eployed to verify whether an instance of (CCC) is feasible. Section 5 provides the ain theoretical underpinning of the algorith the so-called threshold theore. Section 6 describes the entire algorith and its correctness and coplexity. Section 7 details the ipleentations in strongly polynoial tie for the quadratic case, and the O(n log U) for the isotonic regression proble. In Section 8 we provide an algorith for the continuous version of (CCC). In Section 9 we conclude with several rearks and extensions. 2. Related applications and literature. We sketch first several classes of applications for the convex cost closure proble. In the proble of selection of discrete contingent projects a nuber of projects j N can be undertaken, but only at discrete levels l j, l j + 1,..., u j. These projects are contingent in that, for specified pairs of projects (i, j) A, each unit of project i requires one unit of project j: x i x j. Different projects i, k,..., however, can use the sae units of project j (otherwise, we ight consider j as part of project i). For instance, projects i and k ay use j at different ties. The objective is to axiize the total net profit associated with the selected project levels. Here, f j (x j ) denotes the net profit associated with level x j of project j. The convexity of f j thus reflects decreasing returns of scale for project j. Additional inforation about this proble is provided by Picard and Queyranne [18]. Maxwell and Muckstadt, [16], consider nested power-of-two policies in a ultistage production/inventory proble. In this continuous-tie deterinistic odel deand for end-products arises at a constant rate. Interediate products are consued in the production of other products, as reflected by the directed graph (V, A). Given are positive inventory-related holding costs g j and production setup costs K j. The proble is to find production intervals T j = T 0 2 k j, with k j integer, that are nested. That is T i T j for (i, j) A. The objective is to iniize the average total

3 convex optiization over order constraints 3 cost per unit tie, c(t ) = j g j T j + K j /T j. Roundy [20] extends the Maxwell-Muckstadt odel by considering joint setup costs and relaxing the nestedness condition. He shows that the total cost is now c(t ) = R G R ax{t j : j R} + F K F /(in{t j : j F }), where R and F are suitably defined subsets of products, and G R and K F are corresponding (nonnegative) costs. Although the constraints T i T j thus disappear, the odelling capabilities of variable upper bound constraints are reflected in the handling of joint setup costs and holding costs. For that, Roundy extends the product set N by adding all the R and F sets, and defines corresponding variables T R (T F, respectively) by the inequalities T j T R (T F T j, resp.) for all j R (F, resp.). The resulting proble is thus cast again into the Maxwell-Muckstadt for above. Roundy s ajor result is that optial power-of-two policies thus constructed are 94%-effective, that is: the cost of an optial policy cannot be less than 94% that of an optiu power-of-two policy. He also shows that searching for an optial base interval T 0 yields a 98%-effective solution. The present paper extends this approach to general convex average cost functions f j (k j ) = c j (T 0 2 k j ). The 94% and 98% effectiveness results, however, hold only for the specific functions c above. Sokkalinga, Ahuja and Orlin, [23], discuss the inverse spanning tree proble. In this proble there is a spanning tree T given in an edge weighted graph. The proble is to odify the edge weights so that the given tree is a iniu spanning tree and so that the cost of the deviation is iniu. In order for a tree to be a iniu spanning tree each out of tree edge j ust have a weight w j greater or equal to each of the edges in the tree on the unique path between its endpoints. That is, the constraints that enforce that T is a iniu spanning tree are of the for w j w i for j E \ T, i T. The corresponding graph is a bipartite graph a structure that can be used to reduce the coplexity of our algorith for the resulting (CCC). Sokkalinga, Ahuja and Orlin devise algoriths for three specific convex deviation functions: Su of absolute differences, weighted su of absolute differences and axiu absolute differences. All these functions are convex for which the inia can be found in a single step. Our algorith s run tie is thus strongly polynoial with run tie better than O(n log 2 n) for this type of functions, and with additional additive factor of n log C for general convex functions where C is the axiu edge weight. The coplexity reported in [23] for the weighted absolute deviation proble is weakly polynoial O(n 2 log(nc)). Our algorith can further be adapted to provide substantial iproveents for special cases as reported in [14]. Statistical probles of partially ordered estiation have been discussed extensively in the literature, see e.g. Veinott [24] and Barlow et al., [4]. Let p 1,..., p n denote paraeters to be jointly estiated and let f j (x j ) denote the loss associated with estiating that p j = x j for j = 1,..., n. A typical instance is when f j (x j ) is the negative of the logarith of the likelihood, given p j = x j, that related rando variables assue observed values. The odel being estiated ay specify a partial order on the paraeters, as reflected by constraints x j x i for a set A of pairs (i, j), as well as siple upper and lower bounds on the paraeter values. If, in addition, the odel requires the paraeter values to be integer, then the proble of jointly

4 4 D. S. HOCHBAUM AND M. QUEYRANNE estiating the paraeter values so as to iniize total loss is precisely an instance of proble (CCC). If there is no such integrality restriction, then the proble is an instance of the continuous relaxation of (CCC) which is discussed in Section 8. Algoriths for the convex cost closure proble have been previously devised. Picard and Queyranne [19] proposed an algorith solving the proble with running tie of O(n(n log n2 +n log U)). Ahuja, Hochbau and Orlin [3], addressed a generalization of (CCC) a convex cost dual of iniu cost network flow. Their algorith for this convex cost dual of iniu cost flow has running tie of O(n log n2 log(nu)). The ethod of Hochbau and Naor, [9] solves integer probles on onotone inequalities in two variables at ost per inequality. A onotone inequality is of the for ax by c, where the coefficients of x and y are of opposite signs. Obviously, the constraints of (CCC) are onotone inequalities. The algorith of Hochbau and Naor runs in pseudopolynoial tie O( i (u i l i )U log n2 U ). It is possible to cobine that algorith with a scaling approach that is ipleented for (CCC) in tie O(n log U log n2 U ), [2]. The algorith described here solves (CCC) in tie O(n log n2 + n log U). The first ter in the coplexity expression is the run tie required to solve the iniu closure proble, and the second factor is the run tie required to find the integer inia of n convex functions. Since (CCC) generalizes both these probles as discussed above this is the best coplexity achievable for (CCC). It is likely that if a faster algorith for the iniu closure proble is discovered then the run tie of the algorith can be respectively iproved. For the second ter the factor of log U cannot be avoided as any algorith solving a constrained nonlinear and nonquadratic optiization proble ay not run in strongly polynoial tie, [11]. When the functions f i are quadratic convex, the algorith runs in strongly polynoial tie O(n log n2 + n log n). For the isotonic regression proble the running tie iproves to O(n log n + n log U), and thus the coplexity of our algorith is O(n log(ax{n, U})). There are efficient algoriths known for solving several special cases of (CCC). Any axiu flow algorith can be used to solve the iniu (or axiu) closure proble. The ost efficient algorith known to date solving the axiu flow and thus the iniu cut and the closure probles has coplexity of O(n log n2 ) due to Goldberg and Tarjan, [8]. The isotonic regression proble is an instance of (CCC) defined on a linear order. Ahuja and Orlin report on an O(n log U) tie algorith for this proble, [1]. The proble has been studied extensively in the statistical study of observations. The book of Barlow et. al. [4] provides an excellent review of applications and algoriths for the isotonic regression proble as well as the (CCC) proble. 3. Solving the iniu closure as a iniu cut proble. Recall that the iniu closure proble is a special case of (CCC) attained by setting the variables to be binary. We review here the procedure for solving the iniu closure proble with a iniu cut algorith. Although (CCC) is a proble far ore general than the iniu closure, our algorith for (CCC) also solves the iniu closure proble in the ost efficient coplexity known. A set of nodes S V in a directed graph G = (V, A) is said to be closed if all predecessor nodes of S are also included in S, i.e. if j S and (i, j) A then i S. Equivalently, S is said to be closed if there are no incoing arcs into S. We review here the reduction of Picard [17] deonstrating that the iniu

5 convex optiization over order constraints 5 closure proble is solved using a iniu cut procedure. We first define an s, t-graph, that contains a source and a sink, associated with the iniu closure proble: The graph has a node j associated with each variable x j. A source and sink nodes, s and t are now added to the graph. If the weight of the variable w j is positive then node j has an arc fro the source into it with capacity w j. If the node has weight w j which is negative, then there is an arc fro j to t with capacity w j. Let V + be the set of nodes with positive weights, and V the set of nodes with negative weights. Each inequality x i x j is associated with an arc (i, j) of infinite capacity. Consider any finite s, t-cut in the graph that partitions the set of nodes into two subsets coonly referred to as the source set of the cut and the sink set of the cut, {s} S and {t} S. It is easy to see that S is a closed set if there are no infinite capacity arcs fro S to S. We denote by (A, B) the collection of arcs with tails at A and heads at B. The corresponding su of capacities of these arcs is denoted by C(A, B), C(A, B) = i A,j B c ij where c ij is the capacity of arc (i, j). Let w(a) = j A w j. Given a finite cut ({s} S, S {t}). in S V [C({s} S, S {t})] = in S V j S V + w j + j S V ( w j) = in S V j S V + w j ( i V w i i S V w i) = in S V j S w j w(v ). In the last expression the ter w(v ) is a constant. Thus the closed set S of iniu weight is also the sink set of a iniu cut and, vice versa the sink set of a iniu cut (without t) (which has to be finite) also iniizes the weight of the closure. 4. Verifying feasibility in linear tie. We define a graph associated with (CCC) that has one node representing each variable in the proble. We let the set of nodes be denoted by V. Each inequality x i x j is associated with an arc (i, j). We let the set of arcs be denoted by A. If the directed graph (V, A) has strongly connected coponents then each node in the strongly connected coponent shares a directed cycle with each of the other nodes in the strongly connected coponent and thus the values of the corresponding variables ust be equal. Finding the strongly connected coponents of a graph can be accoplished in O() tie, see e.g. Coren, Leiserson and Rivest [6] chapter 23. The strongly connected coponents partition the nodes of the graph into V 1 V k. In each strongly connected coponent V i we let l(v i ) be the tightest lower bound in V i, and u(v i ) be the tightest upper bound in V i. That is, l(v i ) = ax v V i l v, and u(v i ) = in v V i u v. A necessary condition for feasibility is that all variables in the sae strongly coponent assue the sae value that falls in the interval range [l(v i ), u(v i )]. Since the above recursion is perfored in linear tie, verifying this necessary condition aounts to checking that l(v i ) u(v i ) in O( + n) steps. We now consolidate each strongly connected coponent into a single node V i defined on the interval [l(v i ), u(v i )]. The function associated with such node (or variable) is the su of the convex functions associated with all nodes in V i, which is a convex function. The graph of strongly connected coponents is thus a Directed Acyclic Graph, DAG.

6 6 D. S. HOCHBAUM AND M. QUEYRANNE Let V i is a predecessor of V j in different strongly connected coponents. Then the following updates are valid l(v i ) ax{l(v j ), l(v i }), u(v j ) in{u(v i ), u(v j )}. All these updates can be perfored in tie O(). A necessary condition for feasibility is that for i = 1,..., k, l(v i ) u(v i ). This condition is also sufficient since, if satisfied, there exists a feasible solution which is, say, to set all variables to the lower bounds of their corresponding intervals. Our optiization algorith runs faster if the feasibility preprocessing step is perfored and the interval bounds are adjusted. This preprocessing step however is not essential, and does not affect the worst case coplexity. 5. The threshold theore. The threshold theore is the corner stone of our algorith. The theore is an extension of an earlier result of Picard and Queyranne, [19]. Let α be a scalar in the interval (l, u) = (in i V l i, ax i V u i ). Consider further the convex extension of the functions f i () on the real line by setting f i (x) to be equal to M at values of x > u i, and to M for values x < l i, for M a suitably large value. We will coent after the stateent of the theore on how large M should be. The functions f i () are therefore defined for any real value x as follows: f i (u i ) + M(x u i ) if x > u i f i (x) = f i (x) if l i x u i f i (l i ) + M(l i x) if x < l i. Consider now the iniu closure proble with variable weights w i = w i (α) that are the subgradients of f i at α, w i = f i (α) = f i(α + 1) f i (α). The theore establishes that all eleents i in the iniu weight closed set S satisfy that for any optial solution x, x i > α and for all eleents j in the copleent of S satisfy that x j α. Consequently, the theore allows the reduction of (CCC) to a sequence of iniu closure probles. In case there are several optial iniu closed sets we ter a inial iniu closed set that iniu closed set that does not contain other iniu closed sets. Siilarly a axial iniu closed set is a iniu closed set that is not contained in another iniu closed set. Theore 5.1. Let w i = f i (α) be the weight assigned to node i, i = 1,..., n in a iniu closure proble defined on the directed graph G = (V, A). Let S be the inial iniu weight closed set in this graph. Then an optial solution x to the convex cost closure proble satisfies, x i > α if i S and x i α if i S. Proof. The proof is by contradiction. Let S be the inial iniu weight closed set, and suppose there is a nonepty subset S o S such that at an optial solution x, x j α for all j So. Since at the optiu x j > α for j S \ S o, then the set S \ S o ust be closed, as it has no predecessors (larger values) in S o. But this set is not a inial iniu closed set as S is inial. Thus the weight of nodes in S o the total su of subgradients ust be negative, j S f o j (α) < 0. Furtherore increasing the values of all x j in this set to α + ɛ in i S \S o x i for soe ɛ > 0 does not violate feasibility, since the values of their predecessors in S \ S o are all α + ɛ. Thus replacing x j for j So by α is feasible and strictly reduces the weight of the closure copared to an optial solution. This contradicts the assuption that x is optial.

7 convex optiization over order constraints 7 An analogous contradiction is reached if we assue that an optial solution has in the set S a variable with value > α. As a result of the theore, we can decopose the set of nodes into subsets that iply a narrowing of the interval in which the optial value of the respective variable is to be found: For a given value of α we solve the iniu closure proble with w i = f i (α) for a inial iniu closure S. For all i S we conclude that x i (ax{l i, α}, u i ] and for all j S, x j [l j, in{α, u j }]. Concerning the value of M, it is sufficient to set M = i ax{f i (u i), f i (l i) }. We clai that for a feasible proble a node with weight M is never in a iniu weight closed set and a node with weight M is always in a iniu weight closed set. If that were not the case then either we can generate a closed set of a strictly lower value by including nodes of weight M and excluding nodes of weight M, or else there is a node j of weight M that has as predecessor a node i of weight M. But that eans that the given value of α satisfies u i < α < l j and there is no feasible solution where the value of x i [l i, u i ] is at least as large as the value of x j [l j, u j ]. Indeed the algorith we eploy to solve (CCC) can be used to verify feasibility as well for every value of α the nodes of weight M ust be in the source set and the nodes of weight M ust be in the sink set or else the proble is infeasible. Yet, if the feasibility test of Section 4 is used then whenever the threshold theore is invoked in the algorith the nodes of weight M are known a priori to be in the iniu closure and thus in the sink set and those node of weight M are known to be in the source set. This perits the shrinking of nodes of weight M with the source and nodes of weight M with the sink. The size of the graph is thus reduced and the value of M is not explicitly used if the feasibility test is invoked as a preprocessing step. 6. The algorith. One obvious ethod of using the threshold theore for solving (CCC) is to perfor a search by calling for the solution of the iniu closure proble for all integer values of α in the interval (l, u). When done, the output of such process is a partition of the set of variables V into q sets, and the interval into q disjoint intervals, so that all variables in the sae set have their optial value lie in the sae interval. The goal would be to find for each variable x j the largest of value of α for which it is still in the source set, and the sallest value of α for which it is no longer in the source set. With this inforation we narrow down the value of x j at an optial solution to an interval defined by these values. We later show that once these intervals are identified, all variables assigned to the sae interval assue the sae value in that interval, and that value can be deterined in polynoial tie. One drawback of the approach just described is that it akes U calls to a iniu cut procedure, and is thus pseudopolynoial. It is easy to see that a binary search type approach could be used to ipleent the procedure of identifying the intervals to a polynoial tie procedure. Next we show that one can do better still by ipleenting the process of identifying the set and interval partitioning in strongly polynoial tie and in the coplexity of solving a single iniu cut proble. The key to our approach is to utilize paraetric iniu cut to generate all the breakpoints of the decopositions. This can be done, as is shown here, by adapting the ethod of Gallo, Grigoriadis and Tarjan [7] that works in the sae running tie as a single iniu cut procedure The paraetric graph G λ. We create a graph with paraetric capacities, G λ = (V {s, t}, A). Each node j V has an incoing arc fro s with capacity

8 8 D. S. HOCHBAUM AND M. QUEYRANNE ax{0, f j (λ)}, and an outgoing arc to the sink t with capacity in{0, f j (λ)}. The capacities of the arcs adjacent to the source in this graph are onotone nondecreasing as a function of λ, and the arcs adjacent to the sink are all with capacities that are onotone nonincreasing as a function of λ. Note that each node is connected with a positive capacity arc, either to source, or to sink, but not to both. Denote the source set of a iniu cut in the graph G λ by S λ. Restating the threshold theore in ters of the corresponding iniu cut for the graph G λ associated with the closure graph, any optial solution x satisfies that x j > λ for j S λ and x j λ for j S λ, where S λ is the axial source set of a iniu cut. Let l be the lowest lower bound on any of the variables and u the largest upper bound. Consider varying the value of λ in the interval [l, u]. As the value of λ increases, the sink set becoes saller and contained in the previous sink sets corresponding to saller values of λ. Specifically, for soe λ l S λ = {s}, and for soe λ u S λ = V {s}. We call each value of λ where S λ strictly increases a node shifting breakpoint. For λ 1 <... < λ l the set of all node shifting breakpoints we get a corresponding nested collection of source sets: {s} = S λ1 S λ2... S λl = {s} V. Our goal is to partition the variables into the subsets S(k) = S λk S λk 1, k = 2,..., l. The property of each subset S(k) is that all variables in the set have optial value in the interval (λ k 1, λ k ]. As we prove next, the optial value of all variables in S(k) is x where f i (x ) = in f i (x). i S(k) x (λ k 1,λ k ] i S(k) Lea 6.1. For j S(k), the value of x j at an optial solution, x j, is the arguent iniizing, in f i (x). x (λ k 1,λ k ] i S(k) Proof. First we show that all variables in each set S(k) assue the sae value in an optial solution. According to the threshold theore λ k 1 is the largest value so that for j S(k), x j > λ k 1, and λ k is the sallest value so that x j λ k. Thus all variables x i, with i S(k), have their optial value lie in the interval (λ k 1, λ k ]. Suppose two variables in S(k) assue different optial values in the interval x i 1 < x i 2. Then there exists a value λ [x i 1, x i 2 ] so that the source set S λ contains the node i 2 but not i 1. But there is no breakpoint in (λ k 1, λ k ] which is a contradiction. Let α k (λ k 1, λ k ] be the coon value of the variables in S(k), for k = 1,..., q. Then α 1 < α 2 <... < α q. Now the objective value of the convex cost closure proble is, q in f i (α k ). k=1 i S(k) This expression is separable into functions of α k and is iniized when each one of these functions is iniized separately.

9 S ax λ 1 convex optiization over order constraints Identifying an integer node-shifting breakpoint. Since we are only interested in integer valued solutions, we can consider the convex functions f j (x) to be piecewise linear segents connecting the values of f j (k) on integer points l j k u j. For such functions the derivatives at the integer points are not well defined, and indeed could be any subgradient of the function at the respective integer point. We will consider the derivative f j (x) to be a step function with the value in the interval (k 1, k] equal to f j (k) f j (k 1). We denote a axial iniu cut source set in G λ by Sλ ax and a inial iniu cut source set, by Sλ in. The source set of a iniu cut of G λ reains invariant for λ (k 1, k]. Thus, in order to verify that λ is a node-shifting breakpoint, it suffices to copare Sλ ax with Sλ+ɛ in for ɛ > 0 sufficiently sall. In our case we only consider integer values of λ and ɛ = 1 is a sall enough value. So if Sλ ax Sλ+1 in, then λ is a node shifting breakpoint. The existence of a breakpoint in an interval (λ 1, λ 2 ) is confired if and only if Sλ in Paraetric analysis. Gallo, Grigoriadis and Tarjan [7] devised a coplete paraetric analysis algorith using the push-relabel algorith that runs in the sae tie as a single push-relabel algorith and identifies all node-shifting breakpoints. The algorith is applicable to graphs with source adjacent arcs having capacities onotone nondecreasing in the paraeter λ, and sink adjacent arcs having capacities nonincreasing in λ. The running tie of the algorith is O(n log n2 ). The sae result is achieved using the pseudoflow algorith [13] with running tie of O(n log n). We let the generic run tie be Qn where Q is a constant ties log n2 for push-relabel and a constant ties log n for pseudoflow. Whenever we refer in the analysis below to a iniu cut algorith it can be either the push-relabel algorith or the pseudoflow algorith (and its variants). Other iniu cut algoriths do not satisfy the necessary requireents to ake the aenable to the analysis of the paraetric procedure. We assue henceforth that the procedure in-cut(g λ ) returns both the inial and axial source sets of iniu cuts (if different), Sλ in, Sλ ax and Sλ+1 in. The procedure also returns the state of the graph at the end of the run which includes node labels and preflows for the push-relabel algorith, and node labels and pseudoflows for the pseudoflow algorith. For a given interval (λ 1, λ 2 ) we can find all node shifting breakpoints by using the procedure paraetric. The input to the procedure includes R 1 and R 2 which are runs of the iniu cut algorith that are initiated on an s, t graph G and the reverse graph G R respectively. Procedure paraetric (G, λ 1, λ 2, Sλ ax 1, Sλ in 2, R 1, R 2 ) Contract in G: s s Sλ ax in 1, t t S λ 2. If V = {s, t}, or, if λ 2 λ 1 1, halt no breakpoints. Else, let λ = λ 1+λ 2 2. Call in-cut(g λ, R 1, R 2 ) for the output S in, Sax λ and R. If λ is a breakpoint, output λ and S in Call paraetric (G, λ 1, λ, Sλ ax 1 end Call paraetric (G, λ, λ 2, S ax λ λ. λ, Sλ in, R 1, R ), Sin λ 2, R, R 2 )

10 10 D. S. HOCHBAUM AND M. QUEYRANNE The choice of λ as the edian in the interval (λ 1, λ 2 ) is uniportant. Any integer value interior to the interval will work correctly. The analysis of the coplexity of the procedure follows arguents used in [7] 1. It is essential that the algorith used in the runs for iniu cut satisfies the properties Reflectivity: The coplexity of the algorith reains the sae whether running it on the graph or reverse graph. Monotonicity: Running the algorith on a onotone sequence of paraeter values has the sae coplexity as a single run. The ain recursive procedure is in-cut(g λ, R 1, R 2 ) where R 1 is the status of the graph (labels assigned to nodes and flow values) G λ1 after a iniu cut was identified and R 2 is the state of the graph after the iniu cut was found on the reverse graph G R λ 2. The procedure is ipleented as follows: Run a axiu flow algorith on G λ as a onotone continuation of the run R 1. Concurrently, run a axiu flow algorith on G R λ as a onotone continuation of the run R 2. Suppose that the algorith for the forward direction (on G) stops first (the other case is syetric.) If Sλ in > n/2 coplete the execution of the axiu flow algorith on GR and let R be the resulting state of the graph. Consider the execution that follows iediately of the recursive calls to paraetric (G, λ 1, λ, Sλ ax 1, Sλ in, R 1, R ), and to paraetric(g, λ, λ 2, Sλ ax, Sin λ 2, R in, R 2 ). Consider graphs G( S λ ) and G(Sax λ ) on which in-cut is called recursively. Let R 3 and R 4, respectively, be the forward and backward runs on G( S in λ ), when in-cut is applied. Let R 5 and R 6, respectively, be the forward and backward runs on G(Sλ ax ) when in-cut is applied. We distinguish two cases. Case 1. If n 1 > n/2, we regard R 4 as a continuation of R 2, and R 3 as a restart of R 1 ; that is, as a continuation of the run of which R 1 was a continuation. We ust charge for R 5 and R 6 as starts of new runs. The 2Q 1 n 1 ter in the recurrence for T (, n) below accounts for the new runs of the push-relabel algorith that begin with R 5 and R 6. Case 2. Syetrically, if n 1 n/2, we regard R 5 as a continuation of R 1, and R 6 as a restart of R 2. In this case, the 2Q 1 n 1 ter in the recurrence for T (, n) below accounts for the new runs that begin with R 3 and R 4. In Case 1, we ust still account for the cost of R 1. In Case 2, we ust still account for the cost of R 2. in-cut runs R 1 and R 2 concurrently, stopping when the first one stops. Suppose R 1 stops first. Then the cost of R 1 is covered by the cost of R 2, which takes care of Case 1. Note that in this case R 2 is run to copletion, even though it takes longer than R 1 (see ipleentation); R 1 is the abandoned run, but it is cheaper than R 2, which is the good run. Suppose R 1 stops first, but we are in Case 2. Then run R 2 is abandoned, but we have spent no ore tie on it than the tie spent running R 1, which was a good run. In this case, the run of R 1 covers the tie spent on (partially) running R 2. The situation is syetric if R 2 stops first. In every case the tie spent on the copleted good run is at least as uch as the tie spent on partially or copletely perforing the run that gets abandoned. Throughout the procedure the total coplexity of abandoned runs is at ost the coplexity of one run with onotonically increasing (or decreasing) paraeter values. In addition the total work for good runs on G λ and G R λ is at ost twice the 1 The source of soe of this analysis is fro private counication of the first author with R. Tarjan 1996.

11 convex optiization over order constraints 11 coplexity of one run on onotone paraeter values. The total coplexity charged for these runs is at ost that of three runs of the iniu cut algorith, 3Qn. Let 1 + 2, n 1 + n 2 n and n 1 1 2n. The running tie T (, n) is the additional running tie required by the algorith taking into account the new runs initiated with each recursive call to in-cut. Let Q be a constant, then T (, n) = T ( 1, n 1 ) + T ( 2, n 2 ) + 2Q 1 n 1. The solution to the recursion is T (, n) = Qn. Thus the overall run tie of the paraetric procedure with the push-relabel algorith is O(n log n2 ) The algorith. Let l be the lowest lower bound on any of the variables and u the largest upper bound. Let U = u l. procedure convex cost closure (G, f j, j = 1,..., n) Step 1: Call paraetric (l, u,, V ). Let the output be a set of up to n breakpoints λ 1, λ 2,..., λ l and the corresponding sets of source sets of iniu cuts S 1 S 2... S l. Step 2: Find the iniu of each convex function in x (λk 1,λ k ] f j S k S j (x), in k, k 1 for k = 2,..., l. Step 3: Output the optial solution x where for j S k S k 1, x j = in k. In step 2, identifying the iniu in k aounts to finding in the sorted array of derivative values j S k S k 1 f j (x) for x integer in (λ k 1, λ k ], the first index k so that f j (k ) is nonnegative. This can be accoplished using binary search in O(log U) tie per function and O(n log U) total coplexity. The coplexity of the algorith is O(n log n2 +n log U): The paraetric search generating all breakpoints is ipleented in tie of O(n log n2 ); the second step of the algorith requires finding all the integer inia of the convex functions for a total coplexity of O(n log U); and Step 3 is accoplished in O(n) tie. 7. Special cases The quadratic convex cost closure proble. Nonlinear and nonquadratic optiization probles with linear constraints were proved ipossible to solve in strongly polynoial tie in a coplexity odel of the arithetic operations, coparisons, and the rounding operation, [11]. That negative result however is not applicable to the quadratic case, thus it ay be possible to solve constrained quadratic optiization probles in strongly polynoial tie. Yet, very few quadratic optiization probles are known to be solvable in strongly polynoial tie. For instance, it is not known how to solve the iniu quadratic cost network flow proble in strongly polynoial tie. For the convex quadratic cost closure proble our result adds to the liited repertoire of quadratic probles solved in strongly polynoial tie. In the quadratic case our algorith is ipleented to run in strongly polynoial tie. This is easily achieved since finding the inia in the second phase of the algorith aounts to solving a linear equation in one variable. So the run tie required for finding all the inia is O(n) and thus the overall run tie of the algorith is doinated by the coplexity of the iniu cut, O(n log n2 ).

12 12 D. S. HOCHBAUM AND M. QUEYRANNE 7.2. Isotonic regression. The isotonic regression proble is a special case of (CCC) where the order is linear and the corresponding graph G = (V, A) is a path fro node n to node 1. In other words, the inequalities associated are of the type, x i x i+1 for all i = 1,..., n. There are only n possible cuts in such graph, each with a source set S i of the for, S i = {1,..., i}. Each cut is thus ({1,..., i}, {i + 1,..., n}). The iniu cut for such graphs is trivially identified in O(n) tie, by coparing the capacities of the n possible cuts. The capacity of cut (S i, S i ) is coputed in O(1) by subtracting fro the capacity of (S i 1, S i 1 ) the weight of node i, w i. Indeed, if the weight w i is positive then it contributes w i to the capacity of the cut (S i 1, S i 1 ), but not to the capacity of the cut (S i, S i ). If w i < 0 then node i contributes w i to the capacity of (S i, S i ), but 0 to the capacity of (S i 1, S i 1 ). Consider the closure graph in which each node has a weight f j (x) associated with it for a given value of x. Miniizing the value of the cut is equivalent to iniizing the su of weights of the sink set, (see Section 3). Alternatively, the cut is iniized when the weight of the corresponding source set is axiized thus seeking an index i to axiize F i (x) = i j=1 f j (x). We thus conclude, Lea 7.1. If i j=1 f j (x) = ax k k=1,...,n j=1 f j the graph G x is (S i, S i ). Consider the partial su functions F 1 (x), F 2 (x),..., F n (x) (x) then the iniu cut in where F i (x) = i j=1 f j (x). Recall that the functions f j (x) are onotone nondecreasing in x. Denote the roots of the partial su functions by b i. So F i (b i ) = 0. If the function is negative in the interval [l, u] then we let b i = u + 1. If the function is positive throughout the interval then we let b i = l 1. Let b i1 = in i b i, then for x b i1 the optial iniu cut is (, V ). For this cut, the axiu weight source set is epty, since all the partial sus of weights are nonpositive. The value of λ 1 = b i1 is thus a breakpoint beyond which, for x > b i1, the source set of the iniu cut is {1,..., i 1 }. As the value of x increases sufficiently so that i 2 j=i 1 +1 f j (x) = F i 2 (x) F i1 (x) 0, the nodes {i 1,..., i 2 } join the source set of the iniu cut. In other words, the second breakpoint is the sallest value λ 2 so that there is an index i 2 > i 1 such that The general procedure is as follows: F i2 (λ 2 ) F i1 (λ 2 ) 0. procedure Isotonic regression breakpoints i 0 = 0, λ 0 = l 1, k = 1 while i k 1 < n, do Find sallest integer value of λ k such that for i k > i k 1, F ik (λ k ) F ik 1 (λ k ) 0. k k + 1 repeat Output λ 1,..., λ k. end A naive ipleentation of this algorith has n iterations with each iteration involving the finding of the roots of O(n) functions. The total coplexity is O(n 2 log U).

13 convex optiization over order constraints 13 We can do better with the ipleentation of the paraetric search algorith that requires the solution of the iniu cut proble for a specific paraeter value in O(n). However, each tie the procedure calls for the iniu cut, the weights of the nodes ust be updated for the new paraeter value. This update requires O(n) operation. The additional work of finding the roots of the n functions adds up to a total coplexity of O(n 2 + n log U). To achieve a better yet running tie we investigate further the properties of the partial su functions F i (x). These functions are obviously onotone nondecreasing as sus of onotone nondecreasing functions. Another iportant property proved in the next lea is that each pair of such functions intersects at ost once. Lea 7.2. For i < j and functions F i and F j, if for soe value of the arguent λ, F i (λ) < F j (λ) then F i (x) < F j (x) for any x > λ. Proof. F j (x) F i (x) is a su of onotone nondecreasing functions j k=i+1 f k (x). Thus the difference F j (x) F i (x) > F j (λ) F i (λ) > 0 and can only increase as the value of x grows. Thus the two functions do not intersect for any value of x > λ. An iediate corollary of the lea is that any pair of functions F i, F j can intersect at ost once. Consider the upper envelope of the functions F i represented as an array of functions and breakpoints (l, 0, b i1, F i1, b i2, F i2,..., b in, F in, u). The functions on the envelope have the property that for all j, F ik (x) F j (x) x [b ik 1, b ik ]. Fro the lea it follows that the upper envelope of the partial sus functions has at ost n breakpoints where the function on the envelope changes. The first breakpoint is b i1. The next breakpoint occurs for a value of x when soe function F i2 (x) = F i1 (x). It is easy to see fro procedure Isotonic regression breakpoints that the list of breakpoints of this envelope is precisely the list of the breakpoints that deterines the sequence of cuts. The following sweep algorith ay be used to find the upper or upper envelope of a set of functions: Partition arbitrarily the set of functions into two equal size sets F 1, F 2. Copute recursively the upper envelopes of F 1, F 2. Let E 1, E 2 denote the two resulting upper envelopes. Sweep the two upper envelopes E 1, E 2 fro left to right, and copute the upper envelope of the two upper envelopes. For a detailed description of the above algorith the reader is referred to [22] (page ) or to [5]. It reains to show how to ipleent the sweep algorith for our particular set of functions. Instead of partitioning arbitrarily the set of functions we choose the partition of F 1 = {1,..., n} to {1,..., n 2 } and F 2 = { n 2 + 1,..., n }. That is, one set contains the lower half set of indices and the other the upper half set of indices. Consider the first breakpoint in E 1 and in E 2 (recall that at that point the partial su values are still 0). If the first breakpoint of E 1 is larger then the first breakpoint of E 2 than the first portion of E 1 is below the first portion of E 2. Fro the lea no pair of the functions fro the two sets intersect, and the entire envelope E 1 lies below the envelope E 2. Thus the erged envelope is E 2. If, on the other hand, the first breakpoint of E 1 is saller than the first breakpoint of E 2 then there could be a point where a functions fro F 2 crosses a function fro F 1. We consider the array of breakpoints of the envelope E 1 for the last breakpoint where it is still above E 2. Siilarly, we search the array of breakpoints of the envelope

14 14 D. S. HOCHBAUM AND M. QUEYRANNE E 2 for the last breakpoint where it is still below E 1. Since there are O(n) breakpoints per envelope, the search for that breakpoint is done by binary search in O(log n) steps. The intersection point is then to be deterined between the function between this breakpoint and the next one on each envelope. Finding the intersection of F i (x) and F j (x) takes at ost O(log U) steps. Thus the erger of two envelopes of functions is executed in O(log n+log U). Since there are at ost n ergers in the procedure, the total running tie is O(n log n + n log U). Once all the upper envelope has been identified we have the iplied source sets of the associated cuts: {1,..., i 1 }, {1,..., i 2 },..., {1,..., i q }. If i {i k 1,..., i k } then x i [b i k 1, b ik ]. It reains to apply the equivalent of step 3 in procedure convex cost closure in order to deterine an optial solution: b ik if in i b ik 1 x i = b ik if in i b ik in i if b ik 1 in i b ik. Thus the total coplexity of the algorith for the isotonic regression proble is O(n(log U + log n). In the quadratic case this leads to a coplexity of O(n log n). 8. The continuous convex cost closure proble. When solving the proble in continuous variables, one has to deterine how to output the solution. For instance, the iniu of a cubic function can be irrational even if all coefficients are integers. To fully provide the output would then require infinite coplexity. To that end we eploy the ɛ-accurate coplexity odel introduced in [10]. According to this odel a solution x (ɛ) is specified as an integer ultiple of ɛ, i.e. it lies on the so-called ɛ-grid. The solution is such that there is an optial vector x so that, x (ɛ) x < ɛ. The continuous proble can be solved using the sae algorith as used for the integer case. The only odification required is in the finding of ɛ-accurate roots of the functions. This can be done in O(log(U/ɛ) per function using binary search. In the paraetric analysis procedure the choice of λ is such that it is any interior point in the interval (λ 1, λ 2 ) that lies on the ɛ-grid. The coplexity of the algorith is thus the coplexity of finding the roots of the n functions plus the coplexity of a iniu cut, O(n log n2 + n log(u/ɛ)). 9. Conclusions and extensions. The results here have been extended to a proble that is ore general than the convex cost closure proble. The proble is the convex s-excess proble which generalizes the s-excess proble discussed in [13]. The proble is forulated as follows, (Convex s-excess) Min j V f j(x j ) + e ij z ij subject to x i x j z ij for (i, j) A u j x j l j j = 1,..., n z ij 0 (i, j) A. This proble is solved with precisely the sae coplexity as the iniu closure proble. To that end, we proved a generalization of the threshold theore that is reported in [15].

15 convex optiization over order constraints 15 There are applications of the convex s-excess proble in the areas of iage segentation and Markov Rando Fields. The proble is of further interest because of its relationship to the iniu cost network flow proble. Notice that the ters associated with the variables z ij is linear. This is significant as the dual of the iniu cost network flow (MCNF) proble is, (Dual MCNF) Min j V b jx j + e ij z ij subject to x i x j c ij + z ij for (i, j) A u j x j l j j = 1,..., n z ij 0 (i, j) A. That is, the constraints right hand side has a constant ter in addition to the ter z ij. Thus, if one can solve the convex s-excess proble with convex function ter eij (z ij ) then it would have been possible to solve also the dual of the iniu cost network flow in the sae running tie as a single application of axiu flow or iniu cut. Acknowledgeent. The first author wishes to extend thanks to Sariel Har- Peled for his input on the state-of-the art of anipulating lower and upper envelopes of functions and pointing out references [5] and [22]. REFERENCES [1] R. K. Ahuja and J. B. Orlin, A fast scaling algorith for iniizing separable convex functions subject to chain constraints, Operations Research, Vol 49:5, (2001) pp [2] R. K. Ahuja, D. S. Hochbau and J. B. Orlin, A cut based algorith for the convex dual of the iniu cost network flow proble., UC Berkeley anuscript, (1999). [3] R. K. Ahuja, D. S. Hochbau and J. B. Orlin, Solving the convex cost integer dual network flow proble, Manageent Science to appear. Extended abstract in, Proceedings of IPCO 99, G. Cornuejols, R.E. Burkard and G.J. Woeginger (Eds.), Lecture Notes in Coputer Science 1610, (1999), pp [4] R. E. Barlow, D. J. Bartholoew, J. M. Breer and H. D. Brunk, Statistical Inference Under Order Restrictions, Wiley, New York, [5] M. de Berg and M. van Kreveld and M. Overars and O. Schwarzkopf, Coputational Geoetry: Algoriths and Applications. Springer-Verlag, Berlin, [6] T. H. Coren, C. E. Leiserson and R. L. Rivest, Introduction to Algoriths, MIT Press,Cabridge Mass., [7] G. Gallo, M. D. Grigoriadis and R. E. Tarjan, A fast paraetric axiu flow algorith and applications, SIAM Journal of Coputing, 18:1 (1989), pp [8] A. V. Goldberg and R. E. Tarjan, A new approach to the axiu flow proble, J. Assoc. Coput. Mach., 35 (1988), pp [9] D. S. Hochbau and J. Naor, Siple and fast algoriths for linear and integer progras with two variables per inequality, SIAM Journal on Coputing, 23:6 (1994), pp [10] D. S. Hochbau and J. G. Shanthikuar, Convex Separable Optiization is not Much Harder Than Linear Optiization, Journal of the ACM, 37:4 (1990), pp [11] D. S. Hochbau, Lower and upper bounds for allocation probles, Matheatics of Operations Research, 19:2 (1994), pp [12] D. S. Hochbau, A fraework for half integrality and 2-approxiations. Extended abstract as Instant Recognition of Half Integrality and 2-Approxiations. Lecture Notes in Coputer Science 1444, APPROX98, Jansen and Roli (Eds.), (1998), pp [13] D. S. Hochbau, The pseudoflow algorith for the axiu flow proble, anuscript, UC Berkeley, [14] D. S. Hochbau, Efficient algoriths for the inverse spanning tree proble, anuscript, UC Berkeley, [15] D. S. Hochbau, An efficient algorith for iage segentation, Markov Rando Fields and related probles, Journal of the ACM, 48:4 (2001), pp [16] W. L. Maxwell and J. A. Muckstadt, Establishing consistent and realistic reorder intervals in production/distribution systes, Operations Research, 33:6 (1985), pp

16 16 D. S. HOCHBAUM AND M. QUEYRANNE [17] J. C. Picard, Maxial Closure of a Graph and Applications to Cobinatorial Probles, Manageent Science, 22 (1976), pp [18] J.-C. Picard and M. Queyranne, Selected applications of iniu cuts in networks, INFOR, 20 (1982), pp [19] J. C. Picard and M. Queyranne, Integer iniization of a separable convex function subject to variable upper bound constraints, anuscript, [20] R. Roundy, A 98%-effective integer-ratio lot-sizing for one-warehouse ulti-retailer systes, Manageent Science, 31 (1985), pp [21] M. I. Shaos and D. Hoey, Geoetric intersection probles, in Proc. 17th Ann. Syp. on Foundations of Coputer Science, (1976), pp [22] M. Sharir and P. K. Agarwal, Davenport-Schinzel Sequences and Their Geoetric Applications. Cabridge University Press, New York, [23] P. T. Sokkalinga, R. Ahuja and J. B. Orlin, Solving inverse spanning tree probles through network flow techniques, Operations Research, 47 (1999), pp [24] A. F. Veinott, Jr., Least d-ajorized network flows with inventory and statistical applications, Manageent Science, 17 (1971), pp