Combinatorial optimization problems Heuristic Algorithms Giovanni Righini University of Milan Department of Computer Science (Crema)
Optimization In general an optimization problem can be formulated as: where x is a vector of variables; z(x) is the objective function; minimize z(x) subject to x X X is the feasible region, i.e. the set of solutions satisfying the constraints. A solution is an assignment of values to the variables.
Combinatorial optimization In general a combinatorial optimization problem can be formulated as: minimize z(x) subject to x {0, 1} E The feasible region X is defined as a subset of the set of all possible subsets of a given ground set (!). Let s say it again with an example...
Combinatorial optimization: example Ground set E: the set of edges of a given graph G. All possible subsets of E are the 2 E subsets of edges. Only a subset X of them are, for instance, spanning trees. The condition x X describes the feasible region of any problem involving the search of an optimal spanning tree. Although the ground set E is rather small, the number of its subsets is exponential in its cardinality (2 E ) and hence the number of solutions can be very large. Even restricting the search to feasible solutions, the cardinality of X can be combinatorial: it grows as a combinatorial number when E grows.
The combinatorial problems structure The variables, the constraints and the objective function of a combinatorial optimization problem define its combinatorial structure. This is a semi-informal way to indicate the main characteristics of the problem, that affect the effectiveness of different solution procedures. The analysis of the combinatorial structure of any given problem gives useful indications on the most suitable algorithm to solve it. Moreover it can uncover similarities between seemingly different problems.
Problems on weighted sets: the knapsack problem (KP) From a ground set of items, select a subset such that the value of the selected items is maximum and their weight does not exceed a given capacity. We are given a ground set N of items a weight function a : N N a capacity b N a value function c : N N We can associate a binary variable x with each element of the ground set: the solution space is {0, 1} n, where n = N. In this way every solution corresponds to a subset and to its binary characteristic vector x. The feasible region X contains the subsets with total weight not larger than b: X = {x {0, 1} n : j N a j x j b} The objective is the maximization of the total value: max z(x) = j N c j x j.
Example A D B E C F Knapsack N A B C D E F c 7 2 4 5 4 1 a 5 3 2 3 1 1 b = 8 x = (0, 0, 1, 1, 1, 0, 0) X x = (1, 0, 1, 1, 0, 0, 0) X z(x ) = 13 z(x ) = 16 D Knapsack E D C Knapsack C A
Problems on subsets with a metric: the Max Diversity Problem (MDP) Given a ground set of items, we want to select a subset of given cardinality, maximizing a measure of the pairwise distances between the selected items. We are given a ground set N, a distance function d : N N N, a positive integer number k {1,..., N }. A possible choice of the variables is analogous to the previous case: x is the binary characteristic vector of the selected subset. The feasible region X contains all the subsets of cardinality k: X = {x {0, 1} n : j N x j = k}. The objective is to maximize the sum of the pairwise distances: max z(x) = d ij x i x j. i,j N
Example A B F D C E G x = (0, 0, 1, 1, 1, 0, 0) X x = (1, 0, 1, 0, 0, 0, 1) X z(x ) = 24 z(x ) = 46 A B F D C E G A B F D C E G
Additive and non-additive objective functions In general the objective function associates rational or integer values with (feasible) subsets of the ground set. z : X N Computing its value can be more or less difficult. The KP has an additive (linear) objective function: its value is the sum of the values of another value function c whose domain is the ground set: c : N N The MDP has a non-additive (quadratic) objective-function. Both of them are easy to compute, but the additive objective function of the KP is easier to update when an item is inserted or deleted from the solution. It is enough to add c j for each inserted item j N; subtract c j for each deleted item j N. For the non-additive objective function of the MDP this is not true.
Set partitioning problems: the Bin Packing Problem (BPP) A set of weighted items must be partitioned into the minimum number of subsets, so that the total weight of each subset is within a given capacity. We are given: a set N of items, a set M of bins, a weight function a : N N, a capacity b of the bins. The ground set of the problem contains all the item-bin pairs. E = N M A solution is represented by nm binary variables with two indices i N and j M. The feasible region contains the partitions of the items, complying with the capacity constraints: X = {x {0, 1} nm : j M x ij = 1 i N, i N a i x ij b j M}.
Set partitioning problems: the Bin Packing Problem (BPP) The objective is to minimize the number of bins used. min z(x) = {j M : x ij > 0}. i N To get rid of the function cardinality of, we need additional binary variables: the characteristic vector y of the bin subset. We obtain: X = {(x, y) {0, 1} nm+m : j M x ij = 1 i N, i N a i x ij by j j M}. min z(y) = j M y j.
Example S = { (A, 1),(B, 1),(C, 2),(D, 2),(E, 2),(F, 3), (G, 4),(H, 5),(I, 5) } X y = (1, 1, 1, 1, 1) z(y ) = 5 S = { (A, 1),(B, 1),(C, 2),(D, 2),(E, 2),(F, 3), (G, 4),(H, 1),(I, 4) } X y = (1, 1, 1, 1, 0) z(y ) = 4
Set partitioning problems: the Parallel Machine Scheduling Problem (PMSP) A set of indivisible jobs of given duration must be assigned to a set of machines, minimizing the overall completion time. We are given: a set N of jobs, a set M of machines, a processing time function p : N N. The ground set contains all job-machine pairs. We can use the same variable choice as for the BPP. The feasible region X contains the partitions of N into subsets. X = {x {0, 1} nm : j M x ij = 1 i N.} The objective function is the minimization of the maximum working time among all machines. min z(x) = max j M p i x ij. i N
Example N = {L1, L2, L3, L4, L5, L6} M = {M1, M2, M3} Job L1 L2 L3 L4 L5 L6 p 80 40 20 30 15 80 M1 M2 L1 L2 L3 L4 L5 S = { (L1, M1),(L2, M2),(L3, M2), (L4, M2),(L5, M1) } X M3 L6 z(x ) = 95 95 M1 M2 L3 L1 L4 L5 L2 S = { (L1, M1),(L2, M1),(L3, M2), (L4, M2),(L5, M2) } X M3 L6 z(x ) = 120 120
Sensitive and insensitive objective functions The objective functions of the BPP and the PMSP are not additive, are not easy to compute. Small changes in the solution x may have different impact on the objective function value: variation equal to the duration of the changed job (e.g. L5 on M1); no variation (e.g., L5 on M3); intermediate variation (e.g., L2 on M2). This is because the effect of the change depends both on the modified elements, on the non-modified elements. In both problems the objective function is flat : many different feasible solutions have the same value.
Problems on matrices: the Set Covering Problem (SCP) Given a binary matrix and a vector of costs associated with the columns, select a minimum cost subset of columns covering all the rows. We are given: a binary matrix a B m,n with a set R or m rows and a set C of n columns, a cost function c : C N. A column j C covers a row i R if and only if a ij = 1. The ground set is the set of columns C. The feasible region contains the subsets of columns that cover all the rows. X = {x {0, 1} n : j C a ij x j 1 i R}. The objective is to minimize the total cost of the selected columns: min z(x) = j C c j x j.
Example c 4 6 10 14 5 6 0 1 1 1 1 0 0 0 1 1 0 0 a 1 1 0 0 0 1 0 0 0 1 1 1 1 1 1 0 1 0 0 1 1 1 1 0 2 0 0 1 1 0 0 1 a 1 1 0 0 0 1 1 0 0 0 1 1 1 1 1 1 1 0 1 0 3 0 1 1 1 1 0 1 0 0 1 1 0 0 0 a 1 1 0 0 0 1 2 0 0 0 1 1 1 2 1 1 1 0 1 0 2 x = (1, 0, 1, 0, 1, 0) X z(x ) = 19 x = (1, 0, 0, 0, 1, 1) X z(x ) = 15
The feasibility test In a heuristic algorithm the following sub-problem may often occur: Given a solution x, is it feasible or not? x X? This is a decision problem. The feasibility test may require an instant check on a single number (e.g. the total volume in the KP, the cardinality of the subset in the MDP) a quick scan of some attributes of the solution (e.g. exactly one machine for each job PMSP) the computation and check of many different values (e.g. the volume in each bin in the BPP). The time required may be different according to the feasibility test being done on a generic solution x; on a solution x obtained by a slight modification of a feasible solution x.
Problems on matrices: the Set Packing Problem Given a binary matrix and a weight vector associated with the columns, select a maximum weight subset of columns with no conflicts. We are given: a binary matrix a B m,n with a set R of m rows and a set C of n columns; a weight function w : C N. Two columns j, j C are in conflict if and only if there is at least a row i R such that a ij = a ij = 1. The ground set is the set of columns C. The feasible region contains the subsets of columns with no conflicts. X = {x {0, 1} n : j N a ij x j 1 i R}. The objective is to maximize the total weight of the selected columns. max z(x) = j N w j x j
Example w 4 6 10 14 5 6 0 1 0 0 1 0 0 0 1 1 0 0 a 1 0 0 0 0 1 0 0 0 1 1 1 1 1 1 0 0 0 0 1 0 0 1 0 1 0 0 1 1 0 0 1 a 1 0 0 0 0 1 0 0 0 0 1 1 1 1 1 1 1 0 0 0 1 0 1 0 0 1 0 2 0 0 1 1 0 0 0 a 1 0 0 0 0 1 1 0 0 0 1 1 1 2 1 1 1 0 0 0 1 x = (0, 1, 0, 1, 0, 0) X z(x ) = 20 x = (1, 0, 0, 0, 1, 1) X z(x ) = 15
Problems on matrices: the Set Partitioning Problem (SPP) Given a binary matrix and a cost vector associated with the columns, select a minimum subset of columns covering all the rows and with no conflicts. a binary matrix a B m,n with a set R of m rows and a set C of n columns; a cost function c : C N. The ground set is the set of columns C. The feasible region contains the subsets of columns covering all the rows with no conflicts. X = {x {0, 1} n : j N a ij x j = 1 i R}. The objective is to minimize the total cost of the selected columns. min z(x) = j N c j x j.
Example c 4 6 10 14 5 6 0 1 0 0 1 0 0 0 1 1 0 0 a 1 0 0 0 0 1 0 0 0 1 1 0 1 1 1 0 0 0 0 1 0 0 1 0 1 0 0 1 1 0 0 1 a 1 0 0 0 0 1 1 0 0 0 1 1 0 1 1 1 1 0 0 0 1 0 1 0 0 1 0 2 0 0 1 1 0 0 0 a 1 0 0 0 0 1 1 0 0 0 1 1 0 1 1 1 1 0 0 0 1 x = (0, 1, 0, 1, 0, 1) X z(x ) = 20 x = (1, 0, 0, 0, 1, 1) / X z(x ) = 15
The search for feasible solutions In a heuristic algorithm the following sub-problem may occur: Find a feasible solution x X. This is a search problem. For some problems the solution is trivial. in the KP the empty subset is feasible; in the MDP, any subset of k elements is feasible; in the SCP, the whole column set is feasible (or no feasible solution exists); in other problem it is enough to respect easy consistency constraints. But in some cases the search for a feasible solution is difficult. in the BPP it is not obvious how many bins are needed (an upper bound is the number of items); in the SPP no polynomial-time algorithm is known to provide a feasible solution.
Graph optimization problems. the Vertex Cover Problem (VCP) Given a graph G = (V, E), select a minimum cardinality vertex subset such that every edge is incident to it. We are given a graph G(V, E), where: V is the set of vertices, of cardinality n; E is the set of edges. The ground set is V (we have a binary variable for each vertex). The feasible region contains all vertex subsets covering the edges: X = {x {0, 1} n : x i + x j 1 [i, j] E} The objective is to minimize the number of selected vertices: min z(x) = i V x i
Example A B C D E F G H A B C D E F G H x = (0, 1, 0, 1, 1, 1, 1, 0) X z(x ) = 5 A B C D E F G H x = (1, 0, 1, 0, 0, 0, 0, 1) / X z(x ) = 3
Graph optimization problems: the Max Clique Problem (MCP) Given a graph with weights associated with the vertices, select a maximum weight vertex subset such that all its elements are adjacent to each other. We are given: a graph G = (V, E) with n vertices; a weight function w : V N. The ground set is the vertex set V. The feasible region contains the cliques (complete vertex subsets, i.e. vertex subsets containing all possible edges): X = {x {0, 1} n : x i + x j 1 [i, j] E}. The objective is to maximize the weight of the selected subset: max z(x) = i V w i x i.
Example A B C D E F G H Uniform weight: w i = 1 for each i V A B C D E F G H x = (0, 1, 1, 0, 0, 1, 1) X z(x ) = 4 A B C D E F G H x = (1, 0, 0, 1, 1, 0, 0) X z(x ) = 3
Graph optimization problems: the Max Independent Set Problem (MISP) Given a graph with weights associated with the vertices, select a maximum weight vertex subset such that all its elements are not adjacent to each other. We are given: a graph G = (V, E) with n vertices; a weight function w : V N. The ground set is the vertex set V. The feasible region contains the independent sets (empty vertex subsets, i.e. vertex subsets containing no edges): X = {x {0, 1} n : x i + x j 1 [i, j] E}. The objective is to maximize the weight of the selected subset: max z(x) = i V w i x i.
Example A B C D E F G H A B C D E F G x = (0, 1, 1, 0, 0, 1, 1) X z(x ) = 4 H A B C D E F G x = (1, 0, 0, 1, 1, 0, 0) X z(x ) = 3 H
Relationships between problems (1) Every instance of the MCP is equivalent to an instance of the MISP defined on the complementary graph. A B C A B C D E F G D E F G H H A B C A B C D E F G D E F G H H
Relationships between problems (2) The VCP and the SCP are also linked: every VCP instance can be translated into a SCP instance: each edge of the graph in the VCP corresponds to a row of the matrix in the SCP; each vertex of the graph in the VCP corresponds to a column of the matrix in the SCP; if and only if an edge e is incident to a vertex v, then a ev = 1 (two entries are equal to 1 in each row). The optimal solution of the SCP corresponds to the optimal solution of the VCP.
A B C D E F G H A B C D E F G H (A, D) 1 0 0 1 0 0 0 0 (A, E) 1 0 0 0 1 0 0 0 (B, C) 0 1 1 0 0 0 0 0 (B, F) 0 1 0 0 0 1 0 0 (B, G) 0 1 0 0 0 0 1 0 (C, F) 0 0 1 0 0 1 0 0 (C, G) 0 0 1 0 0 0 1 0 (D, E) 0 0 0 1 1 0 0 0 (D, H) 0 0 0 1 0 0 0 1 (F, G) 0 0 0 0 0 1 1 0 (F, H) 0 0 0 0 0 1 0 1 The converse is not possible in general.
Relationships between problems (3) The BPP and the PMSP are also equivalent, but the correspondence is more complex: jobs in the PMSP correspond to items in the BPP; machines in the PMSP correspond to bins in the BPP, but in the BPP the capacity is given and the number of bins is minimized; in the PMSP the number of machines is given and the completion time is minimized. To find the minimum number of bins of the BPP 1. set a tentative value k; 2. define the corresponding instance of the PMSP (with k machines); 3. find the minimum completion time t; if t is larger than the bin capacity of the BPP, increase k and repeat; if t is not larger than the bin capacity of the BPP, decrease k and repeat.
M1 L1 M1 L1 L5 M2 L2 L3 M2 L2 L3 L4 M3 L6 M3 L6 M4 L4 L5 95 80 The inverse procedure is also possible. The two problems are equivalent but the transformation implies that one of them be solved repeatedly.
Graph optimization problems: the (Asymmetric) Traveling Salesman Problem (ATSP) Given a digraph with costs on the arcs, find a minimum cost Hamiltonian circuit. We are given: a digraph G = (N, A); a cost function c : A N The ground set is the arc set A (we use a binary variable per each arc). The feasible region contains the Hamiltonian circuits How can we describe such a feasible region? How can we modify a feasible solution into another feasible solution? Is it always possible to find a Hamiltonian circuit? The objective is to minimize the total cost of the selected arcs: min z(x) = c ij x ij (i,j) A
Example 6 7 8 2 3 4 5 1 6 7 8 2 3 4 5 1 C = { (1, 4),(4, 5),(5, 8),(8, 7), (7, 6),(6, 2),(2, 3),(3, 1) } X z(x ) = 102 6 7 8 2 3 4 5 1 C = { (4, 5),(5, 8),(8, 7),(7, 4), (1, 2),(2, 3),(3, 6),(6, 1) } / X z(x ) = 106
Graph optimization problems: the (Asymmetric) Vehicle Routing Problem (AVRP) Given a digraph with costs on the arcs, a depot node, a demand for each node and a capacity, find a minimum cost subset of circuits such that they cover all the nodes, they all include the depot and the total demand in each of them does not exceed the capacity. We are given: a digraph G = (N, A), a depot node d N, a cost function c : A N, a demand function w : N N, a capacity W N.
The ground set E can be the arc set A; the set of possible pairs (node,circuit). The feasible region could contain all subsets of arcs satisfying the constraints would require to visit a graph.) (The feasibility test all partitions of the non-depot nodes into subsets of limited weight (demand) that can be visited along a circuit also including the depot (Difficult sub-problem (ATSP)!) The objective is to minimize the total cost of the selected arcs: min z(x) = c j x ij (i,j) A This is the expression of z with A as a ground set.
Example 6 7 8 2 3 4 5 d = 1 The solutions could be represented as: 6 7 8 2 3 4 5 d = 1 subsets of arcs S = { (d, 2),(2, 3),(3, 6),(6, d),(d, 4), (4, 5),(5, 8),(8, 7),(7, d) } X partitions of nodes S = { (2, 1),(3, 1),(6, 1), (4, 2),(5, 2),(7, 2),(8, 2) } X In both cases z(x) = 133.
Combining alternative formulations The AVRP (as many other combinatorial optimization problems) exhibits an important feature: different definitions of the ground set are preferable for different purposes. When the ground set is the arc set it is easy to evaluate the objective function; it is difficult to test the feasibility; When the ground set is made by the (node,circuit) pairs it is easy to test the feasibility; it is difficult to evaluate the objective. The same holds when we update/modify a solution. Which formulation should we adopt? The one that makes the most frequent operations more efficient. Both of them, with the additional task of keeping them consistent.