Conic optimization under combinatorial sparsity constraints

Transcription

1 Conic optimization under combinatorial sparsity constraints Christoph Buchheim and Emiliano Traversi Abstract We present a heuristic approach for conic optimization problems containing sparsity constraints. The latter can be cardinality constraints, but our approach also covers more complex constraints on the support of the solution. For the special case that the support is required to belong to a matroid, we propose an exchange heuristic adapting the support in every iteration. The entering non-zero is determined by considering the dual of the given conic problem where the variables not belonging to the current support are fixed to zero. While this algorithm is purely heuristic, we show experimentally that it often finds solutions very close to the optimal ones in the case of the cardinality-constrained knapsack problem and for mean-variance optimization problems. 1 Introduction Consider a conic optimization problem in standard form: min c x s.t. Ax = b x K. (1) Here, A R m n is a matrix, c R n and b R m are vectors, and K R n is a closed convex cone. We aim at finding sparse solutions of Problem (1), where the notion of sparsity is given by a combinatorial set T {0, 1} n containing the feasible non-zero sets. In the following, we assume w.l.o.g. that T is hereditary, i.e., if t T and t t, we also have t T. Throughout this paper, we only assume that T is accessible via a linear optimization oracle. We are particularly interested in the case where T is a matroid, which directly generalizes the case of an ordinary cardinality contraint of the type x 0 k. This work was partially supported by the German Research Foundation (DFG) under grant no. BU 2313/4. Fakultät für Mathematik, TU Dortmund, Vogelpothsweg 87, Dortmund, Germany, christoph.buchheim@tu-dortmund.de LIPN, Université Paris 13, 99 Avenue Jean-Baptiste Clément, Villetaneuse, France, emiliano.traversi@lipn.univ-paris13.fr 1

2 The problem addressed can thus be written as min c x s.t. Ax = b x K x i = 0 if t i = 0 t T. (2) Note that we optimize over both x and t here. Not surprisingly, Problem (2) is NP-hard. This has been shown by Farias and Nemhauser [4] in the special case that (1) is a continuous knapsack problem and T is a uniform matroid, i.e., we search for cardinality constrained solutions. The second-last constraint in (2) can be replaced by complementarity constraints x i(1 t i) = 0 for i = 1,..., n. Problem (2) is equivalent to finding a t T that minimizes c t := min c x s.t. Ax = b x K x i = 0 if t i = 0. (3) We can thus rewrite (2) as a combinatorial optimization problem over T with a non-linear objective function: min t T c t (4) Our proposed heuristic for solving Problem (2) tries to solve (4) by iteratively adapting the sparsity pattern t. In the general case, we propose to compute a new pattern t T by optimizing a certain function over T, where the coefficient for each t i set to zero in the last solution is calculated based on the dual solution of (3). In the case that T is a matroid, we can also consider iterations where only one pair of entries in t is swapped. More precisely, we first decide which fixing to release and then we fix another variable. Apart from the case where T models a cardinality constraint, our approach is motivated by another application: we are interested in finding a symmetric matrix X such that Q X for a given symmetric matrix Q and such that the nonzero entries in X form a forest in the complete graph indexed by the rows (or columns) of X. Given such a matrix X, one can compute an underestimator of the quadratic binary optimization problem min x Qx s.t. x {0, 1} n (5) by replacing Q by X and then solving the resulting sparse problem, which can be done in polynomial time. The optimal value of the latter problem is then a lower bound for the original problem, which can be used to strengthen the bounds obtained from separable underestimators within the branch-and-bound algorithm presented in [2]. This paper is organized as follows. In Section 2, we give some theoretical background motivating our approach. The primal heuristic is devised 2

3 in Section 3, both for the matroid case and for the general case. Finally, the results of an experimental evaluation are presented in Section 4. They show that the heuristic is capable of finding near-optimal solutions in very short running time for various types of problems, where we consider three different underlying cones: the non-negative orthant, the cone of semidefinite matrices, and the second-order cone. Section 5 concludes. 2 Motivation and preliminaries The crucial step in our algorithm is to decide which fixings x i = 0 should be relaxed in the next iteration. For this, we use dual information. The idea is that the dual variable for the constraint x i = 0 tells us how promising it is to relax the constraint. In the following, let c t denote the restriction of c to the components i with t i = 1 and A t the restriction of A to the columns i with t i = 1. Moreover, let K t denote the intersection of K with the subspace R t, where the latter is given by the dimensions i with t i = 1, and K t the projection of K to R t. It is easy to verify that both K t and K t are convex cones again. Finally, let t denote the binary complement vector of t. After eliminating the fixed variables in Problem (3), we thus obtain c t = min c t ˆx s.t. A tˆx = b ˆx K t. (6) For the following, we have to make an assumption on K and T : Assumption 1 For each t T, we have K t K t. It is easy to verify that Assumption 1 is satisfied in the following cases: (i) K = R n +, the non-negative cone, and T is arbitrary; (ii) K = S n +, the semidefinite cone, and t ii = 1 for all i T ; (iii) K = K n, the second order cone, and t 0 = 1 for all i T. The restriction in (ii) means that only off-diagonal elements may be fixed to zero, while in (iii) we are allowed to fix only the right-hand-side variables in the second order contraint x 0 (x 1,..., x n) 2. Lemma 2 Let K and T satisfy Assumption 1. Let y R m be a dual optimal solution for Problem (6) for some t T. Then (y, c t A t y ) is a dual optimal solution for Problem (3). Proof: Define d := c t A t y and z := c A y I td. As y is dual feasible for (6), we have z t = c t A t y (K t). Moreover, z t = c t A t y d = 0. Hence z (K t) 0 = (K t R t ) (K t R t ) K, and we obtain that (y, d) is dual feasible for (3), with objective value b y = c t. This result shows that optimal values for the dual variables corresponding to the fixings can be easily computed from a dual solution of the lower-dimensional problem (6). However, a direct application of this approach does not work, as the optimal dual solution of Problem (3) is not unique in general, even if Problem (6) has a unique dual optimal solution. This is always the case for linear programs: 3

4 Lemma 3 Let t T be given and let K = R n +. Let y R m be a dual optimal solution of Problem (6). Then (y, d) is a dual optimal solution of Problem (3) if and only if d c t A t y (K ) t, i.e., if and only if d c t A t y. Proof: First, let (y, d) be dual optimal for Problem (3). Then dual feasibility means c A y I td K and hence c t A t y d = (c A y I td) t (K ) t, which implies d c t A t y (K ) t. Conversely, let d c t A t y (K ) t and define z = c A y I td. As y is dual feasible for (3), we have z t = c t A t y (K t) = R t +. Moreover, we obtain z t = c t A t y d (K ) t = R t +. Thus z 0 and hence z K. Now the rest of the proof is analogous to the proof of Lemma 2. However, it is still a natural idea to choose d := c t A t y, since this is what we obtain if we consider the limits of the dual variables of the problem min c x s.t. Ax = b x K x i ε if t i = 0 for ε 0. Apart from the advantage of uniqueness (provided that (6) has a unique optimal dual solution), it is reasonable to consider the dual variables in the limit ε 0 instead of the dual variables for ε = 0, as we are interested in how much improvement we can obtain when moving x i to a non-zero value. 3 Primal heuristic In order to describe our heuristic algorithm for solving Problem (2), we need to specify the choice of an initial solution and the update rule for the support. These topics are discussed in the following subsections. 3.1 Initial solution A simple method to find an initial solution t (0) for our heuristic would be to start with t i = 1 for all i and then greedily remove the index l := argmin l:tl =1c l x l until t T, where x is the solution of (3) for the current t. In our implementation, we use the following alternative, which is again based on the idea of using dual variables: we first solve Problem (3) with t = 0, i.e., we fix all variables to zero, thus obtaining dual multipliers d j for all variables. Then we choose t (0) so as to maximize d t over T, which can be done by calling the linear optimization oracle for T that we assume to have at hand. Compared to the first method, this approach has the advantage that the solver for (3) has to be called only once. Moreover, it is consistent with the following iterations, in which the choice of non-zeroes is based on the value of d. 4

5 3.2 Update rules For the update of t (i) to t (i+1), we distinguish between the matroid and the general case. For general T, we proceed as follows: 1. set i := 0 and choose t (0) T 2. compute c as in (3) and obtain an optimal solution x and dual t (i) multipliers d j for the constraints x j = 0 (for t (i) j = 0) 3. solve the combinatorial optimization problem ( max t T t (i) j =0 d j t j + ) t (i) j =1 cjx j t j using the oracle, let t (i+1) be the resulting optimizer 4. if t (i+1) t (k) for all k i, set i := i + 1 and go to 2 Note that, depending on the underlying cone K and the optimization approach used to solve (3), one could compute c in Step 2 by reoptimization, since usually only few fixings are updated. t (i) In the case that T is a matroid, we propose the following alternative method, which turns out to outperform the previous approach significantly; see Section 4: 1. set i := 0 and choose t (0) T 2. compute c t (i) as in (3) and obtain an optimal solution x and dual multipliers d j for the constraints x j = 0 (for t (i) j = 0) 3. find k := argmax k:t (i) k =0 d k 4. find l := argmin l:t (i) + k l T c lx l 5. set t (i+1) := t (i) + k l 6. if t (i+1) t (k) for all k i, set i := i + 1 and go to 2 The assumption that T is a matroid guarantees that there exists at least one candidate for l different from k in Step 4. For comparison, still in the matroid case, we also consider the straightforward 2-opt update in our experiments. Here, we enumerate all pairs k, l such that t (i) + k l T and choose the one leading to the smallest value c in the next iteration. t (i+1) 4 Experimental results In this section, we provide an experimental evaluation of our approach based on a wide class of test instances from different problem types. More precisely, we test our approach on the following sparse conic optimization problems: The Cardinality Constrained Continuous Knapsack Problem (CCKP) The Sparse Mean-Variance Optimization Problem (MVO) The Tree Underestimator Problem for Binary Quadratic Problems (TUP) 5

6 For each problem, we provide a brief description, a mathematical model in the form of (2), and the test-bed used in the experiments. The aim of this section is two-fold: (i) First of all, we would like to motivate the introduction of the proposed concept of sparsity. The TUP uses the concept of sparsity presented in this paper to generalize the methodology proposed in [2] to solve efficiently combinatorial problems with a quadratic objective function. In Section 4.1 we show how the application of TUD allows to improve the dual bounds obtained in the approach presented in [2]. (ii) Secondly, in the tests concerning the CCKP and the MVO, we show that our heuristic procedure is able to provide almost-optimal solutions within a short amount of time. For assessing the performance of our algorithm, we use CPLEX 12.6 [3] with an optimality tolerance of The second-last constraint in Problem (2) is modeled as a special ordered set (SOS) constraint. For CCKP and MVO, we also use CPLEX in order to solve Problem (3) as needed in Step 2 of the algorithm in the former case, we call the LP solver, while in the second case we need the SOCP solver of CPLEX. In both cases, dual solutions are also provided by CPLEX. For TUP, we use the SDP solver CSDP [1]. All experiments were carried out on Intel Xeon processors running at 2.50 GHz. 4.1 Tree underestimator problem (TUP) Problem Description The TUP is a generalization of the bounding procedure proposed in [2], where the authors devise a branch-and-bound framework to solve combinatorial problems with a quadratic objective function. For completeness, we recall the basic idea in the following, refering the reader to [2] for a detailed description of the approach. In order to be consistent with the rest of this paper, we use a different notation here. The goal is to provide dual bounds for problems of the following shape: min f(z) = z Z z Qz + L z, where Q R n n is a symmetric (not necessarily positive semidefinite) matrix, L R n is a vector, and Z {0, 1} n. More precisely, the idea is to focus on combinatorial optimization problems where the linear counterpart min z Z can be solved efficiently for any vector c R n. For a given vector x R n, let Diag(x) be the quadratic matrix with all entries equal to zero, except for the diagonal, which contains x. For a given objective function f(z) = z Qz + L z, the separable function c z g x(z) = z Diag(x)z + L z = n n x izi 2 + L iz i is a valid underestimator of f provided that Q Diag(x), i.e., that the matrix Q Diag(x) is positive semidefinite. i=1 i=1 6

7 It is possible to identify a good (non-convex) separable underestimator for f(z) by solving the following semidefinite optimization problem: max{1 x : Q Diag(x)}. (7) The idea behind the proposed underestimator method is that minimizing the separable function g x(z) is equivalent to minimizing x z + L z, since the variables z are binary. As already mentioned, the remaining combinatorial optimization problem can be efficiently solved, and hence its computation can be incorporated into a branch-and-bound framework. The computation of the underestimator according to (7) is done in a preprocessing phase via solving a series of semidefinite programs. A generalization of the method proposed in [2] can be obtained by allowing non-zero entries in some of the off-diagonal terms in the quadratic underestimator. It is easy to see that the resulting problem min z Z f(z) = z Xz + L z can still be solved efficiently if the non-zero entries of X correspond to a forest in the complete graph K n on n vertices, the latter corresponding to the rows and columns of Q. Choosing T correspondingly, the problem of finding such a matrix X leading to a good underestimator is then of type (2). Conic formulation The TUP can be formulated as follows: max J n, X Q X X ij = 0 if t ij = 0, i j t T, where Q R n n is a symmetric matrix, J n R n n is the all-ones matrix, and T is the set of incidence vectors of spanning forests in K n. Note that Assumption 1 holds in (8), since we never restrict any diagonal entry of X to be zero. As T is a matroid here, we can use the update rule based on pairwise exchanges; see Section 3.2. Test-bed used Our test-bed consists of the matrices Q used in the binary quadratic instances of the Biq-mac library; see [5]. The linear part is always zero. There are 165 instances in total, subdivided into three problem classes beasley, be, and gka, with a number of variables ranging from 20 to 500. Analysis of the results Our aim is to compare the dual bounds obtained from the new tree underestimators with the ones obtained from separable underestimators as in [2]. The results are given in Table 1. The instances are grouped horizontally according to their family (type) and number of variables (size). For each group of instances, we state the number of corresponding instances (# inst) as well as the following average information: the total cpu time (in seconds) needed by our algorithm to terminate (time), which is mostly spend for iteratively solving (8) 7

8 type size # inst time # iter sep bnd tree bnd beasley beasley beasley beasley be be be be be gka gka gka gka gka gka gka gka gka gka gka gka Table 1: Results for TUP the semidefinite problem (8) for fixed t, the number of iterations (# iter), and the dual bound provided by the separable underestimator and by the tree underestimator (sep bnd and tree bnd). The results clearly show that the use of the tree underestimator improves the quality of the dual bounds significantly. The running times are small enough to be clearly dominated by the time needed for the main phase of the approach presented in [2]. 4.2 Cardinality constrained continuous knapsack problem (CCKP) Problem Description The CCKP is a continuous (m-dimensional) knapsack problem where at most k variables are allowed to be strictly greater than zero, for some given integer k. The CCKP has been introduced and investigated in [4]. Conic formulation max The CCKP can be formulated as follows: c x a j x b j j = 1,..., m x i = 0 if t i 0 t T, (9) 8

9 where c, a j R n + and T is the set of binary vectors containing at most k ones. In other words, T is the uniform matroid, so that we can again use all update rules presented in Section 4. Test-bed used We use two sets of instances: De Farias et al. instances. These are the instances used in [4]. They contain up to 8000 variables (n) and 70 knapsack constraints (m), the values of k and the densities of the non-zero entries of the knapsack constraints (d) were chosen with the objective to make the instances hard from a computational point of view. Random instances. We produced two additional sets of random instances, one set of small instances (n up to 300 and m up to 50) and another set of large instances (n up to 8000 and m up to 5000). In both sets, the knapsack constraints have a density d of 100%, while various cardinalities k are considered, all in the (small) range where the resulting problems turn out to be non-trivial. Whenever showing results for random instances, we state averages over 10 instances in each line of the tables. Analysis of the results We first use Table 2 to compare the performances of three different update rules of our algorithm, using the small instances: (1) the matroid exchange, (2) the general exchange, and (3) the optimal pairwise exchange 2-opt. For each option 1 3, we show the best primal bound obtained (best), the computation time in seconds (time), and the number of iterations (iter). The values of the primal bounds are normalized by dividing them by the best one. As expected, the optimal pairwise exchange provides the best primal solution for each instance. However, it is several orders of magnitude slower in terms of computing time. For this reason, the optimal pairwise exchange is not suitable if we want to tackle bigger instances. Moreover, the difference in the solution quality is very small: it is below 1 % for the matroid exchange and below 4 % for the general exchange. When comparing the general exchange rule with the matroid exchange, it turns out that the matroid rule yields better (or equally good) results on all instances, while using roughly the same number of iterations and the same running time on average. This trend is confirmed by the results for large random instances presented in Table 3, but here the matroid exchange rule even outperforms the general exchange rule in terms of running time. For this reason, we use the matroid exchange rule in the following experiments. Note that the running time of all three heuristics is dominated by the time needed to solve (8) for fixed t using CPLEX. In Table 4 we present the results concerning the de Farias et al. instances. We compare the performance of our heuristic with the performance of the commercial solver CPLEX. The table first presents the time, the primal bound, and the number of iteration of the heuristic (pheur, theur and itheur), and then the best primal and dual bounds and the 9

10 n m k best 1 time 1 iter 1 best 2 time 2 iter 2 best 3 time 3 iter Table 2: Results for small random instances of CCKP, comparing matroid exchange, general exchange, and 2-opt heuristics 10

11 n m k best 1 time 1 iter 1 best 2 time 2 iter Table 3: Results for large random instances of CCKP, comparing matroid exchange and general exchange heuristics 11

12 m n k d pheur theur itheur pcplex dcplex tcplex TL TL TL TL TL Table 4: De Farias et al. instances, matroid heuristic vs. CPLEX computing time needed by CPLEX (pcplex, dcplex, and tcplex). We set a time limit of one cpu hour. The results show that the heuristic in few iterations and seconds provides an optimal solution 5 times out of 16 and in the remaining instances the solution provided is less than 0.3% worse than the optimal solution. To assess the solution quality of our heuristic on a larger set of instances, we again consider random instances and solve them to optimality by CPLEX, now without a time limit; see Table 5. Here, the value %gap states how far our heuristic solution is from optimality. While some of the instances are very hard to solve by CPLEX, our heuristic can always find a solution within 0.88 % from optimality in a running time that never exceeds 0.04 seconds for n = 200 and 0.12 seconds for n = Sparse mean-variance optimization Problem Description The CCKP discussed above plays an important role in portfolio optimization, where the aim is to choose a set of investments that does not exceed the budget of the investor and that is sparse, i.e., only a certain number of different assets may be chosen. However, in such problems, one is usually not only interested in the expected return of the investments, but also in the risk involved, which can be measured by the variance. Conic formulation max One way to model such problems is as follows: c x ε x Q 1 x a j x b j j = 1,..., m x i = 0 if t i 0 t T, 12

13 n m k %gap theur itheur tcplex itcplex Table 5: Random instances of CCKP, comparison with CPLEX 13

14 where c, a j R n + and T are as in CCKP, but now the objective contains the risk term x Qx defined by the covariance matrix Q, weighted by some ε > 0. As in CCKP, the set T is the uniform matroid, but now the risk term can be modeled using the second-order cone K n in a standard way. In the resulting conic problem, Assumption 1 is satisfied again. Test-bed used As test-bed, we use exactly the same instances as in the CCKP case. For the risk term, we set ε = 1 and compute Q 1 randomly as follows: we first compute a matrix A with all entries chosen uniformly at random in [0,1] and then set Q = (AA ) 1. Analysis of the results In Tables 6 8, we repeat the experiments of Tables 2, 3, and 5, now containing the risk part in the objective function. However, since the problem is harder to solve now, we use smaller instances. Altogether, it turns out that the results for our heuristic are similar to the CCKP case. 5 Conclusion We presented a heuristic for a very general class of sparse conic optimization problems, based on the idea of iteratively updating the support of the solution. In the case that the sparsity is defined by a matroid, we devise an exchange heuristic for updating the support that is based on dual information. Our experiments show that this approach often yields very good solutions in a short running time for different types of problems, including the cardinality-constrained continuous knapsack problem. References [1] Brian Borchers. CSDP, a C library for semidefinite programming. Optimization Methods and Software, 11/12(1 4): , [2] Christoph Buchheim and Emiliano Traversi. Quadratic combinatorial optimization using separable underestimators. INFORMS Journal on Computing, To appear. [3] IBM ILOG CPLEX Optimizer [4] Ismael R. de Farias Jr. and George L. Nemhauser. A polyhedral study of the cardinality constrained knapsack problem. Mathematical Programming, 96: , [5] Franz Rendl, Giovanni Rinaldi, and Angelika Wiegele. Solving Max- Cut to optimality by intersecting semidefinite and polyhedral relaxations. Mathematical Programming, 121(2):307,

15 n m k best 1 time 1 iter 1 best 2 time 2 iter 2 best 3 time 3 iter Table 6: Results for small random instances of MVO, comparing matroid exchange, general exchange, and 2-opt heuristics 15

16 n m k best 1 time 1 iter 1 best 2 time 2 iter Table 7: Results for large random instances of MVO, comparing matroid exchange and general exchange heuristics 16

17 n m k %gap theur itheur tcplex itcplex Table 8: Random instances of MVO, comparison with CPLEX 17