A Binarisation Approach to Non-Convex Quadratically Constrained Quadratic Programs

A Binarisation Approach to Non-Convex Quadratically Constrained Quadratic Programs Laura Galli Adam N. Letchford February 2015 Abstract The global optimisation of non-convex quadratically constrained quadratic programs is a notoriously difficult problem, being not only N P-hard in the strong sense, but also very difficult in practice. We present a new heuristic approach to this problem, which enables one to obtain solutions of good quality in reasonable computing times. The heuristic consists of four phases: binarisation, convexification, branchand-bound and local optimisation. Computational results, on boxconstrained and point packing instances, are encouraging. Keywords: quadratically constrained quadratic programming, mixedinteger nonlinear programming, heuristics. 1 Introduction A quadratically constrained quadratic program (QCQP) is an optimisation problem that can be written in the following form: inf x T Q 0 x + c 0 x (1) s.t. x T Q k x + c k x h k (k = 1,..., m) (2) x R n, (3) where the Q k are symmetric matrices of order n, the c k are n-vectors and the h k are scalars. (We write inf rather than min because it is possible that the infimum is not attainable.) If all of the Q k are positive semidefinite (psd) matrices, then QCQPs are convex optimization problems, and therefore can be solved efficiently (see, e.g., [10]). Non-convex QCQPs, on the other hand, are N P-hard. In Dipartimento di Informatica, Università di Pisa, Largo B. Pontecorvo 3, 56127 Pisa, Italy. E-mail: laura.galli@unipi.it Department of Management Science, Lancaster University, Lancaster LA1 4YX, United Kingdom. E-mail: A.N.Letchford@lancaster.ac.uk 1

fact, just checking unboundedness of a quadratic form over the non-negative orthant is N P-complete in the strong sense [28]. From now on, when we say QCQPs, we mean non-convex ones, unless otherwise stated. Solving QCQPs to proven optimality is very hard in practice as well in theory (e.g., [4, 5, 6, 23, 27]). One obvious source of this difficulty is the possibility of an exponentially large number of local minima. Another, less obvious, source of difficulty is that variables can take irrational values at the optimum, which means that one must either be content with solutions that are optimal only up to some given accuracy, or somehow represent solutions implicitly (for example, via the KKT conditions, as in [11]). On the other hand, QCQPs are a powerful modelling tool. A folklore result (see, e.g., [22, 29]) is that mixed 0-1 linear and quadratic programs can be modelled as QCQPs, since the condition x i {0, 1} is equivalent to the quadratic constraint x i = x 2 i. Also, Shor [36] pointed out that any optimisation problem in which the objective and constraint functions are polynomials can be transformed to a QCQP, by adding suitable variables and constraints. (For example, one can replace a cubic term x i x j x k with the quadratic term x i x jk, where x jk is a new variable, if one also adds the quadratic equation x jk = x j x j.) Several other applications of QCQPs can be found in, e.g., [2, 14, 24, 30]. In this paper, we present a new heuristic for QCQPs, which is aimed mainly at loosely-constrained QCQPs, where it is fairly easy to find a feasible solution. The heuristic exploits the fact that there now exist very good software packages for solving convex QCQPs with binary variables. It consists of four phases: binarisation, convexification, branch-and-bound and local optimisation. In our computational experiments, on standard test instances, the heuristic typically found solutions of very good quality in reasonable computing times. The structure of the paper is as follows. The literature is reviewed in Section 2, the heuristic is described in Section 3, the computational results are given in Section 4, and some concluding remarks are made in Section 5. Remark: The idea of a box will appear several times in the paper. For us, it means a region of the form {x R n : l x u}, where l {R { }} n, u {R { }} n and l < u. 2 Literature Review We now review some of the literature on QCQPs. Since the literature is vast, we only review works of direct relevance. We refer the reader to Section 5 of [13] for a more detailed survey. 2

2.1 LP-based approaches Like any other global optimisation problem, QCQPs can be solved (to any fixed precision) using the spatial branch-and-bound method of McCormick [26]. Suppose that each variable is bounded, i.e., l i x i u i for all i. Then, we can construct a linear programming (LP) relaxation in the following way. For each pair i, j for which the quadratic term x i x j appears in the problem, substitute the term with a new variable X ij, and add the constraints X ij l i x j + l j x i l i l j, X ij u i x j + u j x i u i u j X ij l i x j + u j x i l i u j, X ij l j x i + u i x j l j u i. This LP relaxation can then be embedded in a branch-and-bound scheme, in which branching is performed by partitioning the domain of an x variable into two intervals. To our knowledge, the only pure spatial branch-andbound algorithm for QCQPs was that of Al-Khayyal et al. [2]. An enhanced version of spatial branch-and-bound, called branch-andreduce, uses feasibility and optimality arguments to reduce variable domains (Tawarmalani & Sahinidis [38]). Raber [30] presented a branch-and-reduce algorithm for QCQPs. He used a non-standard branching rule, in which the solution space is partitioned into simplices rather than boxes. Linderoth [23] presented another variant, in which the solution space is partitioned into Cartesian products of triangles and rectangles. He also used second orderconic relaxations in addition to linear ones. Bao et al. [6] presented a third version, in which linear over- and under-estimators are derived for certain quadratic functions with multiple terms, instead of treating each quadratic term independently. Sherali & Tuncbilek [35] presented a spatial branch-and-bound algorithm for general polynomial optimisation problems, in which LP relaxations are constructed via the Reformulation-Linearisation Technique (RLT). For nonconvex QPs, the constraints generated by the RLT are obtained by multiplying pairs of linear constraints together, and then linearising by substituting quadratic terms with the X ij variables mentioned above. These constraints include the ones of McCormick [26] as a special case. Audet et al. [5] specialised the RLT approach to QCQP instances in which there are a substantial number of linear constraints, and also showed how to tighten the relaxations using cutting planes. Saxena et al. [33, 34] showed how to strengthen the RLT relaxation further, using specialised disjunctions derived from Eigenvalue considerations. Their algorithm works also for mixed-integer QCQPs, provided the integer-constrained variables are bounded. Misener & Floudas [27] showed how to construct strong mixed-integer linear programming (MILP) relaxations of mixed-integer QCQPs, rather than just linear ones. Although one must use branch-and-bound even to 3

solve these relaxations, the advantage is that one can use the power of existing MILP algorithms, which are now highly sophisticated. 2.2 SDP-based approaches A completely different body of work deals with semidefinite programming (SDP) relaxations of QCQPs and related problems. These papers group the X ij variables in a matrix X = xx T, and introduce the augmented matrix variable Y = ( 1 x )( ) 1 T ( 1 x T = x x X Then, in addition to the linear constraints of McCormick [26], one can add the constraint that Y must be psd. The SDP relaxation originated in Shor [36], but was developed by Ramana [31] and Fujie & Kojima [16]. Burer & Vandenbussche [11] presented a finite exact algorithm for nonconvex QP (rather than QCQP) that uses SDP relaxations together with a specialised branching rule based on complementarity constraints. Luo et al. [24] explored SDP relaxations of QCQP instances arising in signal processing, and propose a rounding heuristic. Bao et al. [7] discussed the doubly-non-negative (DNN) relaxation, which strengthens the SDP relaxation by imposing non-negativity on the off-diagonal elements of the matrix X. Burer & Chen [12] used the DNN relaxation to improve the exact algorithm presented in [11]. Anstreicher [3] considered relaxations for QCQPs that include box constraints, and shows that a combination of SDP and RLT is far superior than either method alone. (Note that the combined SDP / RLT relaxation is stronger than the DNN relaxation.) Still stronger SDP relaxations were presented in Zheng et al. [41]. Another method, called difference-of-convex (DC) programming, expresses each quadratic objective and constraint function as the difference between a convex quadratic function and a concave one (e.g., Bomze & Locatelli [9], Zheng et al. [40]). This enables one to compute bounds by using existing bounding techniques for convex and concave problems. Recently, however, Anstreicher [4] showed that the combined SDP / RLT relaxation dominates all of the standard DC relaxations. 2.3 The binary case We now recall a few key facts about QCQPs in which all variables are required to be binary (0-1 QCQPs). As mentioned in the introduction, they can be viewed as a special kind of QCQP. Clearly, 0-1 QCQPs are N P-hard in the strong sense, since the same is true even for the less general 0-1 LP. On the hand, they are easier than general QCQPs in two senses. First, the issue of irrational numbers and ). 4

limited precision does not arise, which makes finite termination easy to achieve. Second, non-convex instances can be easily convexified, without adding any new variables, using the identities x 2 i x i = 0 for all i. Hammer & Rubin [19] suggested to simply make each matrix Q k psd, by subtracting the minimum Eigenvalue from all diagonal entries, and adjusting the linear term accordingly. A more sophisticated convexification approach, for 0-1 QPs only, was presented in [8]. We extended that approach to 0-1 QCQPs in [17]. The lower bound obtained by solving the continuous relaxation of the resulting convexified 0-1 QCQP is as strong as the standard SDP bound, which is described in, e.g., [22, 29, 32]. We will also need a classical result of Fortet [15], which states that 0-1 QCQPs can be easily converted into 0-1 LPs, at the cost of adding the X ij variables mentioned above. Indeed, in this case, the McCormick inequalities take the much simpler form X ij 0, X ij x i, X ij x j and X ij x i +x j 1. Then, when x i and x j are forced to take binary values, X ij equals x i x j. Some alternative strategies for linearising quadratic problems with 0-1 variables are surveyed in Section 5 of [13]. We do not go into details here, for the sake of brevity. 3 Our Approach We now describe our approach. As mentioned in the introduction, it consists of four phases. These phases are covered in the following four subsections. 3.1 Binarisation By binarisation, we mean the construction of a 0-1 QCQP that approximates the original QCQP, in the following sense: any feasible solution to the 0-1 QCQP can be easily mapped to a feasible solution of the original QCQP with the same cost. Suppose initially that the QCQP contains explicit bounds of the form l x u, where the components of l and u are finite. That is, x is constrained to lie within a finite box. Using the simple scaling x i (x i l i )/(u i l i ) for all i, we can assume that x [0, 1] n. (If u i l i = 0 for some i, then x i is a constant and can be eliminated from the problem.) The most obvious binarisation strategy is then to select a positive integer p, and use only p bits of precision when assigning values to the x variables. The 0-1 QCQP can then be constructed as follows. For i = 1,..., n and for k = 1... p, introduce a binary variable z ik, equal to the kth bit of the binary representation of x i. Then eliminate the x variables from (1) (3) using the identities: p x i = 2 k z ik (i = 1,..., n). k=1 5

The resulting 0-1 QCQP has np variables. Note however that using the standard bit-representation gives a lopsided approximation to the original problem, in the sense that the x variables can take the value 0 but not the value 1. We prefer to use a representation that is symmetrical over the variable domains. The easiest way to do this is to impose the restriction that each x variable must take on a value that is a multiple of 1/(2 p 1). One can then use the modified identities: x i = 1 2 p 1 p 2 p k z ik (i = 1,..., n). (4) k=1 We call this binarisation strategy 1. Now, suppose that x is an optimal solution to the original QCQP. If we use binarisation strategy 1, then there exists a feasible solution x in our restricted set for which x i x i 1 2(2 p 1) value 1 2(2 p 1) for all i. Informally speaking, the can be thought of as the maximum error, even if, in general, there is no guarantee that the cost of x will be close to that of x. In order to reduce the maximum error, but with the same number of bits, one can impose instead the restriction that each x variable must take on a value that is equal to 1/2 p+1 plus a multiple of 1/2 p. To do this, it suffices to use the following modified identities: x i = 1 p 2 p+1 + 2 k z ik (i = 1,..., n). (5) k=1 We call this binarisation strategy 2. With this strategy, the maximum error drops to 2 (p+1). It is of course not obvious how to select the right binarisation strategy, or the appropriate value of p, for a given 0-1 QCQP instance. For this reason, we consider several combinations in our computational experiments (see Section 4). The 0-1 QCQP obtained via either binarisation strategy can be written in the form: min z T Q0 z + c 0 z s.t. z T Qj z + c j z h j (j = 1,..., m) z {0, 1} np, where Q j and c j indicate that the corresponding coefficients must be computed according to (4) or (5). Now consider the case in which the QCQP does not have explicit finite lower and upper bounds. In most cases of practical interest, one can easily derive finite bounds using one s knowledge of the problem under consideration. If those bounds are rather loose, then one can compute tighter bounds 6

by solving a series of LPs or SDPs (two per variable), or by using any of the domain reduction techniques that are used in modern branch-and-reduce algorithms (see Subsection 2.1). 3.2 Convexification Now that we have a 0-1 QCQP that approximates the original QCQP, the next step is to make it convex. As mentioned in Subsection 2.3, there are two main options available. The first is to perturb the diagonals of the Q j in order to make them psd, as done in [17]. The second is convert the 0-1 QCQP into a 0-1 LP, using the Fortet construction. Observe that the first option leads to a convex 0-1 QCQP with np variables and m constraints, whereas the second leads to a 0-1 LP with O(n 2 p 2 ) variables and m+o(n 2 p 2 ) constraints. It is actually not obvious how to select between these two convexification options a priori. There is no dominance relation in general between the lower bounds obtained by solving the continuous relaxations of the two resulting problems. The first option yields a model with far fewer variables and constraints, but the existing software packages for solving 0-1 LPs are far more sophisticated than those for solving 0-1 QCQPs. Moreover, the Fortet constraints, though numerous, are extremely sparse. On the other hand, they can cause the LP relaxations to suffer from degeneracy. For these reasons, we consider both options in our computational experiments (again, see Section 4). 3.3 Branch-and-bound At this point, we have a convex 0-1 QCQP or 0-1 LP that approximates the original QCQP. The next step is to solve it. For the 0-1 LP, things are straightforward: we can just solve it with the classical branch-and-bound approach, using the simplex method to solve the LP relaxations. Rather than coding a branch-and-bound solver of our own, we decided to use the mixed-integer LP solver that is available in the IBM software package CPLEX (version 12.6). We remark that CPLEX has its own internal routines for, e.g., fixing variables, generating cutting planes, producing heuristic solutions, and selecting the variable on which to branch. We decided to use default settings, rather than experimenting with the (many) parameters available in CPLEX. For the convex 0-1 QCQP formulation, things are more complicated. For one thing, a variety of generic exact algorithms are available for convex mixed-integer nonlinear programs, such as outer approximation, generalised Benders decomposition, the extended cutting-plane method, and LP/NLPbased branch-and-bound (see the survey [18]). Moreover, we also have the option of converting the convex 0-1 QCQP into a 0-1 second order cone 7

program (SOCP). The motivation for doing this is that the SOCP relaxations can be solved quite quickly via interior-point methods (see, e.g., [1]). For ease of implementation, we decided to use the mixed-integer SOCP solver that is available in CPLEX. That solver provides two algorithmic options. Although the CPLEX manual does not give full details, our understanding is that the first option is plain branch-and-bound, in which the SOCP relaxations are solved via the Barrier algorithm; whereas the second option is a branch-and-cut scheme, in which the SOCP relaxations are solved only approximately via Kelley s cutting-plane algorithm [21]. Again, we just use the default parameters in both cases. 3.4 Local optimisation If the original QCQP is tightly constrained, or we use too few bits, there is a chance that the binarised and convexified problem is infeasible. In that case, our heuristic fails. Otherwise, the optimal solution z {0, 1} np to the latter problem can be converted, using the bit representation described in Subsection 3.1, into a feasible solution x R n to the original QCQP. Observe that x is not guaranteed even to be a local minimum for the QCQP. Our fourth and final phase is therefore to move from x to a nearby local minimum. To do this, we use the open-source non-linear programming solver IPOPT [20]. As we understand it, IPOPT is a primal-dual interiorpoint solver with a logarithmic barrier function, in which a line-search is conducted in each Newton iteration. As in the case of CPLEX, we use default parameter settings. 4 Computational Results In order to ascertain the potential of our algorithm, we ran experiments on two different problems: non-convex quadratic programs with box-constraints and point packing problems. These problems and experiments are described in the following two subsections. In both experiments, we used the callable library of CPLEX 12.6 to solve all 0-1 LPs and 0-1 QCQPs, and IPOPT 3.11.8 to perform the local optimisation step. The experiments were performed on a 64-bit 2.3 Ghz AMD Opteron 6376 processor with 16Gb RAM, under the Ubuntu 12.4 operating system. All programs were implemented in C++ and compiled with gcc 4.4.3. 4.1 Non-convex QP with box constraints A non-convex quadratic program with box-constraints (QPB) takes the form: max { x T Qx + c x : x [0, 1] n}, 8

Strategy 1 Strategy 2 n p = 1 p = 2 p = 3 p = 1 p = 2 p = 3 IPOPT 20 100 0.00 0.00 0.00 62.12 34.30 18.19 1.46 30 60 0.03 0.03 0.02 58.86 33.04 17.57 1.14 30 70 0.02 0.01 0.00 63.13 34.82 18.29 2.33 30 80 0.05 0.03 0.02 60.93 33.24 17.39 0.87 30 90 0.06 0.00 0.00 60.56 32.97 17.18 0.00 30 100 0.05 0.01 0.00 62.05 34.37 18.02 1.22 40 30 0.00 0.00 0.00 67.62 37.24 19.45 0.57 40 40 0.04 0.04 0.04 66.83 36.06 18.70 2.55 40 50 0.11 0.01 0.00 65.10 35.86 18.85 0.49 40 60 0.03 0.00 0.00 57.98 32.05 16.92 0.48 40 70 0.01 0.01 0.01 59.97 32.36 16.88 0.01 40 80 0.00 0.00 0.00 58.99 32.13 16.69 0.01 40 90 0.00 0.00 0.00 60.00 33.76 17.61 0.83 40 100 0.05 0.00 0.00 59.73 33.10 17.37 0.00 50 30 0.01 0.01 0.00 67.92 37.42 19.54 0.42 50 40 0.01 0.01 0.00 65.93 36.06 18.89 0.37 50 50 0.08 0.00 0.00 70.09 39.19 20.75 2.53 60 20 0.00 0.00 0.00 66.59 36.77 19.19 1.74 Table 1: Average percentage gaps for box-constrained QP instances: no local optimisation where Q Q n n and c Q n. This problem, which is strongly N P-hard, has received much attention (e.g., [3, 4, 11, 16, 39]). We used the instances described in [11, 39], which were created as follows: given a density %, each entry in Q and c is set to zero with probability (100 )%, and to a random integer between 50 and 50 with probability %. For eighteen different combinations of n and %, there are three random instances. So there are 54 instances in total. The instances and their optimal solution values were kindly given to us by Kurt Anstreicher. First, we ran our heuristic without the local optimisation step. Table 1 shows the quality of the bounds obtained with this setting. Each row corresponds to a particular combination of n and. For each of the two binarisation strategies, and for numbers of bits ranging from 1 to 3, we report the average, over the three instances of the given type, of the percentage integrality gap (i.e., the percentage gap between the profit of our heuristic solution and the optimal profit). For interest, we also include the average percentage integrality gap of solutions obtained using IPOPT alone, using five different random feasible initial solutions as starting points. We were surprised to see that, for these instances, binarisation strategy 1 performs very well, whereas strategy 2 performs extremely poorly. Not only that, but just one bit is enough to get very good solutions when strategy 1 9

Strategy 1 Strategy 2 n p = 1 p = 2 p = 3 p = 1 p = 2 p = 3 IPOPT 20 100 0.00 0.00 0.00 0.00 0.00 0.00 1.46 30 60 0.02 0.02 0.02 1.14 0.57 0.03 1.14 30 70 0.00 0.00 0.00 0.78 0.66 0.00 2.33 30 80 0.02 0.02 0.02 0.02 0.67 0.02 0.87 30 90 0.00 0.00 0.00 0.00 0.00 0.00 0.00 30 100 0.02 0.00 0.00 1.47 1.32 0.00 1.22 40 30 0.00 0.00 0.00 0.00 1.26 0.18 0.57 40 40 0.04 0.04 0.04 0.04 0.04 0.04 2.55 40 50 0.00 0.00 0.00 0.76 0.15 0.15 0.49 40 60 0.00 0.00 0.00 0.48 0.29 0.29 0.48 40 70 0.01 0.01 0.01 0.01 0.01 0.01 0.01 40 80 0.00 0.00 0.00 0.45 0.00 0.00 0.01 40 90 0.00 0.00 0.00 1.16 0.00 0.05 0.83 40 100 0.03 0.00 0.00 0.34 0.34 0.03 0.00 50 30 0.00 0.00 0.00 0.38 0.10 0.00 0.42 50 40 0.00 0.00 0.00 0.10 0.00 0.00 0.37 50 50 0.00 0.00 0.00 1.24 0.55 0.00 2.53 60 20 0.00 0.00 0.00 0.80 0.00 0.00 1.74 Table 2: Average percentage gaps for box-constrained QP instances: with local optimisation is used. Note that having only 1 bit in this case corresponds to restricting all variables to be binary. This implies that, for these instances, there exist near-optimal solutions such that no variables take fractional values. As for IPOPT, it obtains solutions that are reasonably good, but almost always worse than those obtained with strategy 1. Next, we wished to apply the local optimisation step to the solutions obtained with our heuristic. Here, we immediately faced a technicality: IPOPT needs to start from solutions that are in the strict interior of the feasible region, but many of the solutions obtained with binarisation strategy 1 lay on the boundary. To deal with this, we simply moved the initial heuristic solutions a little towards the centre of the unit hypercube, before passing them to IPOPT. (More precisely, we changed x i to 0.9x i + 0.05 for i = 1,..., n.) Table 2 shows the quality of the bounds obtained with the local optimisation step. It can be seen that the solutions obtained with binarisation strategy 1 plus local optimisation are of exceptionally good quality. The addition of local optimisation also greatly improves the quality of the solutions obtained with strategy 2, but not enough to obtain results as good as those obtained with strategy 1 (or even with IPOPT alone). Table 3 shows the time taken to solve the convex 0-1 QCQPs and the time 10

Solving 0-1 QCQP Str. Bits Simplex IPM Kelley IPOPT 1 1 207.23 2.48 105.15 0.22 2 6332.47 213.12 212.00 0.23 3 21635.89 3413.31 3421.29 0.24 2 1 111.36 103.92 103.89 0.30 2 1839.86 142.98 142.92 0.23 3 26222.92 1803.07 1814.65 0.22 Table 3: Average times (in seconds) for QPB instances for the local optimisation step, in seconds, averaged over all 54 instances, for various parameter settings. The column headings simplex, IPM and Kelley refer to the manner in which the continuous relaxations of the convex 0-1 QCQPs are solved: via the Fortet linearisation and the simplex method, via SOC reformulation and interior-point methods, or via SOC reformulation and the Kelley cutting plane method, respectively. We see that the IPM option is fastest, and that the time taken by the additional local optimisation step is negligible. In Table 4, we present more detailed running times for our preferred option, which uses binarisation strategy 1, with the IPM option to solve the 0-1 QCQPs, followed by local optimisation. We see that the running time depends heavily on the number of bits used. We remark that our heuristic compares very favourably in terms of running time with the current fastest exact algorithm for QPB, due to Burer and Vandenbussche [11]. Indeed, their exact algorithm took around 45,000 seconds for the instances with n = 50, on a machine that is only a little slower than ours. If we use only two bits, we obtain solutions that are frequently optimal, and always within 0.04% of optimal, while taking about one-hundredth of the time taken by their algorithm. 4.2 Point packing Next, we tested our algorithm on the point packing (PP) problem, which is a much-studied problem in global optimisation (see, e.g., [3, 4, 25, 37]). Given an integer n 2, the problem is to place n points in the unit square, in such a way that the minimum distance between any pair of points is maximised. Letting x i and y i denote the horizontal and vertical coordinates of point i, respectively, and letting θ denote the square of the objective function, PP 11

Number of bits n 1 2 3 IPOPT 20 100 1.17 2.17 5.42 0.18 30 60 1.68 15.20 128.39 0.25 30 70 2.40 13.77 88.28 0.25 30 80 1.83 5.70 23.28 0.28 30 90 1.56 2.86 9.45 0.23 30 100 1.57 7.10 37.88 0.30 40 30 1.91 8.73 54.18 0.30 40 40 3.51 144.41 4495.12 0.29 40 50 2.95 57.82 861.20 0.35 40 60 3.43 145.36 3666.73 0.31 40 70 0.55 3.08 22.93 0.03 40 80 0.68 3.67 30.65 0.03 40 90 0.93 5.41 51.87 0.01 40 100 0.69 6.97 74.62 0.03 50 30 6.01 314.06 6412.27 0.44 50 40 0.98 17.10 325.20 0.06 50 50 4.24 413.46 12752.16 0.05 60 20 8.54 832.35 30691.41 0.42 Table 4: More detailed average times for QPB instances can be formulated as follows: max θ (6) s.t. θ (x i x j ) 2 + (y i y j ) 2 (1 i < j n) (7) x, y [0, 1] n (8) θ R +. (9) Since this formulation has a linear objective and (non-convex) quadratic constraints, it can be tackled with our method. It turns out that PP has four peculiar properties that make it quite different from QPB: 1. For any feasible solution obtained using binarisation strategy 2, one can obtain a better feasible solution using strategy 1. Accordingly, we used only strategy 1 in our experiments. 2. For a fixed number of bits p, there are 2 2p possible point locations. Therefore, in order to obtain a packing in which no two points coincide, the number of bits must be at least 1 p = 2 log 2 n. In our experiments, we set p to 3, which is greater than p for n 16. 12

3. The binarisation of θ can cause the 0-1 QCQP solver to miss good packings. To see why, suppose that we use strategy 1 with 3 bits, together with the trivial bounds 0 θ 2. The binarisation then effectively forces θ to be a multiple of (2 0)/(2 3 1) = 2/7. Then, if an optimal packing has an objective value of 0.3 and a sub-optimal one has an objective value of 0.4, they will both be treated as if they had a value of 2/7 0.285. To address this issue, it helps to use a tighter upper bound on θ. We used the following bound, which is given in Szabó et al. [37]: 2 (n 1) 2 + 2 + 2 1 3(n 1) n 1 (n 1) 2 + 2. 3(n 1) We remark that the upper bounds obtained by Anstreicher [3, 4], using various combinations of RLT and SDP constraints, are all weaker than this one for n 4. 4. PP exhibits a high degree of symmetry. Indeed, given any feasible solution, we can rotate or reflect the unit square, or permute the points, and obtain another solution of the same profit. This symmetry makes the 0-1 QCQP hard to solve. To address this, we add the following constraints to the formulation (6) (9): n i=1 x i n/2 n i=1 y i n/2 i=1 y i i=1 x i 0 x 1 x 2... x n 1. Similar (though not identical) symmetry constraints were used, e.g., by Anstreicher [3, 4]. The best way to solve the resulting 0-1 QCQPs turned out to be to apply the Fortet linearisation and then solve the resulting 0-1 LP by the simplex method. The results that we obtained with this option are displayed in Table 5. The first column displays the number of points, n. To explain the second two columns, consider the triple (x, y, θ ) corresponding to the optimal 0-1 QCQP solution. The column headed %gap1 gives the percentage integrality gap for θ itself, whereas the column headed %gap2 gives the gap for the true (non-discretised) value of θ corresponding to the packing represented by (x, y ). The next column gives the time, in seconds, taken to solve the 0-1 QCQP. The next two columns give the gap obtained after the local optimisation, and the time taken to perform the local optimisation. The last two columns give the gaps and times obtained using IPOPT alone, averaged over 100 random starting points. 13

Initial Loc. Opt IPOPT n %gap1 %gap2 time %gap time %gap time 2 0.00 0.00 0.01 0.00 0.00 0.00 0.00 3 20.55 6.70 0.10 0.00 0.01 0.00 0.00 4 7.72 0.00 0.26 0.00 0.01 0.00 0.01 5 39.15 34.69 0.92 0.00 0.01 11.54 0.01 6 17.85 9.58 1.76 0.00 0.01 1.89 0.02 7 19.53 7.62 5.37 0.00 0.02 0.71 0.03 8 29.88 23.84 16.0 0.00 0.05 2.74 0.04 9 37.01 34.69 45.6 0.00 0.07 0.62 0.06 10 23.91 8.01 164 0.83 0.14 0.53 0.13 11 44.24 35.65 659 1.61 0.20 1.19 0.37 12 48.19 45.98 461 0.00 0.49 4.06 0.55 Table 5: Results for PP instances, with strategy 1 and three bits. The results here are quite different to those obtained for QPB. We see that the initial solutions obtained by our approach are of rather poor quality, and almost always worse than those obtained with IPOPT. On the other hand, when local optimisation is applied, we obtain solutions of extremely good quality in every case, something that IPOPT alone fails to do. As for running times, the bottleneck for our approach is again the solution of the 0-1 QCQP. 5 Conclusion Non-convex QCQPs are very hard to solve to proven optimality, in both theory and practice. We have presented a new heuristic for them, which is based on the solution of a convex 0-1 QCQP that approximates the original instance. Although convex 0-1 QCQPs are also N P-hard, they are much easier to solve in practice than non-convex QCQPs. The computational results show that our heuristic find extremely good solutions, with reasonable computing times in the case of QPB, but rather excessive running times in the case of PP. There are five potential ways to improve our heuristic that we believe would be worth considering: In order to speed up the solution of the 0-1 QCQPs, it might help to reformulate them, using the procedure described in our paper [17], mentioned in Subsection 2.3. (Unfortunately, we were unable to test this, since CPLEX appears to do some kind of internal reformulation of its own, that we were unable to deactivate.) When solving the 0-1 QCQP, it might help to use a specialised rule for selecting the z variable upon which to branch. Intuitively, for any 14

fixed i, it would make sense to branch on z ik before branching on z ik for some k > k, since this corresponds to making a bigger change in the domain of x i. One could use different numbers of bits for different x variables. To do this effectively, one would need heuristic rules for deciding which variables merit more precision than others. One could derive conditions under which one can binarise only a subset of the x variables, yet still convexify the resulting (mixed) 0-1 QCQP. For example, one could binarise only the variables that have non-zero entries in at least one matrix Q k that is not psd. (Note that, in the case of PP, this would enable us to treat θ as continuous.) One could develop methods for handling infeasible 0-1 QCQPs. For example, one could relax the constraints initially, solve the modified 0-1 QCQP to get a nearly feasible QCQP solution, and repair that solution to make it feasible. Finally, on the theoretical side, one could attempt to prove that, if the QCQP satisfies certain conditions, then running our heuristic with a certain number of bits is guaranteed to yield a solution of a certain quality. (Note that such a result could not be proved for general QCQPs, since even finding a feasible solution of a general QCQP is N P-hard in the strong sense.) Acknowledgement We thank Professor Kurt Anstreicher for providing us with the QPB instances and their optimal solution values. References [1] F. Alizadeh & D. Goldfarb (2003) Second-order cone programming. Math. Program., 95, 3 51. [2] F.A. Al-Khayyal, C. Larsen & T. Van Voorhis (1995) A relaxation method for non-convex quadratically constrained quadratic programs. J. Glob. Optim., 6, 215 230. [3] K.M. Anstreicher (2009) Semidefinite programming versus the reformulation-linearization technique for non-convex quadratically constrained quadratic programming. J. Glob. Optim., 43, 471 484. [4] K.M. Anstreicher (2012) On convex relaxations for quadratically constrained quadratic programming. Math. Program., 136, 233 251. 15

[5] C. Audet, P. Hansen, B. Jaumard & G. Savard (2000) A branch-and-cut algorithm for nonconvex quadratically constrained quadratic programming. Math. Program., 87, 131 152. [6] X. Bao, N.V. Sahinidis & M. Tawarmalani (2009) Multiterm polyhedral relaxations for nonconvex, quadratically-constrained quadratic programs. Optim. Methods & Softw., 24, 485 504. [7] X. Bao, N.V. Sahinidis & M. Tawarmalani (2011) Semidefinite relaxations for quadratically constrained quadratic programming: a review and comparisons. Math. Program., 129, 129 157. [8] A. Billionnet, S. Elloumi & M.-C. Plateau (2009) Improving the performance of standard solvers for quadratic 0-1 programs by a tight convex reformulation: the QCR method. Discr. Appl. Math., 157, 1185 1197. 2009. [9] I.M. Bomze & M. Locatelli (2004) Undominated d.c. decompositions of quadratic functions and applications to branch-and-bound approaches. Comput. Optim. & Appl., 28, 227 245. [10] S. Boyd & L. Vandenberghe (2004) Convex Optimization. Cambridge: Cambridge University Press. [11] S.A. Burer & D. Vandenbussche (2008) A finite branch-and-bound algorithm for nonconvex quadratic programming via semidefinite relaxations. Math. Program., 113, 259 282. [12] S.A. Burer & J. Chen (2012) Globally solving nonconvex quadratic programming problems via completely positive programming. Math. Program. Comput., 4, 33 52. [13] S.A. Burer & A.N. Letchford (2012) Non-convex mixed-integer nonlinear programming: a survey. Surveys in Oper. Res. & Mgmt. Sci., 17, 97 106. [14] E. Feron (1999) Nonconvex quadratic programming, semidefinite relaxations and randomization algorithms in information and decision systems. In T. E. Djaferis & I. Schick (eds.) System Theory: Modeling Analysis and Control. Dortrecht: Kluwer. [15] R. Fortet (1959) L Algèbre de Boole et ses applications en recherche opérationnelle. Cahiers Centre Etudes Rech. Oper., 4, 5 36. [16] T. Fujie & M. Kojima (1997) Semidefinite programming relaxation for nonconvex quadratic programs. J. Glob. Optim., 10, 367 380. 16

[17] L. Galli & A.N. Letchford (2014) A compact variant of the QCR method for quadratically constrained quadratic 0-1 programs. Optim. Lett., 8, 1213 1224. [18] I.E. Grossmann (2002) Review of nonlinear mixed-integer and disjunctive techniques. Optim. & Eng., 3, 227 252. [19] P.L. Hammer & A.A. Rubin (1970) Some remarks on quadratic programming with 01 variables. RAIRO, 3, 67 79. [20] IPOPT (Interior-Point Optimizer). Managed by A. Wächter & S. Wigerske. Available at https://projects.coin-or.org/ipopt [21] J.E. Kelley (1960) The cutting-plane method for solving convex programs. SIAM Journal, 8, 703 713. [22] C. Lemarechal & F. Oustry (2001) SDP relaxations in combinatorial optimization from a Lagrangian viewpoint. In N. Hadjisawas & P.M. Pardalos (eds.) Advances in Convex Analysis and Global Optimization. Dortrecht: Kluwer. [23] J. Linderoth (2005) A simplicial branch-and-bound algorithm for solving quadratically constrained quadratic programs. Math. Program., 103, 251 282. [24] Z.-Q. Luo, W.-K. Ma, A.M.-C. So, Y. Ye & S. Zhang (2010) Semidefinite relaxation of quadratic optimization problems. IEEE Signal Processing Magazine, 27, 20 34. [25] M.C. Markót & T. Csendes (2005) A new verified optimization technique for the packing circles in a unit square problem. SIAM J. Optim., 16, 193 219. [26] G.P. McCormick (1976) Computability of global solutions to factorable nonconvex programs: Part I Convex underestimating problems. Math. Program., 10, 147 175. [27] R. Misener & C.A. Floudas (2013) GloMIQO: Global mixed-integer quadratic optimizer. J. Glob. Optim, 57, 3 50. [28] K.G. Murty & S.N. Kabadi (1987) Some N P-complete problems in quadratic and nonlinear programming. Math. Program., 39, 117 129. [29] S. Poljak, F. Rendl & H. Wolkowicz (1995) A recipe for semidefinite relaxation for (0,1)-quadratic programming. J. Glob. Optim., 7, 51 73. [30] U. Raber (1998) A simplicial branch-and-bound method for solving nonconvex all-quadratic programs. J. Glob. Optim., 13, 417 432. 17

[31] M. Ramana (1993) An Algorithmic Analysis of Multiquadratic and Semidefinite Programming Problems. PhD thesis, Johns Hopkins University, Baltimore, MD. [32] F. Roupin (2004) From linear to semidefinite programming: an algorithm to obtain semidefinite relaxations for bivalent quadratic problems. J. Comb. Optim., 8, 469 493. [33] A. Saxena, P. Bonami & J. Lee (2010) Convex relaxations of nonconvex mixed integer quadratically constrained programs: extended formulations. Math. Program., 124, 383 411. [34] A. Saxena, P. Bonami & J. Lee (2011) Convex relaxations of nonconvex mixed integer quadratically constrained programs: projected formulations. Math. Program., 130, 359 413. [35] H.D. Sherali & C.H. Tuncbilek (1992) A global optimization algorithm for polynomial programming problems using a reformulation linearization technique. J. Glob. Optim., 2, 101 112. [36] N.Z. Shor (1987) Quadratic optimization problems. Soviet J. Comput. Syst. Sci., 25, 111. [37] P.G. Szabó, M.C. Markót & T. Csendes (2005) Global optimization in geometry circle packing into the square. In C. Audet, P. Hansen & G. Savard (eds.) Essays and Surveys in Global Optimization, pp. 233 266. Dordrecht: Kluwer. [38] M. Tawarmalani & N.V. Sahinidis (2003) Convexification and Global Optimization in Continuous and Mixed-Integer Nonlinear Programming. Dortrecht: Kluwer. [39] D. Vandenbussche & G.L. Nemhauser (2005) A branch-and-cut algorithm for nonconvex quadatic programs with box constraints. Math. Program., 102, 559 575. [40] X.J. Zheng, X.L. Sun, D. Li. (2011) Nonconvex quadratically constrained quadratic programming: best D.C. decompositions and their SDP representations. J. Glob. Optim., 50, 695 712. [41] X.J. Zheng, X.L. Sun, D. Li. (2011) Convex relaxations for nonconvex quadratically constrained quadratic programming: matrix cone decomposition and polyhedral approximation. Math. Program., 129, 301 329. 18