Advances in CPLEX for Mixed Integer Nonlinear Optimization

Pierre Bonami and Andrea Tramontani IBM ILOG CPLEX ISMP 2015 - Pittsburgh - July 13 2015 Advances in CPLEX for Mixed Integer Nonlinear Optimization 1 2015 IBM Corporation

CPLEX Optimization Studio 12.6.2 is available now! DoCloud services: OPL on the cloud. Performance improvements to CPO CPLEX Optimizer performance: 2 2015 IBM Corporation

CPLEX Optimization Studio 12.6.2 is available now! DoCloud services: OPL on the cloud. Performance improvements to CPO CPLEX Optimizer performance: Exploit symmetry for LP Cuts for non-convex (MI)QP Improvements for SOCP Improvement to convex MIQCP/MISOCP 2 2015 IBM Corporation

Convex MIQCP and MISOCP min c T x x T Q k x +ak Tx a0 k Ax = b, x j Z k = 1,...,m, j = 1,...,p. (MIQCP) Where all quadratic constraints can be represented as second order cones: L d := {(x,x 0 ) R d+1 : d i=1 x 2 i x 2 0,x 0 0}. (L d defines the (d + 1)-dimensional second order cone.) 3 2015 IBM Corporation

Second order cone representability Through basic transformations can model: Convex quadratic constraints: x T Qx +a T x a 0, with Q 0 Rotated second order cones: d xi 2 x 0 x 1, with x 0,x 1 0 or more complicated... i=2 x T Qx +a T x c T x +b, with Q 0 Second Order Cone Programming (p = 0) Solved efficiently by interior point methods Supported by many commercial solvers (CPLEX in particular). CPLEX automatically recognizes and reformulates convex quadratics, rotated SOC, but not every form. 4 2015 IBM Corporation

MISOCP min c T x (x Ji,x hi ) L d i i = 1,...,m Ax = b, x j Z j = 1,...,p. (MISOCP) Algorithms are based on SOCP relaxation. Convex MINLP algorithms work with some added technicality due to non-differentiability [Drewes, 2009, Drewes and Ulbrich, 2012]. Supported by most commercial MIP solvers (CPLEX in particular). 5 2015 IBM Corporation

Agenda The basic algorithms in CPLEX SOCP-BB OA-BB Comparisons in CPLEX 12.6.1 Novelties of CPLEX 12.6.2 Cone disaggregation Lift-and-project for MISOCP Comparisons of CPLEX 12.6.1 vs. 12.6.2 6 2015 IBM Corporation

SOCP based branch-and-bound Straightforward generalization of main MILP algorithm: solve an SOCP at each node of the tree. 7 2015 IBM Corporation

SOCP based branch-and-bound Straightforward generalization of main MILP algorithm: solve an SOCP at each node of the tree. Branch on variables with fractional value. integer feasible infeasible fathomed by bound 7 2015 IBM Corporation

SOCP based branch-and-bound Straightforward generalization of main MILP algorithm: solve an SOCP at each node of the tree. Branch on variables with fractional value. Prune by infeasibility, bounds and integer feasibility. integer feasible infeasible fathomed by bound 7 2015 IBM Corporation

Outer Approximation [Duran and Grossmann, 1986] min c T x s.t. g i (x) 0 Ax = b x j Z, i = 1,...,m, j = 1,...,p. Idea: Take first-order approximations of constraints at different points and build an equivalent MILP. 8 2015 IBM Corporation

OA Branch-and-cut [Quesada and Grossmann, 1992] Initialize by solving SOCP relaxation and taking OA s at the optimum. At each integer feasible node: integer feasible 9 2015 IBM Corporation

OA Branch-and-cut [Quesada and Grossmann, 1992] Initialize by solving SOCP relaxation and taking OA s at the optimum. At each integer feasible node: 1 solve SOCP with integers fixed, and enrich the set of linearizations. 2 Resolve the LP relaxation of the node with the new cuts. 3 Repeat as long as node is integer feasible. integer feasible 9 2015 IBM Corporation

The MISOCP solver in CPLEX Implements two main algorithms; choice controled by parameter CPXPARAM_MIP_Strategy_MIQCPStrat SOCP based Branch-and-bound (miqcpstrat 1). LP based branch-and-cut (miqcpstrat 2). Default (miqcpstrat 0) is deciding which of the two to run in a clever way. History of MISOCP with CPLEX class p algorithm V. (Year) SOCP 0 barrier 9.0 (2003) MISOCP > 0 SOCP based B&B 9.0 (2003) LP based B&C 11.0 (2007) Cone propagations 12.6.1 (2014) 10 2015 IBM Corporation

Comparing OA and SOCP BB s in CPLEX 12.6.1 % model solved 225 models solved by at least one method and failed by none. Default strategy picked 100 80 60 40 20 1261_miqcpstrat_0 1261_miqcpstrat_1 1261_miqcpstrat_2 0 1 10 100 1000 10000 100000 time factor OA 113 times SOCP-BB 46 times 56 models identical with both To be perfect should have picked 14 more models with OA 36 more models with SOCP-BB 11 2015 IBM Corporation

How bad can outer approximation be? Consider the following convex MIQCP: min s.t. c T x n ( ) i=1 xi 1 2 2 n 1 (1) 4 x Z n z (1) is infeasible: The ball is too small to contain integer points. It is large enough to touch every edge of the hypercube. x y 12 2015 IBM Corporation

Solving (1) with OA cuts z No OA constraint can cut 2 vertices of the hypercube. If an inequality cuts two vertices, it cuts the segment joining them. This can not be: the ball has non-empty intersection with any such segment. An OA would need at least 2 n OA cuts to converge. x y 13 2015 IBM Corporation

Experimental illustration 2 CPLEX 12.4 1 SCIP 2.1 Bonmin B-Hyb n 2 n nodes nodes nodes 10 1,024 2,047 720 11,156 15 32,768 65,535 31,993 947,014 20 1,048,576 2,097,151 1,216,354... 1 CPLEX ran in single threaded mode 2 Three years ago I liked this slide 14 2015 IBM Corporation

Experimental illustration 2 Remark CPLEX 12.4 1 SCIP 2.1 Bonmin B-Hyb n 2 n nodes nodes nodes 10 1,024 2,047 720 11,156 15 32,768 65,535 31,993 947,014 20 1,048,576 2,097,151 1,216,354... Problem is simple for CPLEX/SCIP if variables are 0 1: replace xi 2 by x i, the contradiction n 4 n 1 4 follows. 1 CPLEX ran in single threaded mode 2 Three years ago I liked this slide 14 2015 IBM Corporation

Experimental illustration 2 Remark CPLEX 12.4 1 SCIP 2.1 Bonmin B-Hyb n 2 n nodes nodes nodes 10 1,024 2,047 720 11,156 15 32,768 65,535 31,993 947,014 20 1,048,576 2,097,151 1,216,354... Problem is simple for CPLEX/SCIP if variables are 0 1: replace xi 2 by x i, the contradiction n 4 n 1 4 follows. SCIP 3.0 and CPLEX 12.6.1 solve it in a blink. 1 CPLEX ran in single threaded mode 2 Three years ago I liked this slide 14 2015 IBM Corporation

New in CPLEX 12.6.2 Cone disaggregation for MISOCP Lift-and-project cuts for MISOCP Redesigned heuristic choice of most promising algorithm Updated history of MIQCP with CPLEX class p algorithm V. (Year) SOCP 0 barrier 9.0 (2003) MISOCP > 0 barrier based B&B 9.0 (2003) Outer approximation B&C 11.0 (2007) Cone propagations 12.6.1 (2014) Major improvements 12.6.2 (June 2015) 15 2015 IBM Corporation

Separable mixed integer convex programs min c T x s.t. g i (x) 0 i = 1,...,m Ax = b x j Z j = 1,...,p l x u (sminlp) For i = 1,...,m, g i are convex separable: g i (x) = with g ij : [l j,u j ] R convex. n g ij (x j ) j=1 16 2015 IBM Corporation

Disaggregated formulation Introduce one variable y ij for each elementary function: min s.t. c T x n y ij 0 j=1 g ij (x j ) y ij Ax = b, x i Z l x u. i = 1,...,m, i = 1,...,m, j = 1,...,n, i = 1,...,p, (sminlp ) 17 2015 IBM Corporation

Cone disaggregation for MISOCP In standard form the nonlinear constraint describing the second order cone is not convex separable: n xi 2 x0 2 i=1 [Vielma et al., 2015] (go to FC24!), divide the constraint by x 0 0 to get a convex separable constraint: n i=1 Now introduce y 1,...,y n and rewrite as: x 2 i x 0 x 0. n y i x 0 i=1 x 2 i x 0 y i 18 2015 IBM Corporation

Extended formulation of (1) Application to (1) [Hijazi et al., 14] z min c T x n s.t. y i (n 1)/4 i=1 (x i 0.5) 2 y i i = 1,...,n x Z n. Its outer approximation min c T x n s.t. y i (n 1)/4 i=1 2 ( x k i 0.5 ) (x i x k i )+ ( x k i 0.5 ) 2 yi i = 1,...,n k = 1,...,K x Z n x y 19 2015 IBM Corporation

Extended formulation of (1) min c T x n s.t. y i (n 1)/4 i=1 Application to (1) [Hijazi et al., 14] (x i 0.5) 2 y i i = 1,...,n x Z n. Its outer approximation min c T x n s.t. y i (n 1)/4 i=1 2 ( x k i 0.5 ) (x i x k i )+ ( x k i 0.5 ) 2 yi i = 1,...,n k = 1,...,K x Z n 19 2015 IBM Corporation

Extended formulation of (1) Application to (1) [Hijazi et al., 14] min c T x 2 points suffice to make the mixed-integer set infeasible n x 1 = 0 and x 2 = 1: s.t. y i (n 1)/4 x i + 0.25 y i i = 1,...,n i=1 (x i 0.5) 2 x i 0.75+ y y i i = 1,...,n i i = 1,...,n y i 0.25 x Z n x. i Z Its outer approximation min c T x n s.t. y i (n 1)/4 i=1 2 ( x k i 0.5 ) (x i x k i )+ ( x k i 0.5 ) 2 yi i = 1,...,n k = 1,...,K x Z n 19 2015 IBM Corporation

Lift-and-project cuts for MISOCP Cuts are an essential component of MILP solvers Can always apply MILP cuts to a linear OA of MISOCP (and we do it) Can we generate better cuts by looking directly at nonlinear functions? A partial answer: as long as the cut generated is linear it could also have been obtained from a linear outer approximation In the past three years, tremendous activity towards conic cuts for conic programming [Andersen and Jensen, 2013, Belotti et al., 2013, Kılınç-Karzan and Yıldız, 2015, Modaresi et al., 2015] (among others) Our goal Derive linear cutting planes Fast Find an appropriate OA from which to derive a cut 20 2015 IBM Corporation

Split Relaxation Consider C and M := C (Z p R n p ). Let π Z p {0} n p, π 0 Z and ( C (π,π0) := conv C ({ x : π T } x π 0 { x : π T x π 0 + 1 })). α T x = β ˆx C (π,π0) (clearly M C (π,π 0) C). π T x π 0 π T x π 0 + 1 21 2015 IBM Corporation

Two approaches from Mixed Integer Convex Programming Goal: build a linear OA from which a "best" cut can be deduced. Use Disjunctive Programming for MILP to derive the cuts. Using only LP [Kılınc et al., 2011]. Using nonlinear programming [Bonami, 2011] 1 Start with any linear OA of C 1 Solve a single NLP that tells if ˆx is in the split relaxation. 2 Solve a Cut Generation LP, if a cut if found STOP. 3 Deduce from dual of CGLP two points proving that ˆx is in the split relaxation. 4 If point(s) not in C generate new OA and goto 2, otherwise there is no cut, STOP. 2 If not, deduce from solution two points such that ˆx = λx 1 +(1 λ)x 0 and closest to the disjunction. 3 Build OA around these two points. 4 Solve MLP and get the cut. 22 2015 IBM Corporation

Lift-and-project for MISOCP Hybridizing the approaches of [Kılınc et al., 2011] and [Bonami, 2011] From [Kılınc et al., 2011] Pure LP approach, Dynamic generation of additional OA constraints. ˆx From [Bonami, 2011] compact formulation using MLP, penalization of the non-linear constraints. π T x π0 π T x π0 + 1 23 2015 IBM Corporation

Lift-and-project for MISOCP Hybridizing the approaches of [Kılınc et al., 2011] and [Bonami, 2011] From [Kılınc et al., 2011] Pure LP approach, Dynamic generation of additional OA constraints. ˆx x 1 x 0 From [Bonami, 2011] compact formulation using MLP, penalization of the non-linear constraints. π T x π0 π T x π0 + 1 23 2015 IBM Corporation

How good can this be? z [Cornuéjols and Li, 2001] showed that the empty ball in dimension n has split rank n (also holds for any outer approximation in this space) Practically unsolvable using any form of split cuts (even conic ones). x y Instead the OA of the disaggregated formulation has (simple) split rank 1. 24 2015 IBM Corporation

How good can this be? z [Cornuéjols and Li, 2001] showed that the empty ball in dimension n has split rank n (also holds xfor i + any 0.25outer y i i = 1,...,n approximation inx i this 0.75+ space) y i i = 1,...,n Practically unsolvable using any form y i 0.25 x of split cuts (even i 0 OR x conic ones). i 1 x y Instead the OA of the disaggregated formulation has (simple) split rank 1. y i 1 4 x i 0 x i 1 24 2015 IBM Corporation

A more complicated empty ellipse n (100x2i 2 + 100x2i 1 2 4x 2i x 2i 1 98x 2i 98x 2i 1 ) 1 i=1 x Z 2n Results on 12 threads with 12.6.1, 12.6.2 and 12.6.2 aggressive cuts 3, 3 hours time limit. 12.6.1 12.6.2 12.6.2++ n nodes nodes nodes 5 2,261 2,045 1,825 10 20,97,151 29 1 15 > 23,125,426 7,769 1 (Largest model solved in 2.2 sec by 12.6.2, in 5.5 sec by 12.6.2++.) 3 set mip limit cutpasses 100000 set mip cuts liftproj 3 set mip strategy miqcpstrat 2 25 2015 IBM Corporation

A more complicated empty ellipse Similar results previously observed by [Kılınç, 2011] Original Extended n root gap sol time root gap sol time Batch n 10 58.40 376.2 68.77 58.7 Markowitz (100x2i 2 10 + 100x2i 1 2 0.00 4x> 2i x10 2i 1 800 98x 2i 98.07 98x 2i 1 1) 262 1 i=1 SLay 14 68.50 36 86.08 5.0 uflquad x Z 2n 15 10.85 784 96.25 145 SeeResults also related on 12theoretical threads with results 12.6.1, for 12.6.2 MILP case and 12.6.2 [Boduraggressive et al., 2015] cuts and, 3go hours to FD01. time limit. 12.6.1 12.6.2 12.6.2++ n nodes nodes nodes 5 2,261 2,045 1,825 10 20,97,151 29 1 15 > 23,125,426 7,769 1 (Largest model solved in 2.2 sec by 12.6.2, in 5.5 sec by 12.6.2++.) 25 2015 IBM Corporation

Remarks Cone disaggregation Automatically applied by default during presolve, Only interesting (and done): 1 If using the OA-BB, 2 For cones that are long enough. Lift-and-project Also only done in OA based algorithm. Still an expensive algorithm (may not speed up every easy model). Also... Redesigned heuristic to choose algorithm to apply in view of these changes. 26 2015 IBM Corporation

Computational experiments MISOCP 12.6.2 3 Test bed Cplex internal MIQCP test bed: 296 models Benchmarks test set of CBLIB ( http://cblib.zib.de): 80 models Compare the new release Cplex 12.6.2 against Cplex 12.6.1: geo. mean of branch-and-bound computing times. Compare the two algorithmic strategies SOCP-BB and OA-BB in Cplex 12.6.2. 3 All tests are carried on Linux machines: Intel X5650 @ 2.67 GHz, 24 GB RAM, 12 threads, deterministic. 27 2015 IBM Corporation

CPLEX 12.6.1 vs 12.6.2 1.0 0.8 0.6 0.4 0.2 0 1.0 0.8 0.6 0.4 0.2 0 2.86 5.0 > 0 sec 231 Models > 1 sec 149 Models 3.57 6.25 > 0 sec 66 Models > 1 sec 46 Models CPLEX test bed CPLEX 12.6.1: 62 time limits CPLEX 12.6.2: 38 time limits CBLIB CPLEX 12.6.1: 17 time limits CPLEX 12.6.2: 8 time limits 28 2015 IBM Corporation

Comparing OA and SOCP BB s in CPLEX 12.6.2 % model solved 245 models solved by at least one method and failed by none. Default strategy picked 100 80 60 40 20 1262_miqcpstrat_0 1262_miqcpstrat_1 1262_miqcpstrat_2 0 1 10 100 1000 10000 100000 time factor OA-BB 186 times SOCP-BB 4 times 55 models identical with both To be perfect should have picked 2 more models with OA-BB 9 more models with SOCP-BB 29 2015 IBM Corporation

Reminder of CPLEX 12.6.1 % model solved 225 models solved by at least one method and failed by none. Default strategy picked 100 80 60 40 20 1261_miqcpstrat_0 1261_miqcpstrat_1 1261_miqcpstrat_2 0 1 10 100 1000 10000 100000 time factor OA 113 times SOCP-BB 46 times 56 models identical with both To be perfect should have picked 14 more models with OA 36 more models with SOCP-BB 30 2015 IBM Corporation

Other CPLEX and CPO related talks @ISMP2015 Failure-directed Search for Constraint-based Scheduling P. Vilím, Monday 11:20 (MB25) Symmetry in Linear Programming R. Wunderling, Tuesday 10:20 (TB07) Accelerating the Development of Efficient CP Optimizer Models P. Laborie, Tuesday 14:45 (TD19) Advances in the CPLEX Distributed Solver L. Ladanyi, Thursday 13:10 (ThC15) On Mathematical Programming with Indicator Constraints A. Lodi, Thursday 16:35 (ThE01) Max Clique Cuts for Standard Quadratic Programs J. Schweiger, Friday 10:50 (FB29) Zero-Half Cuts for Solving Nonconvex Quadratic Programs with Box Constraints J. Linderoth, Friday 11:20 (FB29) 31 2015 IBM Corporation

Legal Disclaimer IBM Corporation 2015. All Rights Reserved. The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. IBM, the IBM logo, CPLEX and SPSS are trademarks of International Business Machines Corporation in the United States, other countries, or both. Intel, Intel Xeon are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. 33 2015 IBM Corporation

References I K. Andersen and A. Jensen. Intersection cuts for mixed integer conic quadratic sets. In M. Goemans and J. Correa, editors, Integer Programming and Combinatorial Optimization, volume 7801 of Lecture Notes in Computer Science, pages 37 48. Springer Berlin Heidelberg, 2013. ISBN 978-3-642-36693-2. doi: 10.1007/978-3-642-36694-9_4. URL http://dx.doi.org/10.1007/978-3-642-36694-9_4. P. Belotti, J. C. Góez, I. Pólik, T. K. Ralphs, and T. Terlaky. On families of quadratic surfaces having fixed intersections with two hyperplanes. Discrete Applied Mathematics, 161(16 17):2778 2793, 2013. ISSN 0166-218X. doi: http://dx.doi.org/10.1016/j.dam.2013.05.017. URL http://www.sciencedirect.com/science/article/pii/s0166218x13002461. M. Bodur, S. Dash, and O. Günlük. Cutting planes from extended lp formulations. Technical report, 2015. P. Bonami. Lift-and-project cuts for mixed integer convex programs. In IPCO, pages 52 64, 2011. G. Cornuéjols and Y. Li. On the rank of mixed 0,1 polyhedra. In K. Aardal and B. Gerards, editors, Integer Programming and Combinatorial Optimization, volume 2081 of Lecture Notes in Computer Science, pages 71 77. Springer Berlin Heidelberg, 2001. ISBN 978-3-540-42225-9. doi: 10.1007/3-540-45535-3_6. URL http://dx.doi.org/10.1007/3-540-45535-3_6. S. Drewes. Mixed Integer Second Order Cone Programming. PhD thesis, Technische Universität Darmstadt, 2009. S. Drewes and S. Ulbrich. Subgradient based outer approximation for mixed integer second order cone programming. In J. Lee and S. Leyffer, editors, Mixed Integer Nonlinear Programming, volume 154 of The IMA Volumes in Mathematics and its Applications, pages 41 59. Springer New York, 2012. ISBN 978-1-4614-1926-6. doi: 10.1007/978-1-4614-1927-3_2. URL http://dx.doi.org/10.1007/978-1-4614-1927-3_2. M. A. Duran and I. Grossmann. An outer-approximation algorithm for a class of mixed-integer nonlinear programs. Mathematical Programming, 36:307 339, 1986. H. Hijazi, P. Bonami, and A. Ouorou. An outer-inner approximation for separable mixed-integer nonlinear programs. INFORMS Journal on Computing, 26(1):null, 14. doi: 10.1287/ijoc.1120.0545. M. Kılınc, J. Linderoth, and J. Luedtke. Effective separation of disjunctive cuts for convex mixed integer nonlinear programs. Technical Report 1681, 2011. M. R. Kılınç. Disjunctive Cutting Planes ann Algorithms for Convex Mixed Integer Nonlinear Programming. PhD thesis, University of Wisconsin-Madison, 2011. 34 2015 IBM Corporation

References II F. Kılınç-Karzan and S. Yıldız. Two term disjunctions on the second-order cone. Mathematical Programming, April 2015. http://link.springer.com/article/10.1007/s10107-015-0903-4. S. Modaresi, M. Kılınç, and J. Vielma. Intersection cuts for nonlinear integer programming: convexification techniques for structured sets. Mathematical Programming, pages 1 37, 2015. ISSN 0025-5610. doi: 10.1007/s10107-015-0866-5. URL http://dx.doi.org/10.1007/s10107-015-0866-5. I. Quesada and I. E. Grossmann. An LP/NLP based branch and bound algorithm for convex MINLP optimization problems. Computers and Chemical Engineering, 16:937 947, 1992. J. P. Vielma, I. Dunning, J. Huchette, and M. Lubin. Extended Formulations in Mixed Integer Conic Quadratic Programming. Research Report, 2015. 35 2015 IBM Corporation