Online generation via offline selection - Low dimensional linear cuts from QP SDP relaxation -

Transcription

1 Online generation via offline selection - Low dimensional linear cuts from QP SDP relaxation - Radu Baltean-Lugojan Ruth Misener Computational Optimisation Group Department of Computing Pierre Bonami Andrea Tramontani IBM Research CPLEX Team /03/07 Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 1 / 23

2 Deterministically solving non-convex QP QP : min x T Qx + c T x x s.t. Ax b, x [0, 1] N Prior work: Branch & Cut with relaxations RLT McCormick + ext., e.g. triangle ineq. Bonami et al. [2016] (QP) SDP/SOCP/Convex Dong [2016], Saxena et al. [2011], Buchheim and Wiegele [2013], Zheng et al. [2011], Bao et al. [2011], Anstreicher [2009], Chen and Burer [2012] LP relaxation of SDP (typically high dimensional) e.g. Qualizza et al. [2012], Sherali and Fraticelli [2002] Edge-concave Bao et al. [2009], Misener and Floudas [2012] This work: Off-line (learned) selection of low-dim. LP cuts from SDP Develop online strong low dimensional linear cuts; Offline cut selection via neural net estimator trained a priori ; Cheaply outer-approximate SDP esp. in combination with other low-dim cuts (RLT, triangle, edge-concave, Boolean quadric polytope). Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 2 / 23

3 SDP & RLT relaxations (Anstreicher, J Glob Optim, 2009) min x x T Qx + c T x Ax b, min x Q X + c T x Ax b, X xx T SDP relaxation x [0, 1] N X ii x i X ij x i x j 1 X ij x i 0 RLT relaxation X ij x j 0 x [0, 1] N X [0, 1] N N X ij = X ji, Q X is the matrix inner product Q X = N i,j=1 Q ij X ij, SDP Semidefinite programming, RLT Reformulation linearisation technique. Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 3 / 23

4 SDP & RLT relaxations (Anstreicher, J Glob Optim, 2009) min x Q X + c T x Ax b, X = xx T x [0, 1] N X [0, 1] N N min x Q X + c T x Ax b, X xx T SDP relaxation X ii x i X ij x i x j 1 X ij x i 0 RLT relaxation X ij x j 0 x [0, 1] N X [0, 1] N N X ij = X ji, Q X is the matrix inner product Q X = N i,j=1 Q ij X ij, SDP Semidefinite programming, RLT Reformulation linearisation technique. Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 3 / 23

5 SDP & RLT relaxations (Anstreicher, J Glob Optim, 2009) min x Q X + c T x Ax b, X = xx T x [0, 1] N X [0, 1] N N min x Q X + c T x Ax b, X xx T SDP relaxation X ii x i X ij x i x j 1 X ij x i 0 RLT relaxation X ij x j 0 x [0, 1] N X [0, 1] N N X ij = X ji, Q X is the matrix inner product Q X = N i,j=1 Q ij X ij, SDP Semidefinite programming, RLT Reformulation linearisation technique. Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 3 / 23

6 SDP & RLT relaxations (Anstreicher, J Glob Optim, 2009) min x Q X + c T x Ax b, X = xx T x [0, 1] N X [0, 1] N N min x Q X + c T x Ax b, X xx T SDP relaxation X ii x i X ij x i x j 1 X ij x i 0 RLT relaxation X ij x j 0 x [0, 1] N X [0, 1] N N X ij = X ji, Q X is the matrix inner product Q X = N i,j=1 Q ij X ij, SDP Semidefinite programming, RLT Reformulation linearisation technique. Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 3 / 23

7 SDP & RLT relaxations (Anstreicher, J Glob Optim, 2009) min x Q X + c T x Ax b, X = xx T x [0, 1] N X [0, 1] N N min x Q X + c T x Ax b, X xx T SDP relaxation X ii x i X ij x i x j 1 X ij x i 0 RLT relaxation X ij x j 0 x [0, 1] N X [0, 1] N N X ij = X ji, Q X is the matrix inner product Q X = N i,j=1 Q ij X ij, SDP Semidefinite programming, RLT Reformulation linearisation technique. Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 3 / 23

8 SDP & RLT relaxations: Point Packing (Anstreicher, J Glob Optim, 2009) Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 4 / 23

9 Schur s complement & Decomposition for SDP relaxations Schur s complement SDP Relaxation X = xx T relax X xx T 1 x Schur s complement X xx T T 0 x X Key Idea Consider smaller subsets We have X = xx T, so X S = x S xs T 1 x 1 x 2 x 3 x 4 x 1 X 11 X 12 X 13 X 14 x 2 X 21 X 22 X 23 X 24 x 3 X 31 X 32 X 33 X 34 0 = x 4 X 41 X 42 X 43 X 44 for all S 1, N, e.g. 1 x 1 x 2 x 3 x 1 X 11 X 12 X 13 x 2 X 21 X 22 X 23 0 x 3 X 31 X 32 X 33 Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 5 / 23

10 Sum-additive objective decomposition Recall Power set P n If finite set S has S = n elements, then S has P n = 2 n subsets. min x Q X + c T x Ax b, X = xx T x [0, 1] N X [0, 1] N N = min x S P n Q S X S + c T x Ax b, X = xx T x [0, 1] N X [0, 1] N N Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 6 / 23

11 Initial RLT relaxation min x Q X + c T x Ax b, X = xx T x [0, 1] N X [0, 1] N N min x Q X + c T x Ax b, X ij x i x j 1 i, j X ij x i 0 i, j X ij x j 0 i, j X ij = X ji i, j x [0, 1] N X [0, 1] N N x, X Cutting plane motivation x is feasible in the space of the original QP, For nonconvex QP, X may be infeasible in the original QP, I find LP solvers easier to use than SDP solvers, As in MIP & SAT, may want low-dimensional cutting planes. Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 7 / 23

12 A decomposition of the SDP relaxation for QP (1) QP SDP relaxation min Q X + x,x ct x s.t. Ax b, 1 x T 0, x X x [0, 1] N, X ii x i i (2) QP SDP relaxation at given point x min f (X x) = Q X X 1 x T s.t. 0, X ii x i i x X (3) Relaxed QP SDP relaxation at given x min f S (X S x S ) X S P n 1 x T s.t. S 0 S P n, X ii x i i, x S X S where P n = {S 1, N, S = n N} (*) and, f (X x) = Q X = Q S X S = f S (X S x S ). S P n S P n (4) nd-sdp sub-problems Parameters: n, S, Q S, x S S P n S (XS x S ) = min Q S X S X S 1 x S T s.t. 0, X ii x i i S x S X S f Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 8 / 23

13 A decomposition of the SDP relaxation for QP (1) QP SDP relaxation min Q X + x,x ct x s.t. Ax b, 1 x T 0, x X x [0, 1] N, X ii x i i (2) QP SDP relaxation at given point x min f (X x) = Q X X 1 x T s.t. 0, X ii x i i x X (3) Relaxed QP SDP relaxation at given x min f S (X S x S ) X S P n 1 x T s.t. S 0 S P n, X ii x i i, x S X S where P n = {S 1, N, S = n N} (*) and, f (X x) = Q X = Q S X S = f S (X S x S ). S P n S P n (4) nd-sdp sub-problems Parameters: n, S, Q S, x S S P n S (XS x S ) = min Q S X S X S 1 x S T s.t. 0, X ii x i i S x S X S f Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 8 / 23

14 A decomposition of the SDP relaxation for QP (1) QP SDP relaxation min Q X + x,x ct x s.t. Ax b, 1 x T 0, x X x [0, 1] N, X ii x i i (2) QP SDP relaxation at given point x min f (X x) = Q X X 1 x T s.t. 0, X ii x i i x X (*) We take n = 3, 4, 5 in our experiments. (3) Relaxed QP SDP relaxation at given x min f S (X S x S ) X S P n 1 x T s.t. S 0 S P n, X ii x i i, x S X S where P n = {S 1, N, S = n N} (*) and, f (X x) = Q X = Q S X S = f S (X S x S ). S P n S P n (4) nd-sdp sub-problems Parameters: n, S, Q S, x S S P n S (XS x S ) = min Q S X S X S 1 x S T s.t. 0, X ii x i i S x S X S Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 8 / 23 f

15 A decomposition of the SDP relaxation for QP (1) QP SDP relaxation min Q X + x,x ct x s.t. Ax b, 1 x T 0, x X x [0, 1] N, X ii x i i (2) QP SDP relaxation at given point x min f (X x) = Q X X 1 x T s.t. 0, X ii x i i x X (*) We take n = 3, 4, 5 in our experiments. (3) Relaxed QP SDP relaxation at given x min f S (X S x S ) X S P n 1 x T s.t. S 0 S P n, X ii x i i, x S X S where P n = {S 1, N, S = n N} (*) and, f (X x) = Q X = Q S X S = f S (X S x S ). S P n S P n (4) nd-sdp sub-problems Parameters: n, S, Q S, x S S P n S (XS x S ) = min Q S X S X S 1 x S T s.t. 0, X ii x i i S x S X S Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 8 / 23 f

16 Cut selection from nd-sdp sub-problems at given x Generate outer-approximate hyperplanes for each nd-sdp S P n : f S (XS x S ) = min Q S X S, X S 1 x T s.t. S 0, X ii x i i S x S X S (Given n, S, parametric on Q S, x S ) Selection of nd-sdp sub-problems to cut Combinatorial explosion! # sub-problems = ( N n), need quick optimal selection of a few sub-problems for generating hyperplanes Assume current sol. X, x with S sub-problem objective f S ( X S, x S ) Order/select top S (to cut off X, x via X S, x S ) by estimated objective improvement on X, ObjImpX(S): ( f S (XS x S ) f S ( X ) S, x S ) ˆf S (Q S, x S ) f S ( X S, x S ) = ˆf S (Q S, x S ) Q S X S, where ˆf S (Q S, x S ) is a fast estimator of f S ( x S, X S ). ( ObjImpX(S) ) Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 9 / 23

17 Generating cutting hyperplanes at given X S, x S for one nd-sdp sub-problem Generating hyperplanes Could generate separating hyperplane tangent to SDP cone. In practice, generate cuts from negative eigenvalues, Qualizza et al. [2012]: 1 x vk T T S v k = x S X S = vk T λ k v k = λ k < 0. Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 10 / 23

18 Data for learning estimator ˆf S (Q S, x S ) Estimator only as good as data is sampled Critical that sample space {Q S, x S } is uniform in important features for any learner to generalize well Features: x S (positioning), eigenvalues {λ i } of Q S (positive definiteness) Data sampling Uniform x S [0, 1] n Uniform Q S elements Uniform {λ i } and orthonormal basis Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 11 / 23

19 Neural network as estimator ˆf S (Q S, x S ) Q S x s = ˆf S (Q S, x S ) Why neural nets? ˆf S is a nonlinear regression mapping (collection of convex surfaces) Neural nets (NN): regression via trained hidden layers, no need to specify model Flexible model + lots of well sampled data low variance and bias Architecture (for 3-5D cases) 3-4 hidden layers with neurons tanh activation (well-scaled with our data) in hidden layers Trained by 5-fold cross-validation on 1M data pts. with scaled conjugate gradient Early stop on low gradient (10 5 ) Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 12 / 23

20 Engineering Non-linear activation function Non-linear activation function in the hidden layers Hyperbolic tangent (tanh) vs. Rectified linear unit (ReLU): tanh faster to train, tanh has a bounded output of [ 1, 1], tanh has a significantly positive derivative on the domain [ 4, 4], tanh is symmetric around 0. Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 13 / 23

21 Are the domain and co-domain bounds okay? Lemma. If all eigenvalues of a square matrix M are bounded within [ m, m] then any element in M is bounded within [ m, m]. Let M R n n with eigenvalues and eigenvectors λ i and v i for i 1, n, and let v ij be the j-th element of v i. Then the absolute value of element M ij on the i-th row and j-th column can be expressed as: M ij = v ki v jk λ k k 1,n v ki v jk λ k k 1,n k 1,n k 1,n ((v 2 ki + v 2 jk )/2) λ k ((v 2 ki + v 2 jk )/2)m = m Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 14 / 23

22 Neural network training (3D case) - Results on.5m test set 3D-SDP trained NN (9 inputs layer + 3 hidden layers x 50 neurons) Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 15 / 23

23 Neural network training (4D case) - Results on.5m test set 4D-SDP trained NN (14 inputs layer + 3 hidden layers x 64 neurons) Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 16 / 23

24 Neural network training (5D case) - Results on.5m test set 5D-SDP trained NN (20 inputs layer + 4 hidden layers x 64 neurons) Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 17 / 23

25 Cut selection in practice BoxQP spar rounds of cuts, n = 3, 100 S sub-problems selected by ObjImpX (S) (black lines) Better bound by few cuts After each round, overall bound improving as ObjImpX (S) across S Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 18 / 23

26 Cut selection in practice BoxQP spar rounds of cuts, n = 3, 100 S sub-problems selected by ObjImpX (S) (black lines) Limits - NN error After a few rounds, as ObjImpX (S) NN error: Incorrect selection of S where ObjImpX (S) < 0 Missed selection of S where ObjImpX (S) > 0 Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 19 / 23

27 Results for different problem sizes/densities (n = 3) Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 20 / 23

28 Conclusion Pluses Offline cut selection Good bounds with few low-dimensional linear cuts Easily integrate SDP-based linear cuts with other cut classes in Branch&Cut Minuses Weaker bounds then full SDP or convex-based relaxations Best complemented by other linear cutting planes (e.g. RLT-based) Limited to low-dimensionality cuts Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 21 / 23

29 References I Anstreicher. Semidefinite programming versus the reformulation linearization technique for nonconvex quadratically constrained quadratic programming. J Glob Optim, 43(2-3):471, Bao, Sahinidis, and Tawarmalani. Multiterm polyhedral relaxations for nonconvex, quadratically constrained quadratic programs. Optim Met Softw, 24(4-5):485, Bao, Sahinidis, and Tawarmalani. Semidefinite relaxations for quadratically constrained quadratic programming: A review and comparisons. Math Prog, 129(1):129, Bonami, Günlük, and Linderoth. Solving box-constrained nonconvex quadratic programs Buchheim and Wiegele. Semidefinite relaxations for non-convex quadratic mixed-integer programming. Math Prog, 141(1-2):435, Chen and Burer. Globally solving nonconvex quadratic programming problems via completely positive programming. Math Prog Comput, 4(1):33, Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 22 / 23

30 References II Dong. Relaxing nonconvex quadratic functions by multiple adaptive diagonal perturbations. SIAM J Optim, 26(3):1962, Misener and Floudas. Global optimization of mixed integer quadratically constrained quadratic programs (MIQCQP) through piecewise linear and edge-concave relaxations. Math Prog B, 136:155, Qualizza, Belotti, and Margot. Linear programming relaxations of quadratically constrained quadratic programs. In Lee and Leyffer, editors, Mixed Integer Nonlinear Programming, page Saxena, Bonami, and Lee. Convex relaxations of non-convex mixed integer quadratically constrained programs: Projected formulations. Math Prog, 130: 359, Sherali and Fraticelli. Enhancing RLT relaxations via a new class of semidefinite cuts. J Glob Optim, 22(1-4):233, Zheng, Sun, and Li. Convex relaxations for nonconvex quadratically constrained quadratic programming: matrix cone decomposition and polyhedral approximation. Math Prog, 129(2):301, Baltean-Lugojan, Misener et al. Online cuts by offline selection 2018/03/07 23 / 23