BME - 2011 Cone Linear Optimization (CLO) From LO, SOCO and SDO Towards Mixed-Integer CLO Tamás Terlaky George N. and Soteria Kledaras 87 Endowed Chair Professor. Chair, Department of Industrial and Systems Engineering Lehigh University
Outline Motivation: Data uncertainty in linear inequalities-robust LO Data uncertainty in linear inequalities Ellipsoidal uncertainty set norm constraint Eigen- and singular value optimization problems Relaxing integer variables Conic Linear Optimization (CLO): General Convex Cones Second Order Conic Optimization (SOCO) Portfolio optimization as SOCO Semidefinite Optimization (SDO) Interior Point Algorithms for SOCO and SDO: MISOCO: Mixed Integer Second Order Conic Optimization Sisyphus got stuck with a suboptimal solution; don t let it happen to you! http://www.optimize.com 1
Robust Linear Optimization Classic Polyhedral (scenario) approach (P ) min c T x a j b j n j i=1 s.t. a T j x b j 0 j n j = ai j b i j i=1 λ i j ai j b i j T x 1 λ i j n j i=1 0 Let (a j, b j ) be uncertain, it is coming from a polyhedral set (e.g. convex combination of scenario data points): λ i j = 1, λ i j 0 for all n j i=1 The inequality a T j x b j must be true for all possible values of (a T j, b j): λ i j = 1, λ i j 0 Infinitely many constraints! iff [a i j] T x b i j 0 for i = 1,..., n Finally the problem stays linear as: j (RP ) min c T x s.t. [a i j ]T x b i j 0 for i = 1,..., n j j Disadvantages: Huge number of linear inequalities Polyhedral uncertainty set not realistic. 2
Robust Linear Optimization a0 j b 0 j (P ) min c T x s.t. a T j x b j 0 j a j = a0 j + P u b u IRk, u T u 1 j T + P u x 1 b 0 j Let (a j, b j ) be uncertain, it is coming from an ellipsoid (e.g. level set of a distribution): The inequality a T j x b j must be true for all possible values of (a T j, b j): 0 u : u T u 1 iff [a 0 j ] T x b 0 j + min (P u)t x u T u 1 1 0 [a 0 j ] T x b 0 j P T x 1 0 2 This is a nondifferentiable norm constraint: (See second order cones) P T x [a 0 j 1 ]T x b 0 j. 2 Single nonlinear, norm-constraint! 3
Eigenvalue Optimization Given n n symmetric matrices A 1,..., A m. Problem: Find a nonnegative combination of the matrices that has the maximal smallest eigenvalue. Solution: max λ m i=1 A i y i λi is positive semidefinite y i 0 i = 1,..., m Problem: Find a nonnegative combination of the matrices that has the smallest maximal eigenvalue. m λi A i y i Solution: min λ i=1 is positive semidefinite y i 0 i = 1,..., m The semidefiteness constraint is not differentiable, not easy to calculate when formulated by explicit functions, e.g., min-eigenvalue, determinant (of minors). See Semidefinite Optimization. 4
Relaxing Binary Variables Given z 1,..., z n binary, i.e., {0, 1} variables with other, e.g., linear constraints. Problem: Find convex, continuous relaxations of the binary constraints. Old solution: Let 0 z i 1 for all i = 1,..., n and use branch and bound, branch and cut schemes. New opportunity to get tighter relaxations: Let x i = 2z 1 1 2 for all i = 1,..., n. Thus gives x i as a { 1, 1} variable. Now n = n x 2 i = x T x = Tr(x T x) = Tr(xx T ) = Tr(X), i=1 where X = xx T is a rank-1 positive semidefinite matrix with diag(x) = e. Thus, for x i { 1, 1} i we may use the relaxation: X ii = 1 i and X is positive semidefinite The semidefiteness constraint is not differentiable, not easy to calculate when formulated by explicit functions, e.g., min-eigenvalue, determinant (of minors). See Semidefinite Optimization. 5
Stability of Optimal Power Flow The stability of the Power Flow is ensured by a lower bound on the singular value of the Jacobian J pf of the power flow equations: σ min (J pf ) σ cpf equivalently λ min (J pf Jpf T ) σ cpf by substitution X J pf Jpf T = 0 and X σ cpf I is positive semidefinite Again a semidefinite constraint! One may want maximize σ cpf. 6
New/Old Convex Optimization Problems Cone Linear Optimization Problems Primal-dual pair of CLO problems is given as (P ) min c T x (D) max b T y s.t. Ax b C 1 s.t. c A T y C 2 x C 2 y C1, where b, y IR m, c, x IR n, A : m n matrix, C 1, C 2 are convex cones and Ci = {s IRn : x T s 0, x C i } are the dual cones for i = 1, 2. These are solvable efficiently (in polynomial time) by using Interior Point Methods. LO is based on polyhedral cones. Be careful! Perfect duality, strict complementarity lost. Are all convex cones good??? NOT 7
Self-dual embedding of CLO min θβ Ax κb +θ b z = 0 A T y +κc θ c s = 0 b T y c T x +θα ρ = 0 b T y + c T x κα ν = β z C 1, x C 2, y C1, s C 2, ρ 0, θ 0, ν 0, κ 0, where b := b + z 0 Ax 0, c := c s 0 A T y 0, α := 1 + c T x 0 b T y 0, β := (x 0 ) T s 0 + (y 0 ) T z 0 + 2. x 0 int(c 2 ), z 0 int(c 1 ), y 0 int(c 1 ), s0 int(c 2 ), θ0 = ρ 0 = κ 0 = ν 0 = 1 interior feasible; the embedding problem is self-dual; the central path exists and the duality gap equals 2θβ; the central path leads to a maximally complementary solution, in the interior of the optimal set; ANY feasible IPM is applicable and yield an MC optimal solution. This is our self-dual model!!!! 8
Self-Dual Embedding of CLO results For (P ) and (D) we distinguish the following possibilities: (I) A primal-dual optimal pair (x, y) exists with 0 duality gap (c T x b T y = 0); (IIa) a primal ray exists (x C 2 such that Ax C 1 and c T x < 0), the primal is either infeasible or unbounded; (IIb) a dual ray exists (y C 1 such that AT y C 2 and bt y > 0), the dual is either infeasible or unbounded; (III) no optimal pair with zero duality gap exists, and neither (P ) nor (D) has a ray. Theorem 1. For the self-dual embedding problem let a maximally complementary solution (y, x, κ, θ, z, s, ρ, ν ) be given. Then: (i) if κ > 0 then case (I) holds; (ii) if κ = 0 and ρ > 0 and either case (IIa) or case (IIb) holds. (iii) if κ = ρ = 0 then case (III) holds. 9
New Convex Optimization Problems Second Order Conic Optimization (SOCO) The second order cone in IR n is defined as S2 n := x IRn : x 2:n = n x 2 i x 1 i=2. The name ice cream cone is coming from the 3-dimensional shape of the cone. The second order cone is self-dual: (S n 2 ) = S n 2. Optimization problems, where cones C 1 and C 2 are polyhedral or products of second order cones, are second order cone optimization (SOCO) problems. Significance Norm minimization, robust optimization, quadratic, and thus portfolio optimization. 10
SOCO Optimality The primal-dual SOCO problem is defined as (SP ) min c T x s.t. Ax = b, (SD) max b T y s.t. A T y + s = c x k j=1 Sn j 2 s k j=1 Sn j 2. x T = ((x 1 ) T,..., (x j ) T,..., (x k ) T ) IR n ; and s T = ((s 1 ) T,..., (s j ) T,..., (s k ) T ) IR n. Ax = b, Optimality: x k j=1 Sn j 2, AT y + s = c, s k j=1 Sn j 2 (x j ) T s j = 0 x j s j = 0 Arr(x j )s j = Arr(s j )x j = 0 j Here we have used the notation: u v = u T v u 1 v 2:n + v 1 u 2:n and Arr(u) = u 1 u 2... u n u 2 u 1.... u n u 1 11
Notes on SOCO Convex (conic) optimization problem with vectors Vector calculus not associative Duality gap may exists (next page) Strong duality with interior point (Slater) condition Second order cones cannot be combined into larger second order cones, i.e., S n 1 2 Sn 2 2 Sn 1+n 2 2 Generalization of LO: S 1 2 = IR1 + Rotated cone x 2:n 2 x 0 x 1, x 0, x 1 0 n ( ) x 2 x0 i + x 2 ( 1 x0 x 1 2 2 i=2 Efficiently solvable by IPMs. ) 2 12
SOCO: Duality gap example Primal Problem (SP ) min x 2 s.t. x 1 x 3 = 0, x 2 2 + x2 3 x 1 (SD) max 0 y s.t. y + s 1 = 0 0 y + s 2 = 1 y + s 3 = 0 s 2 2 + s2 3 s 1 Dual Problem Primal Optimal solutions: x 1 = x 3 0; x 2 = 0 Dual problem is infeasible!! Slater not satisfied = Zero duality gap may not hold 13
Portfolio optimization SOCO Portfolio optimization: Let A : m n, b IR m, c IR n and Q : n n positive semidefinite matrix, e = (1,..., 1) T IR n, p IR. min c T x + 1 2 xt Qx s.t. Ax = b, e T x = p, x 0. Towards second order optimization: Cholesky factorization Q = 2L T L implying x T Qx = 2(Lx) T (Lx) = z T z with z = Lx. c T x + 1 2 xt Qx ζ z T z ζ c T x = ( ζ c T x + 1 Let: z 0 := ζ ct x+1 2 and z n+1 = ζ ct x 1 2, then 2 ) 2 z z n+1 ( ζ c T x 1 2 z 0. ) 2 14
Portfolio optimization SOCO variants The original portfolio problem maybe written in SOCO form as: min s.t. n+1 i=1 ζ zi 2 z 0, Ax = b, e T x = p, x 0, Lx z = 0, z 0 = ζ ct x + 1, z n+1 = ζ ct x 1. 2 2 Another portfolio problem in SOCO form as: min c T x + τ s.t. z τ, Lx z = 0, Ax = b, e T x = p, x 0. 15
New Convex Optimization Problems Semidefinite Optimization I The semidefinite cone in IR n n is defined as S n := { X IR n n : X = X T, z T Xz 0 z IR n} i.e. the matrices X are symmetric and positive semidefinite, denoted as X 0. The semidefinite cone is self-dual: (S n ) = S n. Optimization problems where the cones C 1 and C 2 are either polyhedral, second order or semidefinite cones are called semidefinite optimization (SDO) problems. 16
New Convex Optimization Problems Semidefinite Optimization Let A i, i = 1,, n and C, X be n n symmetric real matrices, b, y IR m and let Tr( ) denote the trace of a matrix. The primal-dual SDO problem is defined as (SP ) min Tr(CX) s.t. Tr(A i X) b i 0, i X 0 (SD) max b T y s.t. Optimality: Tr(CX) b T y = Tr(XS) = 0 XS = 0 Significance C m i=1 A i y i 0 y 0. Robust optimization, eigenvalue and singular value optimization Linear matrix inequalities, trust design Convex relaxation of nonconvex/integer problems 17
Notes on SDO Convex (conic) optimization problem with matrices Matrix calculus not commutative Product of symmetric matrices is not smmetric Duality gap may exists Strong duality with interior point (Slater) condition Semidefinite cones can be combined into larger semidefinite cones, i.e., S n 1 S n 2 S n 1+n 2 Generalization of LO: S 1 = IR 1 + diagonal matrices Not proper generalization of SOCO: x S n 2 Arr(x) Sn, BUT arrow-head structure cannot be preserved Efficiently solvable by IPMs. 18
LO Linear Optimization v/s Conic LO Conic LO linear objective linear equality constraints linear inequality constraints perfect duality strictly complementary opt.sol. Euclidean linear algebra x T s = 0 xs = 0 linear objective linear equality constraints conic inequality constraints perfect duality only with IPC maximally complementary opt.sol. matrix and Jordan algebra x T s = 0 x s = 0 (SOCO) Tr(XS) = 0 XS = 0 (SDO) Tr(XS) = 0 X 1 2SX 1 2 = S 2XS 1 1 2 = 0 (P XP T ) 1 2(P T SP 1 )(P XP T ) 1 2 = µi (P SP T ) 1 2(P T XP 1 )(P SP T ) 1 2 = µi 19
Solvability of CLO problems Use IPMs Classic Linear Optimization Large scale LO problems are solved efficiently. High performance packages, like (CPLEX, GuRoBi, XPRESS-MP, MOSEK) offer simplex and interior point solvers as well. Problems solved with 10 7 variables. SOCO and SDO Polynomial solvability established. Traditional software is unable to handle conic constraints. Specialized software is developed. (SeDuMi, SDPA, SDPT3, CSDP, DSDP, SDPpack, MOSEK etc.) SOCO: MOSEK - commercial SDO: SDPA, CSDP, DSDP LO-SOCO-SDO: SeDuMi, SDPT3 SOCO: Problems solved with O(10 6 ) variables. SDO: solved with O(10 4 ) dimensional matrices. http://sedumi.ie.lehigh.edu 20
Complexity of IPMs for LO Method Practice Large update Small update θ adaptive 1 1/100 1/ n Iter. bound max 100 O(ϑ log ϑ ϵ ) O( ϑ log ϑ ϵ ) Performance Efficient Efficient Very poor Almost constant (< 100) number of iterations in practice! CLO ϑ = cost/iteration LO # of variables O(n 3 ) sparse SOCO # of second order cones O(n 3 )+update SDO dimension of matrix X O(n 2 m 3 ) dense 26
Several IPM Solvers for CLO Problems What made this major advance possible? Advances in Computers and Software Computers processor speed memory disk space floating point arithmetic architecture (cash...) Software component/algorithms presolve LINEAR ALGEBRA sparse factorizations symmetric square root IPMs, predictor-corrector dense and sparse versions SeDuMi, SDPT3 SDPA-xxx tuned to all three cones CSDP, DSDP tuned to SDO only MOSEK, CPLEX are commercial solves for SOCO. Parallel implementations exist coming. MODELING languages YALMIP (Löfberg); CVx (Boyd/Grant). 27
Further notes Norm and convex quadratic (including portfolio) optimization prob s can be solved with almost the same efficiency as LO. Efficient tools to eigenvalue, singular value optimization, LMI s CLO based approximation algorithms for nonconvex and combinatorial optimization problems. Lots of activity in exploring special structure of conic problems and developing modeling systems that support conic formulation First HPC-massively parallel implementations Cheap first order methods for very large scale SDO problems. Warm start and decomposition/cutting plane algorithms. http://sedumi.ie.lehigh.edu Now: Adding logarithmic objective. 28
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work Disjunctive Conic Cuts for Mixed Integer Second Order Cone Optimization (MISOCO) Tamás Terlaky Joint work with: Pietro Belotti, Julio C. Góez, Imre Pólik, Ted Ralphs With thanks for the AFOSR Award #: FA9550-10-1-0404
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work CONTENTS Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work MIXED INTEGER SECOND ORDER CONE OPTIMIZATION (MISOCO) minimize: c T x subject to: Ax = b x K x Z d R n d, (MISOCO) where, A R m n, c R n, b R m L n = {x x 1 x 2:n } K = L n 1 1 Ln k k Rows of A are linearly independent
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work OBJECTIVES Obtain the convex hull after applying a linear disjunction to a Mixed Integer Second Order Conic Optimization (MISOCO) problem. Design Disjunctive Conic Cuts for MISOCO.
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work PREVIOUS WORK Atamtürk and Narayanan (2010), conic cuts for general MISOCO problems. Drewes (2009), nonlinear cuts for 0-1 MISOCO problems. Krokhmal and Soberanisin(2010), Drewes (2009), Vielma et al. (2008), branch and bound algorithm based on linear outer approximations for Second Order Cones. Drewes (2009), Atamtürk and Narayanan (2009), lifting techniques for MISOCO problems. Çezik and Iyengar (2005), cuts for mixed 0-1 conic programming. Stubbs and Mehrotra (1999), lift-and-project method for 01 mixed convex programming.
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work APPLICATIONS Turbine balancing problems can be modeled as MISOCOs, White (1996). The euclidean Steiner tree problem can be formulated as a MISOCO, Fampa and Maculan (2004) Computer Vision and Pattern Recognition, Kumar, Torr, and Zisserman (2006). Cardinality-constrained portfolio optimization problems, Bertsimas and Shioda (2009).
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work SINGLE CONE PROBLEM Let us consider the special case: minimize: c T x subject to: Ax = b x L n (MISOCO) x Z d R n d, This problem has a single second order cone All the variables are in the single second order cone
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work STEP 1: SOLVE THE RELAXED PROBLEM Find the optimal solution x for the continuous relaxation of the MISOCO problem minimize: 3x 1 +2x 2 +2x 3 +x 4 subject to: 9x 1 +x 2 +x 3 +x 4 = 10 (x 1, x 2, x 3, x 4 ) L 4 x 4 Z. Relaxing the integrality constraint we get the optimal solution: x = (1.36, 0.91, 0.91, 0.45), with and optimal objective value: z = 0.00.
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work STEP 2: FIND A DISJUNCTION a T x β a T x β VIOLATED BY x = (1.36, 0.91, 0.91, 0.45) The disjunction x 4 1 x 4 0 is violated by x 1.5 1 0.5 x 4 0 0 0.5 x 4 1 1.5 2 2.5 3 2 x 0 3 Relaxed optimal solution x 4 1 2 2 1.5 1 0.5 0 0.5 1 1.5 2 x 2
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work STEP 3: APPLY THE DISJUNCTION AND CONVEXIFY min: 3x 1 +2x 2 +2x 3 +x 4 s.t: 9x 1 +x 2 +x 3 +x 4 = 10 0.04x 2 0.04x 3 3.56x 4 +x 5 = 10.14 6.28x 2 6.28x 3 +0.14x 4 +x 6 = 1.65 6.36x 2 6.36x 3 +x 7 = 0 (x 1, x 2, x 3, x 4 ) L 4 (x 5, x 6, x 7 ) L 3 x 4 Z. The constraints in red represent the disjunctive conic cut. An integer optimal solution is obtained after adding one cut: x = (1.32, 0.93, 0.93, 0.00, 10.06, 10.06, 0.00), with and optimal objective value: z = 0.24.
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work CONVEX HULL OF THE INTERSECTION OF A DISJUNCTION AND A CONVEX SET Consider a closed convex set E and two halfspaces H 1 = {x R n : a x α} and H 2 = {x R n : b x β}, such that they do not intersect inside E, i.e., E H 1 H 2 =. Denote H = 1 = {x Rn : a x = α}, and H = 2 = {x Rn : b x = β}. If a convex cone K s.t. H = 1 E = K H= 1 and H= 2 E = K H= 2 are bounded, then conv(e (H 1 H 2 )) = E K.
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work INTERSECTION OF AN AFFINE SPACE AND A SECOND ORDER CONE Consider an affine subspace H = {x Ax = b} and x 0 H. Let H A be s.t rank([h, A T ]) = n, & columns of H are orthonormal. We can write H = {x x = x 0 + Hz, z R}. Then, there exist a matrix Q R n m n m, q R n m, ρ R, s.t. H L n = {y x = x 0 + Hz with z Qz + 2q z + ρ 0}. Further, Q has at most one negative eigenvalue. Define a quadric as the set Q = {z z Qz + 2q z + ρ 0}, which we also denote as Q = (Q, q, ρ).
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work UNI-PARAMETRIC FAMILY OF QUADRICS Q(τ) Given two hyperplanes H 1 = {z a 1 z = α 1} and H 2 = {z a 2 z = α 2}. Let Q = (Q, q, ρ) be a quadric where Q is positive definite. The family of quadrics having the same intersection with H 1 and H 2 as the quadric Q is parametrized by τ R as Q(τ), where Q(τ) = Q + τ a 1a T 2 + a 2a T 1 ω q(τ) = q τ α 2a 1 + α 1 a 2 ω ρ(τ) = ρ + 2τ α 1α 2 ω, where ω = { 2a T 1 a 2 if a T 1 a 2 0 1 if a T 1 a 2 = 0.
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work UNI-PARAMETRIC FAMILY OF QUADRICS Q(τ) Sequence of quadrics x Q(τ)x + 2q(τ) x + ρ(τ) 0, for 106.863 τ 1617
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work UNI-PARAMETRIC FAMILY OF QUADRICS Q(τ) Range Description τ = 8.9946, τ = 1617 Paraboloid τ = 106.863, τ = 9.581 Cones 8.9946 < τ < 1617 Ellipsoids τ > 1617 Two sheets hyperboloids 106.863 < τ < 8.9946 One sheet hyperboloids τ < 106.863, Two sheets hyperboloids Behavior of the quadrics for different ranges of τ
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work THE DISJUNCTIVE CONIC CUT FOR PARALLEL DISJUNCTIONS Theorem Let A 1 = {x a 1 x = α 1} and A 2 = {x a 2 x = α 2}, be two parallel hyperplanes where a 1 = γa 2. The disjunctive conic cut is the quadric generated by Q(ˆτ) = (Q(ˆτ), q(ˆτ), ρ(ˆτ)), where ˆτ is the larger root of equation q(τ) Q(τ)q(τ) ρ(τ) = 0.
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work y y OUR DISJUNCTIVE CONIC CUT IS NEW Atamtürk and Narayanan designed a conic mixed integer rounding inequality. Our disjunctive conic cut is different, sometimes stronger. Consider the problem: minimize: x y subject to: x +y +2t = α (x 4 3 )2 + (y 1) 2 t x Z, y R Relaxed Optimal Solution 1.5 1 Conic mixed integer 0.5 rounding inequality 0 0.5 Disjunctive conic cut 1 1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 x Case with α = 4.66 4 Relaxed Optimal Solution 3 2 1 In this particular example our disjunctive conic cut is stronger. 0 1 Disjunctive conic cut Conic mixed integer rounding inequality 2 1 0 1 2 3 4 x Case with α = 8
Introduction Disjunctive Conic Cuts for MISOCO Formulation of a Disjunctive Conic Cut Conclusions and Future Work CONCLUSIONS We developed a new disjunctive conic cut for MISOCO. It is algebraically simple to find the disjunctive conic cut for MISOCO problems. Next steps Develop disjunctive conic cuts for the case when Q is not positive definite. Develop a prototype branch-and-cut framework for solving MISOCO problems using disjunctive conic cuts. Develop strategies which, and how many disjunctive conic cuts to generate when several cones are in the problem. Develop a comprehensive branch-and-cut framework for solving MISOCO problems using disjunctive conic cuts. Far future work Develop disjunctive conic cuts for Semidefinite Optimization.