Solving conic optimization problems with MOSEK

Transcription

1 Solving conic optimization problems with MOSEK Erling D. Andersen MOSEK ApS, Fruebjergvej 3, Box 16, 2100 Copenhagen, Denmark. WWW: August 8,

2 2 / 42

3 Outline Outline MOSEK What is MOSEK. Conic problems MOSEK can solve. Interfacing with MOSEK. Algorithms employed in MOSEK. Implementation.. Summary and conclusions. 3 / 42

4 MOSEK Outline MOSEK MOSEK is a software package for large scale optimization. Version 1 released April Version 7 released May Linear and conic quadratic (+ mixed-integer). Conic quadratic optimization (+ mixed-integer). Semi-definite optimization starting from version 7. Convex(functional) optimization. C, JAVA,.NET and Python APIs. AMPL, AIMMS, GAMS, MATLAB, and R interfaces. Free for academic use. See 4 / 42

5 5 / 42

6 The primal problem The primal problem min nj=1 c j x j + n j=1 Cj, X j st li c n j=1 a ij x j + n j=1 Āij, X j u c i, i = 1,...,m, lj x x j u x j, j = 1,...,n, x K, X j 0, j = 1,..., n. Explanation: A,B := tr(a T B). x j is a scalar variable. Xj is a square matrix variable. K represents conic quadratic constraints. Xj 0 represents X j = X T j and X j is PSD. C j andāj are assumed to be symmetric. 6 / 42

7 The primal problem An example ofk: { x 1 x 27 x 6 andx 3 x 8 x 9 }. Models the quadratic cones. (Both normal and rotated variant are allowed.) A scalar variable can only be member of one sub cone. 7 / 42

8 The primal problem Employs a primal formulation. Semi-definite addition is an extension on the linear and conic quadratic case. Makes it easy to extend the software. LMIs (=dual formulation) C j i Ā ij y i can be formulated. Requires explicit slacks. Normally C j andāij are sparse (is exploited). Dense but low rank in C j andāij can be exploited. 8 / 42

9 9 / 42

10 What is available What is available The MATLAB toolbox interface Fusion: A new interface Optimizer API. Matrix orientated. Flexible and efficient. Efficiency is more important than ease of use. Available in C, Java,.NET and Python. R and MATLAB toolbox. Matrix orientated. Fusion. Deals with variables and constraints. Good compromise between efficiency and ease of use. Available for Java, MATLAB,.NET and Python. 10 / 42

11 What is available The MATLAB toolbox interface Fusion: A new interface CVX. Yalmip. 11 / 42

12 The MATLAB toolbox interface What is available The MATLAB toolbox interface Fusion: A new interface An example: min x 1 + st x 1 + x 2 +x , X x 1 x 2 2 +x 3 2, X 1 0., X 1, X 1 = 1, = 0.5, 12 / 42

13 Source code What is available The MATLAB toolbox interface Fusion: A new interface prob.c = [1, 0, 0]; prob.bardim = [3]; prob.barc.subj = [1, 1, 1, 1, 1]; prob.barc.subk = [1, 2, 2, 3, 3]; prob.barc.subl = [1, 1, 2, 2, 3]; prob.barc.val = [2.0, 1.0, 2.0, 1.0, 2.0]; subj: Variable index. subk,subl: Local index i.e. ( C j ) kl. Only the lower triangular part is specified. 13 / 42

14 What is available The MATLAB toolbox interface Fusion: A new interface prob.a = sparse([1, 2, 2],... [1, 2, 3],... [1, 1, 1], 2, 3); prob.bara.subi = [1, 1, 1, 2, 2, 2, 2, 2, 2]; prob.bara.subj = [1, 1, 1, 1, 1, 1, 1, 1, 1]; prob.bara.subk = [1, 2, 3, 1, 2, 3, 2, 3, 3]; prob.bara.subl = [1, 2, 3, 1, 1, 1, 2, 2, 3]; prob.bara.val = [1, 1, 1, 1, 1, 1, 1, 1, 1]; prob.blc = [1.0, 0.5]; prob.buc = [1.0, 0.5]; 14 / 42

15 What is available The MATLAB toolbox interface Fusion: A new interface [r, res] = mosekopt( symbcon ); prob.cones.type = [res.symbcon.msk_ct_quad]; prob.cones.sub = [1, 2, 3]; prob.cones.subptr = [1]; [r,res] = mosekopt( minimize,prob); X = zeros(3); X([1,2,3,5,6,9]) = res.sol.itr.barx; X = X + tril(x,-1) ; x = res.sol.itr.xx; 15 / 42

16 What is available The MATLAB toolbox interface Fusion: A new interface Pros: Easy to explain. Efficient. Cons: Problem must be serialized. Small problem changes requires error prone changes to the code. Possible solutions: CVX (MATLAB only currently). YALMIP (MATLAB only). 16 / 42

17 Fusion: A new interface What is available The MATLAB toolbox interface Fusion: A new interface The basic idea of Fusion: min st c T x A i x+b i K i i. A linear objective. Several linear constraints involving the variables can be formulated. K i is a convex set. K i cannot have an arbitrary structure e.g. must be symmetric cones. Several variables (scalar and symmetric matrix). Not just one big giant matrix. (Contrast with SeDuMi). 17 / 42

18 Example: The nearest correlation matrix What is available The MATLAB toolbox interface Fusion: A new interface Given a symmetric matrixathen the nearest correlation matrix is given by X = arg min A X F. X 0,diag(X)=e Conic formulation exploiting symmetry of A X: min t st z 2 t, svec(a X) = z, diag(x) = e, X 0. Simple formulation on paper, but cumbersome to formulate in standard form as in MOSEK, SeDuMi, SDPA, / 42

19 Python Fusion def svec(e): N = e.get_shape().dim(0) S = Matrix.sparse(N * (N+1) / 2, N * N, range(n * (N+1) / 2), [ (i+j*n) for j in xrange(n) for i in xrange(j,n) ], [ (1.0 if i == j else 2**(0.5)) for j in xrange(n) for i in xrange(j,n) ]) return Expr.mul(S,Expr.reshape( e, N * N )) def nearestcorr(a): M = Model("NearestCorrelation") N = len(a) # Setting up the variables X = M.variable("X",Domain.inPSDCone(N)) # t > z _2 tz = M.variable("tz", Domain.inQCone(N*(N+1)/2+1)) t = tz.index(0) z = tz.slice(1,n*(n+1)/2+1) # svec (A-X) = z M.constraint( Expr.sub(svec(Expr.sub(DenseMatrix(A),X)), z), Domain.equalsTo(0.0) ) # diag(x) = e for i in range(n): M.constraint( X.index(i,i), Domain.equalsTo(1.0) ) # Objective: Minimize t M.objective(ObjectiveSense.Minimize, t) M.solve() 19 / 42

20 Fusion summary What is available The MATLAB toolbox interface Fusion: A new interface Easy to add and delete variables and constraints. Not a modeling language but an easy to use API. Designed for efficiency yet still convenient to use. Available for Java, MATLAB,.NET, and Python. An alternative to Yalmip and CVX. 20 / 42

21 21 / 42

22 Simplified notation Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details Primal problem: where min c T x st Ax = b, x 0, x 0 represents a mixture of linear, conic quadratic and semi-definite variables. Dual problem: max b T y st A T y +s = c, s / 42

23 The homogeneous model Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details Simplified homogeneous model: Notes: s 0 κ + 0 A T c A 0 b c T b T 0 (x;τ),(s;κ) 0. (x,y,s,κ,τ) = 0 is a feasible solution. All solutions satisfies: x T s+κτ = 0. x y τ = / 42

24 Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details Ifτ > 0, κ = 0 then(x,y,s)/τ is a primal-dual optimal solution, Ax = bτ, A T y +s = cτ, x T s = 0. Ifκ > 0, τ = 0 then the primal and/or dual is infeasible, Ax = 0, A T y +s = 0, c T x b T y < 0, Primal is infeasible ifb T y > 0, dual is infeasible ifc T x < 0. Comments: Find a solution such thatτ > 0 or κ > 0 if it exists. It exist under suitable regularity conditions. 24 / 42

25 The central path Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details Let (x (0) ;τ (0) ),(s (0) ;κ (0) ) 0 then the central path is defined as: s 0 A T c x A T y (0) +s (0) cτ (0) 0 + A 0 b y = γ Ax (0) +bτ (0) κ c T b T 0 τ c T x (0) b T y (0) +κ (0) x s = γµ (0) e, τκ = γµ (0), whereγ [0,1],µ (0) = (x(0) ) T s (0) +κ (0) τ (0) n+1 > 0 and e = svec(i n1 ). svec(i np ). 25 / 42

26 Notation Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details For symmetric matricesu S n we define svec(u) = (U 11, 2U 21,..., 2U n1,u 22, 2U 32,..., 2U n2,...,u n with inverse operation smat(u) = u 1 u 2 / 2 u n / 2 u 2 / 2 u n+1 u 2n 1 / 2... u n / 2 u n 1 / 2 u n(n+1), and a symmetric product u v := (1/2)(smat(u)smat(v) + smat(v)smat(u)). 26 / 42

27 The search direction Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details Ad x +bd τ = (γ 1)( Ax (0) +bτ (0) ) A T d y +d s cd τ = (γ 1)(A T y (0) +s (0) cτ (0) ) c T d x b T d y +d κ = (γ 1)(c T x (0) b T y (0) +κ (0) ) d x +Π (0) d s = γµ (0) (s (0) ) 1 x (0) d τ +τ (0) /κ (0) κ = γµ (0) /τ (0) κ (0) Residuals reduction: A(x+αd x )+b(τ +αd τ ) = (1 α(1 γ))( Ax (0) +bτ (0) ). Complementarity: (x (0) +αd x ) T (s (0) +αd s )+(κ (0) +α κ)(τ (0) +α τ) = (1 α(1 γ))((x (0) ) T s (0) +κ (0) τ (0) ). }{{} <1 Polynomial complexity for suitable choice ofπ, γ and α. 27 / 42

28 The scaling matrix Π is Nesterov-Todd scaling in MOSEK. Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details 28 / 42

29 Solving for the search direction Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details After block-elimination, we get a2 2 system: AΠA T d y +(AΠc b)d τ =, (AΠc+b)d y +(c T Π (0) c+κ (0) /τ (0) )d τ =. We factoraπa T = LL T and eliminate d y = L T L 1 ( (AΠc b)d τ + ). Then ( ) (AΠc+b) T L T L 1 (AΠc b)+(c T Πc+κ/τ) d τ = An extra (insignificant) solve compared to an infeasible method. 29 / 42

30 Forming the Schur complement Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details Schur complement computed as a sum of outer products: AΠA T = = p k=1 a T a T 1p.. a T m1... a T mp P T k ( P k A :,k Π k A T :,kp T k Π 1... ) P k, Π p a a m1.. a 1p... a mp wherep k is a permutation of A :,k. Complexity depends on choice ofp k. We compute onlytril(aπa T ), so by nnz(p k A :,k ) 1 nnz(p k A :,k ) 2... nnz(p k A :,k ) p, get a good reduction in complexity. More general than SDPA. 30 / 42

31 Forming the Schur complement Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details GivenΠ k = R k R T k then a T ikπ k a jk = R T ka ik R k,r T ka jk R k Using the idea in SeDuMi then = R k R T ka jk R k R T k = ˆMM +M T ˆMT, i.e., a symmetric (low-rank) product. A ik,r k R T ka jk R k R T k Sparse case: Few elements of ˆMM +M T ˆMT needed. Dense case: Many elements of ˆMM +M T ˆMT needed.. May exploit low-rank structure in dense A ik later. 31 / 42

32 Termination Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details Optimal case: A x(k) +b ε τ (k) p (1+ b ), A T x(k) + s(k) c ε τ (k) τ (k) d (1+ c ), ( ) ( min (x (k) ) T s (k) +τ (k) κ (k), ct x (k) bt y (k) ε (τ (k) ) 2 τ (k) τ (k) g max 1, Primal infeasible case: ε i b T y (k) > b max(1, c ) A T y (k) +s (k) ), τ (k) ct x (k) Dual infeasible case: ε i c T x (k) > c max(1, b ) Ax (k) All εs are user specified. 32 / 42

33 Practical details Simplified notation The homogeneous model The central path Notation Schur complement Termination Practical details Employs Mehrotra s predictor-corrector technique. Employs a presolve which can reduce problem size and complexity significantly. Primarily for the linear part. Scale the problem to improve conditioning of a problem. Exploit sparse linear algebra when applicable. Exploit tuned dense BLAS when applicable. Fully parallelized for linear and conic quadratic cases. 33 / 42

34 34 / 42

35 Benchmark environment Benchmark environment Optimized problems Result Profile Windows server CPU: Intel XEON Cores. 3.5 GHz. MOSEK: Only 2 threads allowed. Test problems: Taken from: Exclude the smallest and largest with respect to running time. 35 / 42

36 Optimized problems Benchmark environment Optimized problems Result Profile Name # con. # cone # var. # mat. var. G40 mb G40mc G48mc buck butcher cancer cphil foot hand inc inc mater neu3g nonc rabmo reimer rendl ros rose sensor sensor shmup shmup swissroll taha1a taha1b trto vibra yalsdp / 42

37 Result Benchmark environment Optimized problems Result Profile Name P. obj. # sig. fig. # iter time(s) G40 mb e G40mc e G48mc e buck e butcher e cancer e cphil e foot e hand e inc e inc e mater e neu3g e nonc e rabmo e reimer e rendl e ros e rose e sensor e sensor e shmup e shmup e swissroll e taha1a e taha1b e trto e vibra e yalsdp e / 42

38 Profile Benchmark environment Optimized problems Result Profile In % Name Time Prslv Fac. set. Shur. Chol Upd. G40 mb G40mc G48mc buck butcher cancer cphil foot hand inc inc mater neu3g nonc rabmo reimer rendl ros rose sensor sensor shmup shmup swissroll taha1a taha1b trto vibra yalsdp / 42

39 39 / 42

40 Current status Future work References MOSEK has been extended to handel semi-definite problems. A new modeling tool Fusion has been made available. MOSEK can solve large semi-definite problems (usual restrictions apply). 40 / 42

41 Future work Future work References Add warmstart as discussed by Y. Ye. Improve the parallelization (Exploit Intel Phi 1 tera flop card). Implement stability enhancements for the scaling matrix suggested by J. Sturm [1]. Use an iterative method for solution of the Newton equation system to improve numerical stability (not speed). Improve the presolve for the conic part. Exploit dense but low rank structure in C andā. Fusion: C++ variant. (Requires garbage collection like feature). 41 / 42

42 References [1] J. F. Sturm. Avoiding numerical cancellation in the interior point method for solving semidefinite programs. Math. Programming, 95(1): , Future work References 42 / 42