Semidefinite Optimization using MOSEK

Transcription

1 Semidefinite Optimization using MOSEK Joachim Dahl ISMP Berlin, August 23,

2 Introduction Introduction Semidefinite optimization Algorithms Results and examples Summary MOSEK is a state-of-the-art solver for large-scale linear and conic quadratic problems. Based on the homogeneous model using Nesterov-Todd scaling. Next version includes semidefinite optimization. Goalforfirst versionis to beatsedumi. WhySDP inmosek? It is very powerful and flexible for modeling. We wantthesdp woryou developed to beavailableto our customers! Customers have been asing about it for awhile. 2 /28

3 Semidefinite optimization 3 /28

4 Semidefinite optimization Primal and dual problems: minimize p j=1 C j X j subjectto p j=1 A ij X j = b i X j S n j +, maximize b T y subjectto m i=1 y ia ij +S j = C j S j S n j +, wherea ij,c j S n j. Optimality conditions: p A ij X j = b i, j=1 m i=1 y i A ij +S j = C j, X j S j = 0, X j,s j S n j +. EquivalentcomplementaryconditionsifX j,s j S n j + : X j S j = 0 X j S j = 0 X j S j +S j X j = 0. 4 /28

5 Algorithms 5 /28

6 Notation For symmetricmatricesu S n we define svec(u) = (U 11, 2U 21,..., 2U n1,u 22, 2U 32,..., 2U n2,...,u nn ) T with inverse operation smat(u) = u 1 u 2 / 2 u n / 2 u 2 / 2 u n+1 u 2n 1 / 2... u n / 2 u n 1 / 2 u n(n+1)/2, and a symmetric product u v := (1/2)(smat(u)smat(v) + smat(v)smat(u)). 6 /28

7 Notation Let a ij = svec(a ij ), c j := svec(c j ), x j := svec(x j ), s j := svec(s j ), and A := a T a T 1p., c := c 1., x := x 1., s := s 1.. a T m1... a T mp c p x p s p We then have primal and dual problems: minimize c T x subjectto Ax = b x 0, maximize b T y subjectto A T y +s = c s 0. 7 /28

8 The homogeneous model Simplified homogeneous model: s 0 κ + 0 A T c A 0 b c T b T 0 x y τ = x,s 0, τ,κ 0. Note that (x,y,s,κ,τ) = 0isfeasible,and x T s+κτ = 0. Ifτ > 0, κ = 0 then(x,y,s)/τ is aprimal-dualoptimal solution, Ax = bτ, A T y +s = cτ, x T s = 0. Ifκ > 0, τ = 0 theneitherprimalor dualisinfeasible, Ax = 0, A T y +s = 0, c T x b T y < 0, Primalinfeasibleifb T y > 0, dualinfeasibleifc T x < 0. 8 /28

9 The central path Introduction Semidefinite optimization Algorithms Notation Homogeneous model The central path Primal-dual scaling The search direction Schur complement Practical details Results and examples Summary We definethe centralpath as: s 0 κ + 0 A T c A 0 b c T b T 0 x y τ = γ x s = γµ (0) e, τκ = γµ (0), whereγ [0,1],µ (0) = (x(0) ) T s (0) +κ (0) τ (0) n+1 > 0and e = svec(i n1 ). svec(i np ). A T y (0) +s (0) cτ (0) Ax (0) bτ (0) c T x (0) b T y (0) +κ (0) 9 /28

10 Primal-dual scaling Introduction Semidefinite optimization Algorithms Notation Homogeneous model The central path Primal-dual scaling The search direction Schur complement Practical details A primal-dualscalingisdefined byanon-singularr as W (x ) := svec(r T X R 1 ), W T (s ) := svec(r S R T ), preserves the semidefinite cone, x 0, s 0 W (x ) 0, W T (s ) 0, Results and examples Summary the central path, x s = γµe W (x ) W T (s ) = γµe, andthe complementarity: x T s = W (x ) T W T (s ). 10 /28

11 Two popular primal-dual scalings Introduction Semidefinite optimization Algorithms Notation Homogeneous model The central path Primal-dual scaling The search direction Schur complement Practical details Results and examples Summary Helmberg-Kojima-Monteiro(HKM)R 1 = S 1/2 : W (x ) = svec(s 1/2 X S 1/2 ), W T (s ) = e. Nesterov-Todd (NT): R 1 = ( ) 1/2, S 1/2 (S 1/2 X S 1/2 ) 1/2 S 1/2 Satisfies R 1 X R 1 = R S R. Computed as: R 1 = Λ 1/4 Q T L 1, LL T = X, QΛQ T = L T S L whichsatisfies R T X R 1 = R S R T = Λ. 11 /28

12 The search direction Introduction Semidefinite optimization Algorithms Notation Homogeneous model The central path Primal-dual scaling The search direction Schur complement Practical details Results and examples Summary Linearizing the scaled centrality condition W (x ) W T (s ) = γµe and discarding higher-order terms leads to where x +Π s = γµs 1 x, HKM scaling: Π z = svec(x Z S 1 +S 1 Z X )/2, NT scaling: Π z = svec(r T R Z R T R ). 12 /28

13 The search direction Most expensive part of interior-point optimizer is solving: A x b τ = (γ 1)(Ax bτ) = r 1 A T y + s c τ = (γ 1)(A T y +s cτ) = r 2 c T x b T y + κ = (γ 1)(c T x b T y +κ) = r 3 x+π s = γµs 1 x = r 4 κ+κ/τ τ = γµ/τ κ = r 5 Gives constant decrease in residuals and complementarity: A(x+α x) b(τ +α τ) = (1 α(1 γ))(ax bτ) (x+α x) T (s+α s)+(κ+α κ)(τ +α τ) = (1 α(1 γ))(x T s+κτ). }{{} <1 Polynomial complexity for γ chosen properly. 13 /28

14 Solving for the search direction After bloc-elimination,weget a2 2system: AΠA T y (AΠc+b) τ = r y (AΠc b) y +(c T Πc+κ/τ) τ = r τ. We factoraπa T = LL T and eliminate y = L T L 1 ((AΠc+b) τ +r y ). Then ( (AΠc b) T L T L 1 (AΠc+b)+(c T Πc+κ/τ) ) τ = r τ (AΠc b) T L T L 1 r y. An extra(insignificant) solve compared to an infeasible method. 14 /28

15 Forming the Schur complement Schur complement computed as a sum of outer products: AΠA T = = a T a T 1p Π a T m1... a T mp p ( P A :, Π A T ) :, PT P, =1 P T Π p a a m1.. a 1p... a mp wherep is apermutationof A :,. Complexity depends on choiceofp. We computeonlytril(aπa T ), soby nnz(p A :, ) 1 nnz(p A :, ) 2 nnz(p A :, ) p, get a good reduction in complexity. More general than SDPA. 15 /28

16 Forming the Schur complement To evaluate each term a T i Π a j = (R T A ir ) (R T A jr ) = A i (R R T A jr R T ). we follow SeDuMi. Rewrite R R T A jr R T = ˆMM +M T ˆMT, i.e., a symmetric(low-ran) product. A i sparse: evaluate ˆMM +M T ˆMT elementbyelement. A i dense: evaluate ˆMM +M T ˆMT usinglevel3blas. May exploit low-ran structure later. 16 /28

17 Practical details Introduction Semidefinite optimization Algorithms Notation Homogeneous model The central path Primal-dual scaling The search direction Schur complement Practical details Results and examples Summary Additionally, the MOSEK solver uses Mehrotra s predictor-corrector step. solves mixed linear, conic quadratic and semidefinite problems. employs a presolve step for linear variables, which can reduce problem size and complexity significantly. scales constraints trying to improving conditioning of a problem. 17 /28

18 Results and examples 18 /28

19 SDPLIB benchmar Introduction Semidefinite optimization Algorithms Results and examples SDPLIB benchmar Profiling Conic modeling Summary We usethe subsetofsdplib usedbyh. D.Mittelmann for his SDP solver comparison. For brevity we report only iteration count, computation time, and relative gap at termination. Our SDPA-format converter detects bloc-diagonal structure within the semidefinite blocs(sedumi and SDPA does not), so forafewproblemsoursolution time is much smaller. All solversrunonasingle threadon anintel Xeon E32270with4coresat3.4GHz and32gbmemoryusing Linux. 19 /28

20 SDPLIB benchmar MOSEK 7.0 SeDuMi 1.3 SDPA Problem it time relgap it time relgap it time relgap arch e e e-09 control e e e-07 control e e e-06 control e e e-06 equalg e e e-07 equalg e e e-08 gpp e e e-09 gpp e e e-09 hinf e e e-02 maxg e e e-08 maxg e e e-08 maxg e e e-08 mcp e e e-08 mcp e e e-09 qap e e e-05 qap e e e-04 qpg e e e-08 qpg e e e-08 ss e e e-08 theta e e e-09 theta e e e-08 theta e e e-08 theta e e e-08 thetag e e e-08 thetag e e e-07 truss e e e-08 truss e e e-09 relgap = (c T x b T y)/(1+ c T x + b T y ) 20 /28

21 Profiling Profiling results [%] Problem Schur factor stepsize search dir update arch control control control equalg equalg gpp gpp hinf maxg maxg maxg mcp mcp qap qap qpg qpg ss theta theta theta theta thetag thetag truss truss /28

22 Profiling conclusions Introduction Semidefinite optimization Algorithms Results and examples SDPLIB benchmar Profiling Conic modeling Summary FormingAΠA T (Schur)isnot alwaysmost expensive step (as perhaps believed). A good parallelization speedup is possible for computingaπa T. (Factoris alreadyparallelized). The update step (updating variables, neighborhood chec,newnt scaling) isvery cheapforlpand SOCP, butexpensiveforsdp, mainlydueto NT scaling;hkm scaling would give an improvement. Stepsize computations(e.g., stepsize to boundary) are moderately expensive, but hard to speed up. 22 /28

23 Conic modeling Introduction Semidefinite optimization Algorithms Results and examples SDPLIB benchmar Profiling Conic modeling Summary For A S n, thenearest correlationmatrixis X = arg min X S+ n,diag(x)=e A X F. Conic formulation exploiting symmetry of A X: minimize t subjectto z 2 t svec(a X) = z diag(x) = e X 0. Simple formulation on paper, but complicated to formulate in standard form as in SeDuMi, SDPA, /28

24 Conic modeling Introduction Semidefinite optimization Algorithms Results and examples SDPLIB benchmar Profiling Conic modeling Summary For conic modeling you need a modeling tool! Nice MATLAB toolboxes exist (Yalmip, CVX,...), but not for other languages. We developed a modeling API called MOSEK Fusion: A tool forself-dualconicmodeling;no QPor general convex optimization. Idea: create and manipulate linear expressions, and assign them to different cones. Simple but flexible design; scales well for large problems. Available for Python, Java,.NET. Planned versions include MATLAB (soon), C++ (later). Syntax almost identical across platforms. 24 /28

25 Python/Fusion listing for nearest correlation def svec(e): N = e.get_shape().dim(0) S = Matrix.sparse(N * (N+1) / 2, N * N, range(n * (N+1) / 2), [ (i+j*n) for j in xrange(n) for i in xrange(j,n) ], [ (1.0 if i == j else 2**(0.5)) for j in xrange(n) for i in xrange(j,n) ]) return Expr.mul(S,Expr.reshape( e, N * N )) def nearestcorr(a): M = Model("NearestCorrelation") N = len(a) # Setting up the variables X = M.variable("X",Domain.inPSDCone(N)) # t > z _2 tz = M.variable("tz", Domain.inQCone(N*(N+1)/2+1)) t = tz.index(0) z = tz.slice(1,n*(n+1)/2+1) # svec (A-X) = z M.constraint( Expr.sub(svec(Expr.sub(DenseMatrix(A),X)), z), Domain.equalsTo(0.0) ) # diag(x) = e for i in range(n): M.constraint( X.index(i,i), Domain.equalsTo(1.0) ) # Objective: Minimize t M.objective(ObjectiveSense.Minimize, t) M.solve() 25 /28

26 Summary 26 /28

27 Summary Introduction Semidefinite optimization Algorithms Results and examples Summary References SDP solver in MOSEK 7.0 based on self-dual homogeneous embedding and NT scaling. Faster than SeDuMi, comparable with SDPA. Includes Fusion, a powerful tool for self-dual conic modeling. Future directions: Incorporate multithreading. Exploit low-ran structure in data. Possibly implement the HKM direction. Wantto tryit? Contact support@mose.com for a beta version. 27 /28

28 References [1] E.D. Andersen, C. Roos, and T. Terlay. On implementing a primal-dual interior-point method for conic quadratic optimization. Mathematical Programming, 95(2): , [2] Y.E. Nesterov and M.J. Todd. Primal-dual interior-point methods for self-scaled cones. SIAM Journal on Optimization, 8(2): , [3] J.F. Sturm. Implementation of interior point methods for mixed semidefinite and second order cone optimization problems. Optimization Methods and Software, 17(6): , [4] M. Yamashita, K. Fujisawa, and M. Kojima. Implementation and evaluation of sdpa 6.0(semidefinite programming algorithm 6.0). Optimization Methods and Software, 18(4): , [5] Y.Ye,M.J.Todd,and S.Mizuno. AnO( nl)-iterationhomogeneousand self-dual linear programming algorithm. Mathematics of Operations Research, pages 53 67, /28