15. Conic optimization

Transcription

1 L. Vandenberghe EE236C (Spring 216) 15. Conic optimization conic linear program examples modeling duality 15-1

2 Generalized (conic) inequalities Conic inequality: a constraint x K where K is a convex cone in R m we will require that the cone K is proper: closed pointed: K ( K) = {} with nonempty interior: int K ; equivalently, K + ( K) = R m Notation (for proper K) x K y x y K x K y x y int K Conic optimization 15-2

3 Inequality notation we will use a different convention than in EE236B Vector inequalities: for x, y R m x > y, x y denote componentwise inequality x y, x y denote conic inequality for general (unspecified) proper cone K x K y, x K y denote conic inequality for specific K Matrix inequality: for X, Y S p X Y, X Y mean X Y is positive (semi-)definite Conic optimization 15-3

4 Properties of conic inequalities preserved by nonnegative linear combinations: if x y and u v, then αx + βu αy + βv α, β define a partial order of vectors x x x y z implies x z x y and y x imply y = x in general, not a total order (requires that x y or y x for all x, y) Conic optimization 15-4

5 Conic linear program minimize subject to c T x Ax b F x = g A R m n, F R p n ; without loss of generality, we can assume rank(f ) = p, [ A rank( F ] ) = n K is a proper cone in R m for K = R m +, problem reduces to regular linear program (LP) by defining K = K 1 K r, this can represent multiple conic inequalities A 1 x K1 b 1,..., A r x Kr b r Conic optimization 15-5

6 Outline conic linear program examples modeling duality

7 Norm cones K = { (x, y) R m 1 R x y } 1 y x 2 x 1 for the Euclidean norm this is the second-order cone (notation: Q m ) Conic optimization 15-6

8 Second-order cone program minimize c T x subject to B k x + d k 2 B k1 x + d k1, k = 1,..., r Conic LP formulation: express constraints as Ax K b K = Q m 1 Q m r, A = B 1 B 11. B r, b = d 1 d 11. d r B r1 d r1 (assuming B k, d k have m k 1 rows) Conic optimization 15-7

9 Vector notation for symmetric matrices vectorized symmetric matrix: for U S p vec(u) = 2 ( U11, U 21,..., U p1, U 22, U 32,..., U p2,..., U ) pp inverse operation: for u = (u 1, u 2,..., u n ) R n with n = p(p + 1)/2 mat(u) = 1 2 2u1 u 2 u p u 2 2up+1 u 2p 1. u p u 2p 1.. 2up(p+1)/2 coefficients 2 are added so that standard inner products are preserved: tr(uv ) = vec(u) T vec(v ), u T v = tr(mat(u) mat(v)) Conic optimization 15-8

10 Positive semidefinite cone S p = {vec(x) X S p +} = {x R p(p+1)/2 mat(x) } 1 z S 2 = 1 y { (x, y, z) 1 [ x y/ 2 y/ 2 z x ] } 1 Conic optimization 15-9

11 Semidefinite program with A ij, B i S p i minimize c T x subject to x 1 A 11 + x 2 A x n A 1n B 1... x 1 A r1 + x 2 A r2 + + x n A rn B r Conic LP formulation K = S p 1 S p 2 S p r A = vec(a 11 ) vec(a 12 ) vec(a 1n ) vec(a 21 ) vec(a 22 ) vec(a 2n ).. vec(a r1 ) vec(a r2 ) vec(a rn )., b = vec(b 1 ) vec(b 2 ). vec(b r ) Conic optimization 15-1

12 Exponential cone the epigraph of the perspective of exp x is a non-proper cone K = { } (x, y, z) R 3 ye x/y z, y > the exponential cone is K exp = cl K = K {(x,, z) x, z } 1 z 2 y x Conic optimization 15-11

13 Geometric program minimize subject to c T x log n i k=1 exp(a T ik x + b ik), i = 1,..., r Conic LP formulation minimize subject to c T x at ik x + b ik 1 z ik n i k=1 z ik 1, K exp, k = 1,..., n i, i = 1,..., r i = 1,..., m Conic optimization 15-12

14 Power cone Definition: for α = (α 1, α 2,..., α m ) > and m i=1 α i = 1 K α = { (x, y) R m + R y x α 1 1 } xα m m Examples for m = 2 α = ( 1 2, 1 2 ) α = ( 2 3, 1 3 ) α = ( 3 4, 1 4 ) y x 1 x x 1 x x 1 x 2 Conic optimization 15-13

15 Cones constructed from convex sets Inverse image of convex set under perspective K = {(x, y) R n R y >, x/y C} K {(, )} is a convex cone if C is a convex set cl K is proper if C has nonempty interior, does not contain straight lines Consequence any convex constraint x C can be represented as a conic inequality x C (x, 1) K (with minor modifications to make K proper) Conic optimization 15-14

16 Cones constructed from functions Epigraph of perspective of convex function K = {(x, y, z) R n R R y >, yf(x/y) z} K {(,, )} is a convex cone if f is convex cl K is proper if int dom f, epi f does not contain straight lines Consequence can represent any convex constraint f(x) t as a conic linear inequality f(x) t (x, 1, t) K (with minor modifications to make K proper) Conic optimization 15-15

17 Outline conic linear program examples modeling duality

18 Modeling software Modeling packages for convex optimization CVX, YALMIP (MATLAB) CVXPY, PICOS (Python) MOSEK Fusion (different platforms) assist in formulating convex problems by automating two tasks: verifying convexity from convex calculus rules transforming problem in input format required by standard solvers Related packages general-purpose optimization modeling: AMPL, GAMS Conic optimization 15-16

19 Modeling and conic optimization Convex modeling systems convert problems stated in standard mathematical notation to conic LPs in principle, any convex problem can be represented as a conic LP in practice, a small set of primitive cones is used (R n +, Qp, S p ) choice of cones is limited by available algorithms and solvers (see later) modeling systems implement set of rules for expressing constraints f(x) t as conic inequalities for the implemented cones Conic optimization 15-17

20 Examples of second-order cone representable functions convex quadratic f(x) = x T P x + q T x + r (P ) quadratic-over-linear function f(x, y) = xt x y with dom f = R n R + (assume / = ) convex powers with rational exponent f(x) = x α, f(x) = { x β x > + x for rational α 1 and β p-norm f(x) = x p for rational p 1 Conic optimization 15-18

21 Examples of SD cone representable functions matrix-fractional function f(x, y) = y T X 1 y with dom f = {(X, y) S n + R n y R(X)} maximum eigenvalue of symmetric matrix maximum singular value f(x) = X 2 = σ 1 (X) [ ti X X 2 t X T ti ] nuclear norm f(x) = X = i σ i(x) X t U, V : [ U X X T V ], 1 (tr U + tr V ) t 2 Conic optimization 15-19

22 Functions representable with exponential and power cone Exponential cone exponential and logarithm entropy f(x) = x log x Power cone increasing power of absolute value: f(x) = x p with p 1 decreasing power: f(x) = x q with q and domain R ++ p-norm: f(x) = x p with p 1 Conic optimization 15-2

23 Outline conic linear program examples modeling duality

24 Dual cone K = {y x T y for all x K} Properties (if K is a proper cone) K is a proper cone (K ) = K int K = {y x T y > for all x K, x } Dual inequality: x y means x K y for generic proper cone K Note: dual cone depends on choice of inner product Conic optimization 15-21

25 Examples R p +, Qp, S p are self-dual: K = K dual of norm cone is norm cone for dual norm dual of exponential cone K exp = { (u, v, w) R R R + u log( u/w) + u v } (with log(/w) = if w ) dual of power cone is K α = { (u, v) R m + R v (u 1 /α 1 ) α1 (u m /α m ) α m } Conic optimization 15-22

26 Primal and dual conic LP Primal (optimal value p ) minimize subject to c T x Ax b Dual (optimal value d ) maximize b T z subject to A T z + c = z Weak duality: p d (without exception) Conic optimization 15-23

27 Strong duality Main theorem: p = d if primal or dual problem is strictly feasible Other implications of strict feasibility if primal is strictly feasible, then dual optimum is attained (if d is finite) if dual is strictly feasible then primal optimum is attained (if p is finite) Compare with linear programming duality (K = R m + ) for an LP, only exception to strong duality is p = +, d = strong duality holds if primal or dual is feasible if optimal value is finite then it is attained (in primal and dual) Conic optimization 15-24

28 Example with finite nonzero duality gap Primal problem minimize x 1 subject to [ ] x1 x 1 x 2 x 1 1 optimal value p = Dual problem maximize subject to optimal value d = 1 z [ ] Z11 Z 12, z Z 12 Z 22 2Z 12 + z = 1, Z 22 = Conic optimization 15-25

29 Optimality conditions if strong duality holds, then x and z are optimal if and only if [ s ] = [ A T A ] [ x z ] + [ c b ] (1) s, z, z T s = Primal feasibility: block 2 of (1) and s Dual feasibility: block 1 of (1) and z Zero duality gap: inner product of (x, z) and (, s) gives z T s = c T x + b T z Conic optimization 15-26

30 Strong theorems of alternative Strict primal feasibility exactly one of the following two systems is solvable 1. Ax b 2. A T z =, z, z, b T z Strict dual feasibility if c R(A T ), exactly one of the following two systems is solvable 1. Ax K, Ax, c T x 2. A T z + c =, z K solution of one system is a certificate of infeasibility of the other system Conic optimization 15-27

31 Weak theorems of alternative Primal feasibility at most one of the following two systems is solvable 1. Ax b 2. A T z =, z, b T z < Dual feasibility at most one of the following two systems is solvable 1. Ax, c T x < 2. A T z + c =, z these are strong alternatives if a constraint qualification holds Conic optimization 15-28

32 Self-dual embeddings Idea embed primal, dual conic LPs into a single (self-dual) conic LP, so that: embedded problem is feasible, with known feasible points from the solution of the embedded problem, primal and dual solutions of original problem can be constructed, or certificates of primal or dual infeasibility Purpose: a feasible algorithm applied to the embedded problem can detect infeasibility in original problem does not require a phase I to find initial feasible points used by some interior-point solvers Conic optimization 15-29

33 Basic self-dual embedding minimize subject to s κ = A b AT c c T b T x z τ s, κ, z, τ a self-dual conic LP has a trivial solution (all variables zero) z T s + τκ = for all feasible points (follows from equality constraint) hence, problem is not strictly feasible Conic optimization 15-3

34 Optimality condition for embedded problem s κ = A b AT c c T b T x z τ s, κ, z, τ z T s + τκ = follows from self-dual property a (mixed) linear complementarity problem Conic optimization 15-31

35 Classification of nonzero solution let s, κ, x, z, τ be a nonzero solution of the self-dual embedding Case 1: τ >, κ = ŝ = s/τ, ˆx = x/τ, ẑ = z/τ are primal and dual solutions of the conic LPs, i.e., satisfy [ ŝ ] = [ A T A ] [ ˆx ẑ ] + [ c b ] ŝ, ẑ, ŝ T ẑ = Conic optimization 15-32

36 Classification of nonzero solution Case 2: τ =, κ > ; this implies c T x + b T z < if b T z <, then ẑ = z/( b T z) is a proof of primal infeasibility: A T ẑ =, b T ẑ = 1, ẑ if c T x <, then ˆx = x/( c T x) is a proof of dual infeasibility: Aˆx, c T ˆx = 1 Case 3: τ = κ = ; no conclusion can be made about the original problem Conic optimization 15-33

37 Extended self-dual embedding minimize subject to θ s κ = A T c q x A b q z c T b T q τ qx T qz T q τ x z τ θ + 1 s, κ, z, τ q x, q z, q τ chosen so that (s, κ, x, z, τ, θ) = (s, 1, x, z, 1, z T s + 1) is feasible, for some given s, x, z a self-dual conic LP Conic optimization 15-34

38 Optimality condition s κ = A T c q x A b q z c T b T q τ qx T qz T q τ x z τ θ + 1 s, κ, z, τ z T s + τκ = follows from self-dual property a (mixed) linear complementarity problem Conic optimization 15-35

39 Properties of extended self-dual embedding problem is strictly feasible by construction if s, κ, x, z, τ, θ satisfy equality constraint, then θ = s T z + κτ (take inner product with (x, z, τ, θ) of each side of the equality) at optimum, θ = and problem reduces to the embedding on p.15-3 classification of p also applies to solutions of extended embedding Conic optimization 15-36

40 Reference A. Ben-Tal and A. Nemirovski, Lectures on Modern Convex Optimization. Analysis, Algorithms, and Engineering Applications, (21). Conic optimization 15-37