Agenda. 1 Cone programming. 2 Convex cones. 3 Generalized inequalities. 4 Linear programming (LP) 5 Second-order cone programming (SOCP)

Transcription

1 Agenda 1 Cone programming 2 Convex cones 3 Generalized inequalities 4 Linear programming (LP) 5 Second-order cone programming (SOCP) 6 Semidefinite programming (SDP) 7 Examples

2 Optimization problem in standard form minimize f 0 (x) subject to f i (x) 0 i = 1,..., m h i (x) = 0 i = 1,..., p x R n f 0 : R n R f i : R n R h i : R n R (objective or cost function) (inequality constraint functionals) (equality constraint functionals)

3 Terminology x is feasible if x obeys the constraints feasible set C: set of all feasible points optimal value: p = inf{f 0 (x), x C} can be ; e.g. min log(x), x > 0. by convention, p = if C = (problem infeasible) optimal solution: x s.t. f(x ) = p there may be no optimal solution: e.g. min log(x), x > 0 optimal set: {x : f(x) = p }

4 Convex optimization problem Convex optimization problem in standard form minimize f 0 (x) subject to f i (x) 0 i = 1,..., m a T i x = b i i = 1,..., p f 0, f 1,..., f m convex affine equality constraints Ax = b, feasible set is convex A R p n Abstract convex optimization problem minimize subject to f 0 (x) x C f 0 convex C convex

5 Why convexity? A convex function has no local minimum that is not global convex not convex A convex set is connected and has feasible directions at any point convex + feasible directions not convex A convex function is continuous and has some differentiability properties Convex functions arise prominently in duality

6 Cone programming I LP minimize c T x subject to F x + g 0 Ax = b Nonlinear programming nonlinear constraints Express nonlinearity via generalized inequalities

7 Orderings of R n and convex cones K is a convex cone if (i) K is convex (ii) K is a cone (i.e. x K = λx K λ 0) K is pointed if (iii) x K and x K = x = 0 (K does not contain a straight line through the origin) Example: K = {x R n : x 0} is a pointed convex cone Two additional properties of R n + (iv) R n + is closed (v) R n + has a nonempty interior

8 Implication: ordering a K b a b K (i) - (iii) ensure that this is a good ordering 1 reflexivity: a a follows from 0 K 2 antisymmetry: a b, b a = a = b (since K is pointed) 3 transitivity: a b, b c = a c (since K is convex and a cone) compatibility with linear operations a b & λ 0 = λa λb a b & c d = a + c b + d Good properties of LPs come from these properties 4 closedness: a i b i, a i a, b i b = a b 5 nonempty interior allows us to define strict inequalities: a b a b int(k)

9 Examples of cones Nonnegative orthant R n + {x R n : x 0} Second-order (or Lorentz or ice cream) cone {x R n+1 : x x2 n x n+1 } Positive semidefinite cone {X S n : X 0}

10 Cone programming II K = R n + = linear programming minimize c T x subject to F x + g 0 Ax = b Minimize linear functional over an affine slice of a cone Very fruitful point of view useful theory (duality) useful algorithms (interior point methods)

11 Linear programming (LP) Linear objective minimize c T x subject to F x + g 0 Ax = b Linear equality and inequality constraints Feasible set is a polyhedron c x* (optimal) Many problems can be formulated as LP s c T x = constant

12 Example: Chebyshev approximation A R m n b R m minimize Ax b minimize max i=1,...,m a T i x b i Different from LS problem: minimize Ax b 2 LP formulation (epigraph trick) optimization variables (x, t) R n+1 minimize t subject to a T i x b i t i minimize t subject to t a T i x b i t i

13 Example: basis pursuit A R m n minimize x 1 subject to Ax = b LP formulations: (a) (b) minimize subject to optimization variables (x, t) R 2n minimize subject to optimization variables (x +, x ) R 2n ti t i x i t i Ax = b x + i + x i A(x + x ) = b x +, x 0

14 Second-order cone programming (SOCP) minimize c T x subject to F i x + g i 2 c T i x + d i i = 1,..., m Ax = b F i x + g i 2 c T i x + d i (hence the name) [ ] Fi x + g i c T i x + d L i = {(y i, t) : y i t} i SOCP minimize c [ T x ] Fi subject to x + c T i Ax = b [ gi d i ] L i

15 affine mapping cone product F x + g = { [ ] [ ] F i gi } c T x + i d i i=1,...,m K = L 1 L 2... L m [ Fi c T i ] [ ] gi x + L d i i i F x + g K this is a cone program SOCP minimize subject to c T x F x + g K Ax = b

16 Example: support vector machines n pairs (x i, y i ) x i R p : feature/explanatory variables y i { 1, 1}: response/class label Examples x i: infrared blood absorption spectrum y i: person is diabetic or not SVM model: SVM as a penalized fitting procedure min β n [1 y i f(x i )] + + λ β 2 i=1 hinge loss f(x) = x T β sometimes f(x) = x T β + β 0 and same minimum [1-yf(x)] yf(x)

17 SVM: formulation as an SOCP Variables (β, t) R p n minimize ti + λ β 2 minimize ti + λ β 2 subject to [1 y i f(x i )] + t i subject to y i f(x i ) 1 t i t i 0 this an SOCP, since SOCP s are more general than QP s and QCQP s Equivalence minimize subject to ti + λu β 2 u y i f(x i ) 1 t i t i 0 ( u + 1 ) 2 ( u 1 ) [ ] 2 β 2 u β 2 β u 1 u

18 QP SOCP ( = LP SOCP) QCQP 1 minimize 2 xt P 0 x + q0 T x + r 0 1 subject to 2 xt P i x + qi T x + r i 0 P 0, P i 0 QCQP SOCP quadratic convex inequalities are SOCP-representable

19 Example: total-variation denoising Observe b ij = f ij + σz ij 0 < i, j < n f is original image b is a noisy version Problem: recover original image (de-noise) Min-TV solution TV norm Formulation as an SOCP minimize subject to x T V x b δ x TV = D ij x 2 D ij x = [ ] xi+1,j x i,j x i,j+1 x i,j minimize subject to tij D ij x 2 t ij x b 2 δ

20 Semidefinite programming (SDP) minimize c T x subject to F (x) = x 1 F x n F n F 0 0 F i S p (p p symmetric matrices) linear matrix inequality (LMI): F (x) 0 multiple LMI s can be combined into one: F 1 (x) F i (x) 0 i = 1,..., m... 0 Fm(x)

21 SOCP SDP (but the converse is not true!) (x, t) R n+1 : x t [ ] tim x x T 0 t SOCP constraints are LMI s Hierarchy: LP SOCP SDP Many nonlinear convex problems can be cast as SDP s

22 Example: minimum-norm problem minimize A(x) subject to A(x) = x 1 A x n A n B with A i R p1 p2, is equivalent to minimize t subject to A(x) t A(x) t [ ] tip1 A(x) A T 0 (x) Why? Eigenvalues of this matrix are {t ± σ i (A(x))} ti p2

23 Example: nuclear-norm minimization minimize X = σ i (X) subject to X ij = B ij (i, j) Ω [p 1 ] [p 2 ] This is an SDP (proof, later)

24 Stability analysis for dynamical systems Linear system dv dt = v(t) = Qv(t) Q Rn n Main question: is this system stable? i.e. do all trajectories tend to zero as t? Simple sufficient condition: existence of a quadratic Lyapunov function (i) L(v) = v T Xv X 0 (ii) L = d dtl(v(t)) αl(v(t)) (α > 0) for any trajectory This condition gives L(v(t)) = v T (t)xv(t) exp( αt) L(v(0)) (Gronwall s inequality), whence X 0 = v(t) 0 as t Exsitence of X 0 and α > 0 provides a certificate of stability

25 dv dt = v(t) = Qv(t), L(v) = vt Xv X 0 L = d [ v T (t)xv(t) ] = v T Xv + v T X v = v T (Q T X + XQ)v dt i.e. L αl v T (Q T X + XQ + αx)v < 0 v Q T X + XQ + αx 0 Conclusion: to certify stability, it suffices to find X obeying If the optimal value of SDP X 0, Q T X + XQ 0 minimize subject to t [ ] X + ti 0 0 (Q T 0 X + XQ) + ti is negative, then the system is stable

26 Extension v(t) = Q(t)v(t) Q(t) conv{q 1,..., Q n } time-varying L(v) = v T Xv (X 0) s.t. L αl = stability Similar calculations show that for all v v T (Q T (t)x + XQ(t) + αx)v 0 Q T (t)x + XQ(t) + αx 0, Q(t) conv{q 1,..., Q n } Q T i X + XQ i + αx 0, i = 1,..., k If we can find X such that then we have stability This is an SDP! X 0 & Q T i X + XQ i 0 i = 1,..., k

27 References 1 A. Ben-Tal and A. Nemirovski, Lectures on Modern Convex Optimization: Analysis, Algorithms, and Engineering Applications, MPS-SIAM Series on Optimization 2 S. Boyd and L. Vandenberghe, Convex Optimization, Cambridge University Press