Lecture: Convex Optimization Problems

1/36 Lecture: Convex Optimization Problems http://bicmr.pku.edu.cn/~wenzw/opt-2015-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes

Introduction 2/36 optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization geometric programming generalized inequality constraints semidefinite programming vector optimization

Optimization problem in standard form min f 0 (x) s.t. f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p x R n is the optimization variable f 0 : R n R is the objective or cost function f i : R n R, i = 1,..., m, are the inequality constraint functions h i : R n R are the equality constraint functions optimal value: p = inf{f 0 (x) f i (x) 0, i = 1,..., m, h i (x) = 0, i = 1,..., p} p = if problem is infeasible (no x satisfies the constraints) p = if problem is unbounded below 3/36

Optimal and locally optimal points x is feasible if x dom f 0 and it satisfies the constraints a feasible x is optimal if f 0 (x) = p ; X opt is the set of optimal points x is locally optimal if there is an R > 0 such that x is optimal for min(overz) f 0 (z) s.t. f i (z) 0, i = 1,..., m h i (z) = 0, i = 1,..., p z x 2 R examples (with n = 1, m = p = 0) f 0 (x) = 1/x, dom f 0 = R ++ : p = 0, no optimal point f 0 (x) = log x, dom f 0 = R ++ : p = f 0 (x) = x log x, dom f 0 = R ++ : p = 1/e, x = 1/e is optimal f 0 (x) = x 3 3x, p =, local optimum at x = 1 4/36

Implicit constraints 5/36 the standard form optimization problem has an implicit constraint x D = m dom f i i=0 we call D the domain of the problem p dom h i, the constraints f i (x) 0, h i (x) = 0 are the explicit constraints a problem is unconstrained if it has no explicit constraints (m = p = 0) i=1 example: min f 0 (x) = k i=1 log(b i a T i x) is an unconstrained problem with implicit constraints a T i x < b i

6/36 Feasibility problem find x s.t. f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p can be considered a special case of the general problem with f 0 (x) = 0: min 0 s.t. f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p p = 0 if constraints are feasible; any feasible x is optimal p = if constraints are infeasible

7/36 Convex optimization problem standard form convex optimization problem min f 0 (x) s.t. f i (x) 0, i = 1,..., m a T i x = b i, i = 1,..., p f 0, f 1,..., f m are convex; equality constraints are affine problem is quasiconvex if f 0 is quasiconvex (and f 1,..., f m convex) often written as min f 0 (x) s.t. f i (x) 0, i = 1,..., m Ax = b important property: feasible set of a convex optimization problem is convex

example min f 0 (x) = x 2 1 + x 2 2 s.t. f 1 (x) = x 1 /(1 + x 2 2) 0 h 1 (x) = (x 1 + x 2 ) 2 = 0 f 0 is convex; feasible set {(x 1, x 2 ) x 1 = x 2 0} is convex not a convex problem (according to our definition): f 1 is not convex, h 1 is not affine equivalent (but not identical) to the convex problem min x1 2 + x2 2 s.t. x 1 0 x 1 + x 2 = 0 8/36

Local and global optima 9/36 any locally optimal point of a convex problem is (globally) optimal proof : suppose x is locally optimal and y is optimal with f 0 (y) < f 0 (x) x locally optimal means there is an R > 0 such that z feasible, z x 2 R = f 0 (z) f 0 (x) consider z = θy + (1 θ)x with θ = R/(2 y x 2 ) y x 2 > R, so 0 < θ < 1/2 z is a convex combination of two feasible points, hence also feasible z x 2 = R/2 and f 0 (z) θf 0 (x) + (1 θ)f 0 (y) < f 0 (x) which contradicts our assumption that x is locally optimal

if nonzero, f 0 (x) defines a supporting hyperplane to feasible set X at x zero, f 0 (x) defines a supporting hyperplane to feasible set X at 10/36 Optimality criterion for differentiable f ptimal if and only if it is feasible and 0 x is optimal if and only if it is feasible and f 0 (x) T (y x) 0 for all feasible y f 0 (x) T (y x) 0 for all feasible y X x f 0 (x)

11/36 unconstrained problem: x is optimal if and only if x dom f 0, f 0 (x) = 0 equality constrained problem min f 0 (x) s.t. Ax = b x is optimal if and only if there exists a υ such that x dom f 0, Ax = b, f 0 (x) + A T υ = 0 minimization over nonnegative orthant min f 0 (x) s.t. x 0 x is optimal if and only if x dom f 0, x 0, { f0 (x) i 0 x i = 0 f 0 (x) i = 0 x i > 0

12/36 Equivalent convex problems two problems are (informally) equivalent if the solution of one is readily obtained from the solution of the other, and vice-versa some common transformations that preserve convexity: eliminating equality constraints is equivalent to min f 0 (x) s.t. f i (x) 0, i = 1,..., m Ax = b min(overz) f 0 (Fz + x 0 ) s.t. f i (Fz + x 0 ) 0, i = 1,..., m where F and x 0 are such that Ax = b x = Fz + x 0 for some z

13/36 Equivalent convex problems introducing equality constraints min f 0 (A 0 x + b 0 ) s.t. f i (A i x + b i ) 0, i = 1,..., m is equivalent to min(over x, y i ) f 0 (y 0 ) s.t. f i (y i ) 0, i = 1,..., m y i = A i x + b i, i = 0, 1,..., m introducing slack variables for linear inequalities min f 0 (x) s.t. a T x b i, i = 1,..., m is equivalent to min(over x, s) f 0 (x) s.t. a T x + s i = b i, i = 1,..., m s i 0, i = 1,...m

14/36 Equivalent convex problems epigraph form: standard form convex problem is equivalent to min(over x, t) t s.t. f 0 (x) t 0 f i (x) 0, i = 1,..., m Ax = b minimizing over some variables min f 0 (x 1, x 2 ) s.t. f i (x 1 ) 0, i = 1,..., m is equivalent to min f 0 (x 1 ) s.t. f i (x 1 ) 0, i = 1,..., m where f 0 (x 1 ) = inf x2 f 0 (x 1, x 2 )

15/36 Quasiconvex optimization minimize f 0 (x) subjectmin to f 0 i (x) 0, i = 1,..., m s.t. fax i (x) = 0, b i = 1,..., m th f 0 : R n R quasiconvex, Ax f 1 =,. b.., f m convex with f 0 : R n R quasiconvex, f 1,..., f m convex n have can have locally locally optimal optimal points points that that are are not not (globally) optimal (x, f 0 (x))

16/36 convex representation of sublevel sets of f 0 if f 0 is quasiconvex, there exists a family of functions φ t such that: φ t (x) is convex in x for fixed t t-sublevel set of f 0 is 0-sublevel set of φ t,i.e., f 0 (x) t φ t (x) 0 example f 0 (x) = p(x) q(x) with p convex, q concave, and p(x) 0, q(x) > 0 on dom f 0 can take φ t (x) = p(x) tq(x): for t 0, φ t convex in x p(x)/q(x) t if and only if φ t (x) 0

quasiconvex optimization via convex feasibility problems φ t (x) 0, f i (x) 0, i = 1,..., m, Ax = b (1) for fixed t, a convex feasibility problem in x if feasible, we can conclude that t p ; if infeasible, t p Bisection method for quasiconvex optimization given l p, u p, tolerance ɛ > 0. repeat 1 t := (l + u)/2. 2 Solve the convex feasibility problem (1). 3 if (1) is feasible, u := t; else l := t. until u l ɛ. requires exactly log 2 ((u l)/ɛ) iterations (where u, l are initial values) 17/36

18/36 Linear program (LP) minimize c T x + d subject minto c T Gx x + d h s.t. Gx Ax = h b Ax = b convex problem with affine objective and constraint functions convex problem with affine objective and constraint functions feasible set is a polyhedron feasible set is a polyhedron P x c

19/36 Examples diet problem: choose quantities x 1,..., x n of n foods one unit of food j costs c j, contains amount a ij of nutrient i healthy diet requires nutrient i in quantity at least b i to find cheapest healthy diet, min c T x s.t. Ax b, x 0 piecewise-linear minimization min max i=1,...,m (a T i x + b i ) equivalent to an LP min t s.t. a T i x + b i t, i = 1,..., m

20/36 Chebyshev center of a polyhedron Chebyshev center of a polyhedron Chebyshev center center of of P = P {x = {x a a T T i x b i, i 1,..., m} i x b i, i = 1,..., m} is center of largest inscribed ball is center of largest inscribed ball B = {x c + u u 2 r} B = {x c + u u 2 r} x cheb a T i x a T i xb i for b i for all xall x B ifb and if and only only if if sup{a sup{a T i (x T i c (x + c u) + u) u u 2 2 r} r} = = a T a i x T i c x c + + r a r a i i 2 2 b i b i hence, x c, r can be determined by solving the LP hence, x c, r can be determined by solving the LP max r maximize r s.t. subject to a T a T i i x x c + r a c r a i i 2 b 2 i, i = 1,..., m i, i 1,...,

Linear-fractional program linear-fractional program min s.t. f 0 (x) Gx h Ax = b f 0 (x) = ct x + d e T x + f, dom f 0(x) = {x e T x + f > 0} a quasiconvex optimization problem; can be solved by bisection also equivalent to the LP (variables y, z) min c T y + dz s.t. Gy hz Ay = bz e T y + fz = 1 z 0 21/36

Quadratic program (QP) 22/36 Quadratic program (QP) min (1/2)x T Px + q T x + r s.t. Gx h minimize (1/2)x T P x + q T x + r subject to Ax Gx= bh Ax = b P S n +, so objective is convex quadratic P S n +, so objective is convex quadratic minimize a convex quadratic function over a polyhedron minimize a convex quadratic function over a polyhedron f 0 (x ) x P

Examples least-squares min Ax b 2 2 analytical solution x = A b (A is pseudo-inverse) can add linear constraints, e.g., l x u linear program with random cost min c T x + γx T Σx = Ec T x + γvar(c T x) s.t. Gx h, Ax = b c is random vector with mean c and covariance Σ hence, c T x is random variable with mean c T x and variance x T Σx γ > 0 is risk aversion parameter; controls the trade-off between expected cost and variance (risk) 23/36

24/36 Quadratically constrained quadratic program (QCQP) min (1/2)x T P 0 x + q T 0 x + r 0 s.t. (1/2)x T P i x + q T i x + r i 0, i = 1,..., m Ax = b P i S n + ; objective and constraints are convex quadratic if P 1,..., P m S n ++, feasible region is intersection of m ellipsoids and an affine set

25/36 Second-order cone programming min f T x s.t. A i x + b i 2 c T i x + d i, i = 1,..., m Fx = g (A i R ni n, F R p n ) inequalities are called second-order cone (SOC) constraints: (A i x + b i, c T i x + d i ) second-order cone in R n i+1 for n i = 0, reduces to an LP; if c i = 0, reduces to a QCQP more general than QCQP and LP

Robust linear programming the parameters in optimization problems are often uncertain, e.g., in an LP min c T x s.t. a T i x b i, i = 1,..., m, there can be uncertainty in c, a i, b i two common approaches to handling uncertainty (in a i, for simplicity) deterministic model: constraints must hold for all a i E i min c T x s.t. a T i x b i for all a i E i, i = 1,..., m, stochastic model: a i is random variable; constraints must hold with probability η min c T x s.t. prob(a T i x b i ) η, i = 1,..., m 26/36

27/36 deterministic approach via SOCP choose an ellipsoid as E i : E i = {ā i + P i u u 2 1} (ā i R n, P i R n n ) center is ā i, semi-axes determined by singular values/vectors of P i robust LP min c T x s.t. a T i x b i a i E i, i = 1,..., m is equivalent to the SOCP min c T x s.t. ā T i x + P T i x 2 b i, i = 1,..., m (follows from sup u 2 1 (ā i + P i u) T x = ā T i x + PT i x 2)

28/36 stochastic approach via SOCP assume a i is Gaussian with mean ā i, covariance Σ i (a i N (ā i, Σ i )) a T x is Gaussian r.v. with mean ā T i x, variance xt Σ i x; hence ( ) prob(a T b i ā T i i x b i ) = Φ x Σ 1/2 i x 2 where Φ(x) = (1/ 2π) x e t2 /2 dt is CDF of N (0, 1) robust LP min c T x s.t. prob(a T i x b i ) η, i = 1,..., m, with η 1/2, is equivalent to the SOCP min c T x s.t. ā T i x + Φ 1 (η) Σ 1/2 i x 2 b i, i = 1,..., m

Geometric programming monomial function f (x) = cx a 1 1 xa 2 2 xan n, dom f = R n ++ with c > 0; exponent α i can be any real number posynomial function: sum of monomials f (x) = K k=1 geometric program (GP) min c k x a 1k 1 xa 2k 2 x a nk n, dom f = R n ++ f 0 (x) s.t. f i (x) 1, i = 1,..., m h i (x) = 1, i = 1,..., p with f i posynomial, h i monomial 29/36

Geometric program in convex form 30/36 change variables to y i = log x i, and take logarithm of cost, constraints monomial f (x) = cx a 1 1 xan n transforms to log f (e y 1,..., e yn ) = a T y + b (b = log c) posynomial f (x) = K k=1 c kx a 1k 1 xa 2k 2 x a nk n transforms to ( K ) log f (e y 1,..., e yn ) = log e at k y+b k (b k = log c k ) k=1 geometric program transforms to convex problem ( K ) min log k=1 exp(at 0ky + b 0k ) ( K ) s.t. log k=1 exp(at iky + b ik ) 0, i = 1,..., m Gy + d = 0

Minimizing spectral radius of nonnegative matrix Perron-Frobenius eigenvalue λ pf (A) exists for (elementwise) positive A R n n a real, positive eigenvalue of A, equal to spectral radius max i λ i (A) determines asymptotic growth (decay) rate of A k : A k λ k pf as k alternative characterization: λ pf (A) = inf{λ Av λv for some v 0} minimizing spectral radius of matrix of posynomials minimize λ pf (A(x)), where the elements A(x) ij are posynomials of x equivalent geometric program: min s.t. λ n j=1 A(x) ijv j /(λv i ) 1, i = 1,..., n variables λ, v, x 31/36

Generalized inequality constraints convex problem with generalized inequality constraints min f 0 (x) s.t. f i (x) Ki 0, i = 1,..., m Ax = b f 0 : R n R convex; f i : R n R k i K i -convex w.r.t. proper cone K i same properties as standard convex problem (convex feasible set, local optimum is global, etc.) conic form problem: special case with affine objective and constraints min c T x s.t. Fx + g K 0 Ax = b extends linear programming (K = R m +) to nonpolyhedral cones 32/36

Semidefinite program (SDP) min c T x s.t. x 1 F 1 + x 2 F 2 + + x n F n + G 0 Ax = b with F i, G S k inequality constraint is called linear matrix inequality (LMI) includes problems with multiple LMI constraints: for example, x 1ˆF 1 + + x nˆf n + Ĝ 0, x 1 F 1 + + x n F n + G 0 is equivalent to single LMI x 1 [ F 1 0 0 F 1 ] +x 2 [ F 2 0 0 F 2 ] + +x n [ F n 0 0 F n ] [ G 0 + 0 G ] 0 33/36

LP and SOCP as SDP 34/36 LP and equivalent SDP LP: min c T x s.t. Ax b SDP: min c T x s.t. diag(ax b) 0 (note different interpretation of generalized inequality ) SOCP and equivalent SDP SOCP: min f T x s.t. A i x + b i 2 c T x + d i, i = 1,..., m SDP: min f T x [ (c T s.t. i x + d i )I A i x + b i (A i x + b i ) T c T i x + d i ] 0, i = 1,..., m

35/36 Eigenvalue minimization min λ max (A(x)) where A(x) = A 0 + x 1 A 1 + + x n A n (with given A i S k ) equivalent SDP min s.t. t A(x) ti variables x R n, t R follows from λ max (A) t A ti

Matrix norm minimization 36/36 min A(x) 2 = ( λ max (A(x) T A(x)) ) 1/2 where A(x) = A 0 + x 1 A 1 + + x n A n (with given A i R p q ) equivalent SDP min s.t. t [ ti A(x) T A(x) ti ] 0 variables x R n, t R constraint follows from A 2 t A T A t 2 I, t 0 [ ] ti A A T 0 ti