4. Convex optimization problems

Transcription

1 Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization geometric programming generalized inequality constraints semidefinite programming vector optimization 4 1

2 Optimization problem in standard form minimize f 0 (x) subject to f i (x) 0, h i (x) = 0, i = 1,..., m i = 1,..., p x R n is the optimization variable f 0 : R n R is the objective or cost function f i : R n R, i = 1,..., m, are the inequality constraint functions h i : R n R are the equality constraint functions optimal value: p = inf{f 0 (x) f i (x) 0, i = 1,..., m, h i (x) = 0, i = 1,..., p} p = if problem is infeasible (no x satisfies the constraints) p = if problem is unbounded below Convex optimization problems 4 2

3 Optimal and locally optimal points x is feasible if x dom f 0 and it satisfies the constraints a feasible x is optimal if f 0 (x) = p ; X opt is the set of optimal points x is locally optimal if there is an R > 0 such that x is optimal for minimize (over z) f 0 (z) subject to f i (z) 0, i = 1,..., m, h i (z) = 0, i = 1,..., p z x 2 R examples (with n = 1, m = p = 0) f 0 (x) = 1/x, dom f 0 = R ++ : p = 0, no optimal point f 0 (x) = log x, dom f 0 = R ++ : p = f 0 (x) = xlog x, dom f 0 = R ++ : p = 1/e, x = 1/e is optimal f 0 (x) = x 3 3x, p =, local optimum at x = 1 Convex optimization problems 4 3

4 Implicit constraints the standard form optimization problem has an implicit constraint m p x D = dom f i dom h i, i=0 i=1 we call D the domain of the problem the constraints f i (x) 0, h i (x) = 0 are the explicit constraints a problem is unconstrained if it has no explicit constraints (m = p = 0) example: k minimize f 0 (x) = i=1 log(b i a T i x) is an unconstrained problem with implicit constraints a i T x < b i Convex optimization problems 4 4

5 Feasibility problem find x subject to f i (x) 0, h i (x) = 0, i = 1,..., m i = 1,..., p can be considered a special case of the general problem with f 0 (x) = 0: minimize 0 subject to f i (x) 0, h i (x) = 0, i = 1,..., m i = 1,..., p p = 0 if constraints are feasible; any feasible x is optimal p = if constraints are infeasible Convex optimization problems 4 5

6 Convex optimization problem standard form convex optimization problem minimize f 0 (x) subject to f i (x) 0, a T i x = b i, i = 1,..., m i = 1,..., p f 0, f 1,..., f m are convex; equality constraints are affine problem is quasiconvex if f 0 is quasiconvex (and f 1,..., f m convex) often written as minimize f 0 (x) subject to f i (x) 0, Ax = b i = 1,..., m important property: feasible set of a convex optimization problem is convex Convex optimization problems 4 6

7 example minimize f 0 (x) = x x 2 2 subject to f 1 (x) = x 1 /(1 + x 2 2) 0 h 1 (x) = (x 1 + x 2 ) 2 = 0 f 0 is convex; feasible set {(x 1, x 2 ) x 1 = x 2 0} is convex not a convex problem (according to our definition): f 1 is not convex, h 1 is not affine equivalent (but not identical) to the convex problem minimize x x 2 2 subject to x 1 0 x 1 + x 2 = 0 Convex optimization problems 4 7

8 Local and global optima any locally optimal point of a convex problem is (globally) optimal proof: suppose x is locally optimal and y is optimal with f 0 (y) < f 0 (x) x locally optimal means there is an R > 0 such that z feasible, z x 2 R = f 0 (z) f 0 (x) consider z = θy + (1 θ)x with θ = R/(2 y x 2 ) y x 2 > R, so 0 < θ < 1/2 z is a convex combination of two feasible points, hence also feasible z x 2 = R/2 and f 0 (z) θf 0 (x) + (1 θ)f 0 (y) < f 0 (x) which contradicts our assumption that x is locally optimal Convex optimization problems 4 8

9 Optimality criterion for differentiable f 0 x is optimal if and only if it is feasible and f 0 (x) T (y x) 0 for all feasible y X x f 0 (x) if nonzero, f 0 (x) defines a supporting hyperplane to feasible set X at x Convex optimization problems 4 9

10 unconstrained problem: x is optimal if and only if x dom f 0, f 0 (x) = 0 equality constrained problem minimize f 0 (x) subject to Ax = b x is optimal if and only if there exists a ν such that x dom f 0, Ax = b, f 0 (x) + A T ν = 0 minimization over nonnegative orthant minimize f 0 (x) subject to x 0 x is optimal if and only if x dom f 0, x 0, { f 0 (x) i 0 x i = 0 f 0 (x) i = 0 x i > 0 Convex optimization problems 4 10

11 Equivalent convex problems two problems are (informally) equivalent if the solution of one is readily obtained from the solution of the other, and vice-versa some common transformations that preserve convexity: eliminating equality constraints minimize f 0 (x) subject to f i (x) 0, Ax = b i = 1,..., m is equivalent to minimize (over z) f 0 (Fz + x 0 ) subject to f i (Fz + x 0 ) 0, i = 1,..., m where F and x 0 are such that Ax = b x = Fz + x 0 for some z Convex optimization problems 4 11

12 introducing equality constraints minimize f 0 (A 0 x + b 0 ) subject to f i (A i x + b i ) 0, i = 1,..., m is equivalent to minimize (over x, y i ) f 0 (y 0 ) subject to f i (y i ) 0, i = 1,..., m y i = A i x + b i, i = 0,1,..., m introducing slack variables for linear inequalities minimize f 0 (x) subject to a T i x b i, i = 1,..., m is equivalent to minimize (over x, s) f 0 (x) subject to a T i x + s i = b i, i = 1,..., m s i 0, i = 1,... m Convex optimization problems 4 12

13 epigraph form: standard form convex problem is equivalent to minimize (over x, t) t subject to f 0 (x) t 0 f i (x) 0, i = 1,..., m Ax = b minimizing over some variables minimize f 0 (x 1, x 2 ) subject to f i (x 1 ) 0, i = 1,..., m is equivalent to minimize f 0(x 1 ) subject to f i (x 1 ) 0, i = 1,..., m where f 0(x 1 ) = inf x2 f 0 (x 1, x 2 ) Convex optimization problems 4 13

14 Quasiconvex optimization with f 0 : R n minimize f 0 (x) subject to f i (x) 0, i = 1,..., m Ax = b R quasiconvex, f 1,..., f m convex can have locally optimal points that are not (globally) optimal (x, f 0 (x)) Convex optimization problems 4 14

15 convex representation of sublevel sets of f 0 if f 0 is quasiconvex, there exists a family of functions φ t such that: φ t (x) is convex in x for fixed t t-sublevel set of f 0 is 0-sublevel set of φ t, i.e., f 0 (x) t φ t (x) 0 example f 0 (x) = p(x) q(x) with p convex, q concave, and p(x) 0, q(x) > 0 on dom f 0 can take φ t (x) = p(x) tq(x): for t 0, φ t convex in x p(x)/q(x) t if and only if φ t (x) 0 Convex optimization problems 4 15

16 quasiconvex optimization via convex feasibility problems φ t (x) 0, f i (x) 0, i = 1,..., m, Ax = b (1) for fixed t, a convex feasibility problem in x if feasible, we can conclude that t p ; if infeasible, t p Bisection method for quasiconvex optimization given l p, u p, tolerance ǫ > 0. repeat 1. t := (l + u)/2. 2. Solve the convex feasibility problem (1). 3. if (1) is feasible, u := t; else l := t. until u l ǫ. requires exactly log 2 ((u l)/ǫ) iterations (where u, l are initial values) Convex optimization problems 4 16

17 Linear program (LP) minimize c T x + d subject to Gx h Ax = b convex problem with affine objective and constraint functions feasible set is a polyhedron c P x Convex optimization problems 4 17

18 Examples diet problem: choose quantities x 1,..., x n of n foods one unit of food j costs c j, contains amount a ij of nutrient i healthy diet requires nutrient i in quantity at least b i to find cheapest healthy diet, minimize c T x subject to Ax b, x 0 piecewise-linear minimization equivalent to an LP minimize max i=1,...,m (a i T x + b i ) minimize t subject to a T i x + b i t, i = 1,..., m Convex optimization problems 4 18

19 Chebyshev center of a polyhedron Chebyshev center of P = {x a i T x b i, i = 1,..., m} is center of largest inscribed ball x che eb B = {x c + u u 2 r} a i T x b i for all x B if and only if sup{a i T (x c + u) u 2 r} = a i T x c + r a i 2 b i hence, x c, r can be determined by solving the LP maximize r subject to a T i x c + r a i 2 b i, i = 1,..., m Convex optimization problems 4 19

20 (Generalized) linear-fractional program linear-fractional program minimize f 0 (x) subject to Gx h Ax = b f 0 (x) = c T x + d T e T x + f, dom f 0(x) = {x e x + f > 0} a quasiconvex optimization problem; can be solved by bisection also equivalent to the LP (variables y, z) minimize c T y + dz subject to Gy hz Ay = bz e T y + fz = 1 z 0 Convex optimization problems 4 20

21 generalized linear-fractional program c T i x + d i T f 0 (x) = max, dom f T 0 (x) = {x e i x+f i > 0, i = 1,..., r} i=1,...,r e i x + f i a quasiconvex optimization problem; can be solved by bisection example: Von Neumann model of a growing economy maximize (over x, x + + ) min i=1,...,n x i /x i subject to x + 0, Bx + Ax x, x + R n : activity levels of n sectors, in current and next period (Ax) i, (Bx + ) i : produced, resp. consumed, amounts of good i + x i /x i : growth rate of sector i allocate activity to maximize growth rate of slowest growing sector Convex optimization problems 4 21

22 Quadratic program (QP) minimize (1/2)x T Px + q T x + r subject to Gx h Ax = b P S n, so objective is convex quadratic + minimize a convex quadratic function over a polyhedron f 0 (x ) x P Convex optimization problems 4 22

23 Examples least-squares minimize Ax b 2 2 analytical solution x = A b (A is pseudo-inverse) can add linear constraints, e.g., l x u linear program with random cost minimize c Tx + γx T Σx = E c T x + γ var(c T x) subject to Gx h, Ax = b c is random vector with mean c and covariance Σ hence, c T x is random variable with mean c Tx and variance x T Σx γ > 0 is risk aversion parameter; controls the trade-off between expected cost and variance (risk) Convex optimization problems 4 23

24 Quadratically constrained quadratic program (QCQP) minimize (1/2)x T P 0 x + q T 0 x + r 0 subject to (1/2)x T P i x + q T i x + r i 0, Ax = b i = 1,..., m n P i S + ; objective and constraints are convex quadratic if P 1,..., P m S n an affine set ++, feasible region is intersection of m ellipsoids and Convex optimization problems 4 24

25 Second-order cone programming minimize f T x subject to A i x + b i 2 c T i x + d i, Fx = g i = 1,..., m (A i R n i n, F R p n ) inequalities are called second-order cone (SOC) constraints: (A i x + b i, ci T x + d i ) second-order cone in R n i+1 for n i = 0, reduces to an LP; if c i = 0, reduces to a QCQP more general than QCQP and LP Convex optimization problems 4 25

26 Robust linear programming the parameters in optimization problems are often uncertain, e.g., in an LP minimize c T x subject to a T i x b i, i = 1,..., m, there can be uncertainty in c, a i, b i two common approaches to handling uncertainty (in a i, for simplicity) deterministic model: constraints must hold for all a i E i minimize c T x subject to a T i x b i for all a i E i, i = 1,..., m, stochastic model: a i is random variable; constraints must hold with probability η minimize c T x subject to prob(a T i x b i ) η, i = 1,..., m Convex optimization problems 4 26

27 deterministic approach via SOCP choose an ellipsoid as E i : E i = {ā i + P i u u 2 1} (ā i R n, P i R n n ) center is ā i, semi-axes determined by singular values/vectors of P i robust LP minimize c T x subject to a T i x b i a i E i, i = 1,..., m is equivalent to the SOCP minimize c T x subject to ā T i x + P T i x 2 b i, i = 1,..., m (follows from sup u 2 1(ā i + P i u) T x = ā T i x + P T i x 2 ) Convex optimization problems 4 27

28 stochastic approach via SOCP assume a i is Gaussian with mean ā i, covariance Σ i (a i N(ā i,σ i )) a T i x is Gaussian r.v. with mean ā T i x, variance x T Σ i x; hence ( ) T b i āi T x prob(a i x b i ) = Φ 1/2 Σ i x 2 x t 2 where Φ(x) = (1/ 2π) e /2 dt is CDF of N(0,1) robust LP minimize c T x subject to prob(a T i x b i ) η, i = 1,..., m, with η 1/2, is equivalent to the SOCP minimize c T x subject to ā T i x + Φ 1 (η) Σ 1 i/2 x 2 b i, i = 1,..., m Convex optimization problems 4 28

29 Geometric programming monomial function f(x) = cxa 1 xa 2 x K f(x) = c k x 1 a n 1 2 n, ++ with c > 0; exponent α i can be any real number posynomial function: sum of monomials k=1 geometric program (GP) a 1k x 2 dom f = R n a 2k x a nk, dom f = R n n ++ minimize f 0 (x) subject to f i (x) 1, h i (x) = 1, i = 1,..., m i = 1,..., p with f i posynomial, h i monomial Convex optimization problems 4 29

30 Geometric program in convex form change variables to y i = log x i, and take logarithm of cost, constraints a monomial f(x) = cx1 1 a x n n transforms to log f(e y 1,..., e y n T ) = a y + b (b = log c) K a1k a 2k a nk posynomial f(x) = k=1 c kx 1 x 2 x n transforms to ( ) K a log f(e y 1,..., e y n ) = log e k T y+b k (b k = log c k ) k=1 geometric program transforms to convex problem ( ) K minimize log k=1 ( exp(at 0k y + b 0k) ) K T subject to log k=1 exp(a ik y + b ik) 0, i = 1,..., m Gy + d = 0 Convex optimization problems 4 30

31 Design of cantilever beam segment 4 segment 3 segment 2 segment 1 F N segments with unit lengths, rectangular cross-sections of size w i h i given vertical force F applied at the right end design problem minimize total weight subject to upper & lower bounds on w i, h i upper bound & lower bounds on aspect ratios h i /w i upper bound on stress in each segment upper bound on vertical deflection at the end of the beam variables: w i, h i for i = 1,..., N Convex optimization problems 4 31

32 objective and constraint functions total weight w 1 h w N h N is posynomial aspect ratio h i /w i and inverse aspect ratio w i /h i are monomials maximum stress in segment i is given by 6iF/(w i h 2 i ), a monomial the vertical deflection y i and slope v i of central axis at the right end of segment i are defined recursively as v i F = 12(i 1/2) Ew i h 3 + v i+1 i y i F = 6(i 1/3) Ew i h 3 + v i+1 + y i+1 i for i = N,N 1,..., 1, with v N+1 = y N+1 = 0 (E is Young s modulus) v i and y i are posynomial functions of w, h Convex optimization problems 4 32

33 formulation as a GP note minimize w 1 h w N h N subject to w 1 maxw i 1, h 1 maxh i 1, Smaxw 1 1 6iFσ 1 1 w min w i 1, i = 1,..., N h min h 1 i 1, i = 1,..., N i h i 1, S min w i h 1 i 1, i = 1,..., N 1 maxw i h 2 1 y y 1 1 max i 1, i = 1,..., N we write w min w i w max and h min h i h max w min /w i 1, w i /w max 1, h min /h i 1, h i /h max 1 we write S min h i /w i S max as S min w i /h i 1, h i /(w i S max ) 1 Convex optimization problems 4 33

34 Minimizing spectral radius of nonnegative matrix Perron-Frobenius eigenvalue λ pf (A) exists for (elementwise) positive A R n n a real, positive eigenvalue of A, equal to spectral radius max i λ i (A) determines asymptotic growth (decay) rate of A k : A k k λ pf as k alternative characterization: λ pf (A) = inf{λ Av λv for some v 0} minimizing spectral radius of matrix of posynomials minimize λ pf (A(x)), where the elements A(x) ij are posynomials of x equivalent geometric program: minimize subject to λ n j=1 A(x) ij v j /(λv i ) 1, i = 1,..., n variables λ, v, x Convex optimization problems 4 34

35 Generalized inequality constraints convex problem with generalized inequality constraints minimize f 0 (x) subject to f i (x) Ki 0, Ax = b i = 1,..., m f 0 : R n R convex; f i : R n R k i K i -convex w.r.t. proper cone K i same properties as standard convex problem (convex feasible set, local optimum is global, etc.) conic form problem: special case with affine objective and constraints minimize c T x subject to Fx + g K 0 Ax = b extends linear programming (K = R m ) to nonpolyhedral cones + Convex optimization problems 4 35

36 Semidefinite program (SDP) with F i, G S k minimize c T x subject to x 1 F 1 + x 2 F x n F n + G 0 Ax = b inequality constraint is called linear matrix inequality (LMI) includes problems with multiple LMI constraints: for example, x 1 Fˆ1 + + x n Fˆn + Ĝ 0, x 1 F x n F n + G 0 is equivalent to single LMI [ ] [ ] [ ] [ ] Fˆ1 0 Fˆ2 0 Fˆn 0 Ĝ 0 x 1 +x 2 + +x n F 1 0 F 2 0 F n 0 G Convex optimization problems 4 36

37 LP and SOCP as SDP LP and equivalent SDP LP: minimize c T x SDP: minimize c T x subject to Ax b subject to diag(ax b) 0 (note different interpretation of generalized inequality ) SOCP and equivalent SDP SOCP: minimize f T x subject to A i x + b i 2 c i T x + d i, i = 1,..., m SDP: minimize [ f T x ] (c T i x + d i )I A i x + b subject to i (A i x + b i ) T c T 0, i = 1,..., m i x + d i Convex optimization problems 4 37

38 Eigenvalue minimization minimize λ max (A(x)) where A(x) = A 0 + x 1 A x n A n (with given A i S k ) equivalent SDP minimize t subject to A(x) ti variables x R n, t R follows from λ max (A) t A ti Convex optimization problems 4 38

39 Matrix norm minimization minimize A(x) 2 = ( λmax (A(x) T A(x)) ) 1/2 where A(x) = A 0 + x 1 A x n A n (with given A i R p q ) equivalent SDP minimize subject to [ t ] ti (x) A( x) AtI T 0 variables x R n, t R constraint follows from A 2 t A T A t 2 I, t 0 [ ] ti A A T 0 ti Convex optimization problems 4 39

40 Vector optimization general vector optimization problem minimize (w.r.t. K) f 0 (x) subject to f i (x) 0, i = 1,..., m h i (x) 0, i = 1,..., p vector objective f 0 : R n R q, minimized w.r.t. proper cone K R q convex vector optimization problem minimize (w.r.t. K) f 0 (x) subject to f i (x) 0, i = 1,..., m Ax = b with f 0 K-convex, f 1,..., f m convex Convex optimization problems 4 40

41 Optimal and Pareto optimal points set of achievable objective values O = {f 0 (x) x feasible} feasible x is optimal if f 0 (x) is a minimum value of O feasible x is Pareto optimal if f 0 (x) is a minimal value of O O f 0 (x po ) O f 0 (x ) x is optimal x po is Pareto optimal Convex optimization problems 4 41

42 Multicriterion optimization vector optimization problem with K = R q + f 0 (x) = (F 1 (x),..., F q (x)) q different objectives F i ; roughly speaking we want all F i s to be small feasible x is optimal if y feasible = f 0 (x ) f 0 (y) if there exists an optimal point, the objectives are noncompeting feasible x po is Pareto optimal if y feasible, f 0 (y) f 0 (x po ) = f 0 (x po ) = f 0 (y) if there are multiple Pareto optimal values, there is a trade-off between the objectives Convex optimization problems 4 42

43 Regularized least-squares minimize (w.r.t. R 2 + ) ( Ax b 2 2, x 2 2 ) 25 F2 (x) = x O F 1 (x) = Ax b 2 2 example for A R ; heavy line is formed by Pareto optimal points Convex optimization problems 4 43

44 Risk return trade-off in portfolio optimization minimize (w.r.t. R 2 +) ( p T x,x T Σx) subject to 1 T x = 1, x 0 x R n is investment portfolio; x i is fraction invested in asset i p R n is vector of relative asset price changes; modeled as a random variable with mean p, covariance Σ p Tx = E r is expected return; x T Σx = var r is return variance example 15% 1 mean return 10% 5% allocation x 0.5 x(4) x(3) x(2) x(1) 0 0% 0% 10% 20% standard deviation of return 0% 10% 20% standard deviation of return Convex optimization problems 4 44

45 Scalarization to find Pareto optimal points: choose λ K 0 and solve scalar problem minimize λ T f 0 (x) subject to f i (x) 0, h i (x) = 0, i = 1,..., m i = 1,..., p if x is optimal for scalar problem, then it is Pareto-optimal for vector optimization problem f 0 (x 1 ) O f λ 0 (x 3 ) 1 f 0 (x 2 ) λ 2 for convex vector optimization problems, can find (almost) all Pareto optimal points by varying λ K 0 Convex optimization problems 4 45

46 Scalarization for multicriterion problems to find Pareto optimal points, minimize positive weighted sum λ T f 0 (x) = λ 1 F 1 (x) + + λ q F q (x) examples regularized least-squares problem of page take λ = (1, γ) with γ > minimize Ax b + γ x x 10 for fixed γ, a LS problem 5 γ = Ax b 2 2 Convex optimization problems 4 46

47 risk-return trade-off of page 4 44 minimize p Tx + γx T Σx subject to 1 T x = 1, x 0 for fixed γ > 0, a quadratic program Convex optimization problems 4 47

48 MIT OpenCourseWare / Introduction to Convex Optimization Fall 2009 For information about citing these materials or our Terms of Use, visit: