Conve Optimization 4. Conve Optimization Problems Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2017 Autumn Semester SJTU Ying Cui 1 / 58
Outline Optimization problems Conve optimization Linear optimization problems Quadratic optimization problems Geometric programg Semidefinite prograg Vector optimization SJTU Ying Cui 2 / 58
Optimization problem in standard form f 0 () s.t. f i () 0, i = 1,,m optimization variable: R n objective function: f 0 : R n R h i () = 0, i = 1,,p inequality constraint functions: f i : R n R, i = 1,,m equality constraint functions: h i : R n R, i = 1,,p domain: D = m i=0 domf i p i=0 domh i feasible point : D and satisfies all constraints feasible set or constraint set X: set of all feasible points feasible problem: problem with nonempty feasible set SJTU Ying Cui 3 / 58
Implicit constraints the standard form optimization problem has an implicit constraint m p D = domf i domh i i=0 i=1 eplicit constraints: f i () 0, i = 1,,m, h i () = 0,i = 1,,p a problem is unconstrained if it has no eplicit constraints (m = p = 0) eample: f 0 () = k log(b i ai T ) is an unconstrained problem with implicit constraints a T i < b i i=1 SJTU Ying Cui 4 / 58
Optimal and locally optimal points optimal value: p = inf{f 0 () f i () 0, i = 1,,m, h i () = 0,i = 1,,p} p = if problem is infeasible p = if problem is unbounded below (globally) optimal point : is feasible and f 0 () = p optimal set X opt : set of optimal points if Xopt is nonempty, the optimal value is achieved; otherwise not achieved (always occurs when problem unbounded below) locally optimal point : R > 0 such that is optimal for f 0 (z) z s.t. f i (z) 0, i = 1,,m h i (z) = 0, i = 1,,p z 2 R SJTU Ying Cui 5 / 58
Optimal and locally optimal points eamples (with n = 1, m = p = 0) f 0 () = 1/, domf 0 = R ++ : p = 0, no optimal point f 0 () = log, domf 0 = R ++ : p = f 0 () = log, domf 0 = R ++ : p = 1/e, unique optimal point = 1/e f 0 () = 3 3, p =, locally optimal point = 1 SJTU Ying Cui 6 / 58
Feasibility problems The feasibility problem is to detere whether the constraints are consistent, and if so, find a point that satisfies them, i.e., find s.t. f i () 0, i = 1,...,m h i () = 0, i = 1,...,p It can be considered a special case of the general problem with f 0 () = 0, i.e., 0 s.t. f i () 0, i = 1,...,m h i () = 0, i = 1,...,p p = 0 if constraints are feasible; any feasible is optimal p = if constraints are infeasible SJTU Ying Cui 7 / 58
Conve optimization problems in standard form s.t. f 0 () f i () 0, i = 1,...,m a T i = b i, i = 1,...,p (or A = b) objective function f 0 and inequality constraint functions f 1,...,f m are conve; equality constraints are affine problem is quasiconve if f0 is quasiconve and f 1,...,f m conve feasible set of a conve optimization problem is conve intersection of domain D = m i=0 domf i with m sub level sets { f i () 0} and p hyperplanes { a T i = b i } (all conve) SJTU Ying Cui 8 / 58
Abstract form conve optimization problem eample f 0 () = 1 2 +2 2 s.t. f 1 () = 1 /(1+2) 2 0 h 1 () = ( 1 + 2 ) 2 = 0 imize a conve function over a conve set f0 is conve, feasible set {( 1, 2 ) 1 = 2 0} is conve not a conve optimization problem in standard form (according to our definition) f1 is not conve, h 1 is not affine equivalent (but not identical) to the conve problem 1 2 +2 2 s.t. 1 0 1 + 2 = 0 SJTU Ying Cui 9 / 58
Local and global optima any locally optimal point of a conve problem is (globally) optimal proof: Suppose is locally optimal, i.e., is feasible and f 0 () (a) = inf{f 0 (z) z feasible, z 2 R} for some R > 0. Suppose is not globally optimal, i.e., there eists a feasible y such that f 0 (y) < f 0 (). Evidently y 2 > R. Consider z = θy +(1 θ) with θ = Then, we have z 2 = R/2 < R, and by conveity of the feasible set, z is feasible. By conveity of f 0, we have which contradicts (a). f 0 (z) θf 0 (y)+(1 θ)f 0 () < f 0 () R 2 y 2. SJTU Ying Cui 10 / 58
Optimality criterion for differentiable f 0 is optimal iff it is feasible and f 0 () T (y ) 0 for all y X X f0() Figure 4.2 Geometric interpretation of the optimality condition (4.21). The feasible set X is shown shaded. Some level curves of f0 are shown as dashed lines. The point is optimal: f0() defines a supporting hyperplane (shown as a solid line) to X at. geometric interpretation: if f 0 () 0, f 0 () defines a supporting hyperplane to feasible set X at SJTU Ying Cui 11 / 58
Optimality criterion for differentiable f 0 unconstrained problem: f 0 () is optimal iff domf 0, f 0 () = 0 equality constrained problem: f 0 () s.t. A = b is optimal iff there eists a v such that domf 0, A = b, f 0 ()+Av = 0 imization over nonnegative orthant: f 0 () s.t. 0 is optimal iff domf 0, 0, f 0 () 0, ( f 0 ()) i i = 0 condition ( f0 ()) i i = 0 is called complementarity: the sparsity patterns (i.e., the set of indices corresponding to nonzero components) of the vectors and f 0 () are complementary (i.e., have empty intersection) ( { f0 () i 0 i =0 f 0 () i =0 i >0 SJTU Ying Cui 12 / 58 )
Equivalent conve problems two problems are (informally) equivalent if the solution of one is readily obtained from the solution of the other, and vice-versa some common transformations that preserve conveity SJTU Ying Cui 13 / 58
Equivalent conve problems eliating equality constraints s.t. f 0 () f i () 0, i = 1,...,m A = b is equivalent to z f 0 (Fz + 0 ) s.t. f i (Fz + 0 ) 0, i = 1,...,m where F and 0 are such that A = b = Fz + 0 for some z in principle, we can restrict our attention to conve optimization problems without equality constraints in many cases, however, it is better to retain equality constraints, for ease of analysis, or not to ruin efficiency of an algorithm that solves it SJTU Ying Cui 14 / 58
Equivalent conve problems introducing equality constraints is equivalent to f 0 (A 0 +b 0 ) s.t. f i (A i +b i ) 0, i = 1,...,m,y s.t. f 0 (y 0 ) f i (y i ) 0, i = 1,...,m y i = A i +b i, i = 0,...,m SJTU Ying Cui 15 / 58
Equivalent conve problems introducing slack variables for linear inequalities s.t. f 0 () a T i b i, i = 1,...,m is equivalent to,s s.t. f 0 () a T i +s i = b i, i = 1,...,m s i 0, i = 1,...,m s i is called the slack variable associated with a T i b i each inequality constraint is replaced with an equality constraint and a nonnegativity constraint SJTU Ying Cui 16 / 58
Equivalent conve problems epigraph problem form s.t. f 0 () f i () 0, i = 1,...,m A = b is equivalent to,t t s.t. f 0 () t 0 f i () 0, i = 1,...,m A = b an optimization problem in the graph space (, t): imize t over the epigraph of f 0, subject to the constraints on. a linear objective is universal for conve optimization can simplify theoretical analysis and algorithm development SJTU Ying Cui 17 / 58
Equivalent conve problems imizing over some variables is equivalent to f 0 ( 1, 2 ) 1, 2 s.t. f i ( 1 ) 0, i = 1,...,m 1 f i ( 2 ) 0, i = 1,...,m 2 1 f0 ( 1 ) s.t. f i ( 1 ) 0, i = 1,...,m 1 where f 0 ( 1 ) = inf{f 0 ( 1,z) f i (z) 0, i = 1,...,m 2 } SJTU Ying Cui 18 / 58
Quasiconve optimization p = s.t. f 0 () f i () 0, i = 1,...,m A = b where f 0 : R n R is quasiconve, and f 1,...,f m are conve locally optimal solutions and optimality conditions Quasiconve optimization problem can have locally optimal points that are not (globally) optimal (,f()) Figure 4.3 A quasiconve function f on R, with a locally optimal point that is not globally optimal. This eample shows that the simple optimality condition f () = 0, valid for conve functions, does not hold for quasiconve functions. SJTU Ying Cui 19 / 58
Quasiconve optimization conve representation of sublevel sets of f 0 if f 0 is quasiconve, there eists a family of functions φ t such that: φ t () is conve in for fied t t-sublevel set of f 0 is 0-sublevel set of φ t,i.e., f 0 () t φ t () 0 for each, φ t () is nondecreasing in t eample (conve over concave function) f 0 () = p() q() with p conve, q concave, and p() 0,q() > 0 on domf 0 can take φ t () = p() tq() : for t 0,φ t conve in p()/q() t if and only if φ t () 0 for each, φ t () is nondecreasing in t SJTU Ying Cui 20 / 58
Quasiconve optimization quasiconve optimization via conve feasibility problems conve feasibility problem in for any fied t find φ t () 0, f i () 0, i = 1,...,m A = b if feasible, t p ; if infeasible, t p Bisection method for quasiconve optimization ( log 2 ((u l)/ǫ) iterations) given l p,u p, tolerance ǫ > 0. repeat 1. t := (l +u)/2. 2. Solve the conve feasibility problem (1). 3. if feasible, u := t; else l := t. until u l ǫ SJTU Ying Cui 21 / 58
Linear program (LP) c T +d s.t. G h A = b where c R n, d R, G R m n, h R m, A R p n and b R p objective and constraint functions are all affine linear programs are, of course, conve optimization problems omit d, i.e., linear objective function (not affect X opt or X) feasible set is a polyhedron c P Figure 4.4 Geometric interpretation of an LP. The feasible set P, which is a polyhedron, is shaded. The objective c T is linear, so its level curves are hyperplanes orthogonal to c (shown as dashed lines). The point is optimal; it is the point in P as far as possible in the direction c. SJTU Ying Cui 22 / 58
Linear program (LP) Two special cases of LP: Standard form LPs c T s.t. A = b 0 only inequalities are component-wise nonnegativity constraints Inequality form LPs no equality constraints c T s.t. A b SJTU Ying Cui 23 / 58
Linear program (LP) converting LPs to standard form sometimes useful to transform a general LP to a standard form LP (e.g., in order to use an algorithm for standard form LPs) introduce slack variables s i,i = 1,,m for inequalities:,s c T +d s.t. G +s = h A = b s 0 epress + and, i.e., = +, +, 0: +,,s c T + c T +d s.t. G + G +s = h A + A = b + 0, 0, s 0 SJTU Ying Cui 24 / 58
LP-eamples diet problem: choose quantities 1,, n of n foods one unit of food j costs c j, contains amount a ij of nutrient i healthy diet requires nutrient i in quantity at least b i to find cheapest healthy diet: piecewise-linear imization: c T s.t. A b, 0 ma i=1,,m (at i +b i ) equivalent to an LP by first forg the epigraph problem and then epressing the inequality as a set of m separate inequalities,t t s.t. a T i +b i t, i = 1,,m SJTU Ying Cui 25 / 58
LP-eamples Chebyshev center of a polyhedron: find the center c of the largest Euclidean ball B = { c +u u 2 r} that lies in a polyhedron P = { a T i b i,i = 1,,m} cheb Figure 8.5 Chebyshev center of a polyhedron C, in the Euclidean norm. The center cheb is the deepest point inside C, in the sense that it is farthest from the eterior, or complement, of C. The center cheb is also the center of the largest Euclidean ball (shown lightly shaded) that lies inside C. a T i b i for all B iff sup{a T i ( c +u) u 2 r} = a T i c +sup{a T i u u 2 r} = a T i c +r a i 2 b i Chebyshev center of polyhedron P can be detered by solving the LP c,r r s.t. a T i c +r a i 2 b i, i = 1,,m SJTU Ying Cui 26 / 58
Linear-fractional program imize a ratio of affine functions over a polyhedron: f 0 () = ct +d e T +f s.t. G h A = b (domf 0 () = { e T +f > 0}) f 0 () is quasiconve (quasilinear) so linear-fractional programs are quasiconve problems and can be solved by bisection method Transforg to an LP: let y = and z = 1 e T +f e T +f y,z c T y +dz s.t. Gy hz Ay = bz e T y +fz = 1 z 0 SJTU Ying Cui 27 / 58
Linear-fractional program generalized linear-fractional program ci T +d i f 0 () = ma i=1,,r ei T, domf 0 () = { ei T +f i > 0,i = 1,,r} +f i f 0 () is the pointwise maimum of r quasiconve functions, and therefore quasiconve, so this problem is quasiconve and can be solved by bisection method eample Von Neumann growth problem: allocate activity to maimize growth rate of slowest growing sector ma, + ) i=1,,n + i / i s.t. + 0, B + A, + R n : activity levels of n sectors, in current, net period (A) i, (B + ) i : produced, consumed amounts of good i + i / i : growth rate of sector i SJTU Ying Cui 28 / 58
Quadratic program (QP) (1/2) T P +q T +r s.t. G h A = b where P S n +, q Rn, r R, G R m n and A R p n objective function is conve quadratic and constraints are affine imize a conve quadratic function over a polyhedron QPs include LPs as a special case, by taking P = 0 f0( ) P Figure 4.5 Geometric illustration of QP. The feasible set P, which is a polyhedron, SJTU is shown shaded. The contour Ying lines Cuiof the objective function, which 29 / 58 is conve quadratic, are shown as dashed curves. The point is optimal.
QP-eamples least-squares: unconstrained QP (A R k n ) A b 2 2 = T A T A 2b T A +b T b analytical solution = A b singular value decomposition of A: A = UΣV T pseudo-inverse of A: A = VΣ 1 U T R n k constrained least-squares: add linear constraints, e.g., upper and lower bounds on A b 2 2 s.t. l u SJTU Ying Cui 30 / 58
QP-eamples linear program with random cost: imize risk-sensitive cost (a linear combination of epected cost and cost variance), captureing a trade-off between small epected cost and small cost variance c T +γ T Σ = Ec T +γvar(c T ) s.t. G h, A = b c R n is random vector with mean c and covariance E(c c)(c c) T ) = Σ for given, c T R is random variable with mean c T and variance E(c T Ec T ) 2 = T Σ risk aversion parameter γ > 0: control the trade-off between epected cost and cost variance (risk) SJTU Ying Cui 31 / 58
Quadratically constrained quadratic program (QCQP) (1/2) T P 0 +q T 0 +r 0 s.t. (1/2) T P i +q T i +r i 0, i = 1,,m A = b where P i S n +,i = 1,,m objective and inequality constraint functions are conve quadratic if P 1,,P m S n ++, feasible region is intersection of m ellipsoids and an affine set QCQPs include LPs as a special case, by taking P i = 0,i = 1,,m SJTU Ying Cui 32 / 58
Second-order cone programg (SOCP) f T s.t. A i +b i 2 ci T +d i, i = 1,,m F = g where f,c i R n, A i R n i n, d i R, F R p n and g R p each inequality constraint is a second-order cone (SOC) constraint: (A i +b i,c T i +d i ) second-order cone in R n i+1 more general than QCQP (and of course LP) if ci = 0, SOCP reduces to QCQP, by squaring each inequality constraint if Ai = 0, SOCP reduces to LP SJTU Ying Cui 33 / 58
SOCP-eample Robust linear programg: when parameters c,a i,b i c T s.t. ai T b i, i = 1,,m can be uncertain, two common approaches to handling uncertainty (in a i, for simplicity) deteristic model: a T i b i must hold for all a i E i c T s.t. ai T b i,for all a i E i i = 1,,m stochastic model: a i is random variable, and each constraint must hold with probability η (chance constraint) c T s.t. prob(ai T b i ) η i = 1,,m SJTU Ying Cui 34 / 58
SOCP-eample deteristic approach via SOCP choose an ellipsoid E i with center ā i R n, semi-aes detered by singular values/vectors of P i R n n : E i = {ā i +P i u u 2 1} a T i b i for all a i E i iff sup u 2 1(ā i +P i u) T = ā T i +sup u 2 1(P i u) T = ā T i + P T i 2 < b i robust LP c T s.t. ai T b i for all a i E i, i = 1,,m is equivalent to the SOCP c T s.t. āi T + Pi T 2 b i, i = 1,,m SJTU Ying Cui 35 / 58
SOCP-eample stochastic approach via SOCP Assume a i R n is Gaussian with mean ā i and covariance Σ i, i.e., a i N(ā i,σ i ) ai T is Gaussian r.v. with mean āi T and variance T Σ i : ( ) prob(ai T b i āi T b i ) = Φ Σ 1/2 i 2 where Φ() = (1/ 2π) ep t2 /2 dt is CDF of N(0,1) robust LP c T s.t. prob(a T i b i ) η, i = 1,,m with η 1/2, is equivalent to the SOCP c T s.t. ā T i +Φ 1 (η) Σ 1/2 i 2 b i i = 1,,m SJTU Ying Cui 36 / 58
Geometric programg (GP) monomial function: f : R n R with c > 0 and a i R f() = c a 1 1 a 2 2 an n, dom f = R n ++ monomials are closed under multiplication and division posynomial function (sum of monomials): f : R n R f() = K k=1 c k a 1k 1 a 2k 2 a nk n, dom f = R n ++ posynomials are closed under addition, multiplication, and nonnegative scaling a posynomial multiplied by a monomial is a posynomial a posynomial divided by a monomial is a posynomial SJTU Ying Cui 37 / 58
Geometric programg (GP) geometric program (GP) f 0 () s.t. f i () 1, i = 1,,m h i () = 1, i = 1,,p with f i posynomial and h i monomial, implying domain D = R n ++ (i.e., implicit constraint 0) etensions of GP: f posinomial and h (h i ) monomial f() h() f()/h() 1 (f/h posinomial) f() a (a > 0) f()/a 1 (f/a posinomial) h 1 () = h 2 () h 1 ()/h 2 () 1 (h 1 /h 2 monomial) maimize a nonzero monomial objective function, by imizing its inverse (which is also a monomial) SJTU Ying Cui 38 / 58
Geometric program in conve form GPs are not (in general) but can be transformed to conve problems by changing variables to y i = log i and taking log monomial f() = c a 1 1 a 2 2 an n transforms to logf(e y 1,,e yn ) = a T y +b (b = logc) posynomial f() = K k=1 c k a 1k 1 a 2k 2 a nk n transforms to K logf(e y 1,,e yn ) = log( e at k y+b k ) (b k = logc k ) k=1 geometric program transforms to conve problem log( s.t. log( K ep(a0k T y +b 0k)) k=1 K ep(aik T y +b ik)) 0, i = 1,,m k=1 Gy +d = 0 SJTU Ying Cui 39 / 58
GP-eamples design of cantilever beam: N segments with unit lengths and rectangular cross-sections of width w i and height h i, and given vertical force F applied at the right end, causing the beam to deflect (downward) and inducing stress in each segment segment 4 segment 3 segment 2 segment 1 Figure 4.6 Segmented cantilever beam with 4 segments. Each segment has unit length and a rectangular profile. A vertical force F is applied at the right end of the beam. F imize total weight subject to upper & lower bounds on w i,h i upper bound & lower bounds on aspect ratios h i /w i upper bound on stress in each segment upper bound on vertical deflection at the end of the beam variables: w i,h i for i = 1,,N SJTU Ying Cui 40 / 58
GP-eamples objective and constraint functions total weight w 1 h 1 + +w N h N is a posynomial aspect ratio h i /w i and inverse aspect ratio w i /h i are monomials maimum stress in segment i given by 6iF/(w i hi 2) is a monomial vertical deflection y i and slope v i of central ais at the right end of segment i are defined recursively as F v i = 12(i 1/2) Ew i hi 3 +v i+1 F y i = 6(i 1/3) Ew i hi 3 +v i+1 +y i+1 for i = N,N 1,...,1 with v N+1 = y N+1 = 0 (E: Young s modulus) v i and y i can be shown to be posynomial functions of w,h by induction SJTU Ying Cui 41 / 58
GP-eamples formulation as a GP w,h s.t. w 1 h 1 +...+w N h N wmaw 1 i 1, w w 1 i 1, i = 1,...,N hmah 1 i 1, h h 1 i 1, i = 1,...,N Smaw 1 1 i h i 1, S w i h 1 i 1, i = 1,...,N 6iFσmaw 1 1 i h 2 i 1, i = 1,...,N ymay 1 1 1 write w w i w ma and h h i h ma as w /w i 1, w i /w ma 1, h /h i 1, h i /h ma 1 write S h i /w i S ma as S w i /h i 1, h i /(w i S ma ) 1 SJTU Ying Cui 42 / 58
GP-eamples Minimizing spectral radius of nonnegative matri Perron-Frobenius eigenvalue: λ pf (A) suppose matri A R n n is (elementwise) positive and irreducible (i.e., matri (I +A) n 1 is elementwise positive) Perron-Frobenius theorem: A has a positive real eigenvalue λ pf (A) equal to its spectral radius, i.e., largest magnitude of its eigenvalues ma i λ i (A) λ pf (A) deteres asymptotic growth (decay) rate of A k (i.e., A k λ k pf ) as k and (1/λ pf)a) k converges as k alternative characterization: λ pf (A) = inf{λ Av λv for some v 0} Av λv can be epressed as n A() ij v j /(λv i ) 1, i = 1,...,n j=1 SJTU Ying Cui 43 / 58
GP-eamples imizing spectral radius of matri of posynomials: suppose entries of matri A are posynomial functions of some underlying variable R k imize λ pf (A()) possibly subject to posynomial inequalities on equivalent geometric program: λ,v, s.t. λ n A() ij v j /(λv i ) 1, i = 1,...,n j=1 SJTU Ying Cui 44 / 58
Generalized inequality constraints conve problem with generalized inequality constraints f 0 () s.t. f i () Ki 0, i = 1,...,m A = b f 0 : R n R conve; f i : R n R k i K i -conve w.r.t proper cone K i R k i results for ordinary conve optimization problems still hold the feasible set, any sublevel set, and the optimal set are conve any point that is locally optimal for the problem (4.48) is globally optimal. optimality condition for differentiable f0 : is optimal iff it is feasible and f 0 () T (y ) 0 for all y X SJTU Ying Cui 45 / 58
Conic form problems (cone programs) simplest conve optimization problems with generalized inequalities with a linear objective and one affine inequality constraint function c T s.t. F +g K 0 A = b generalize LPs (reduce to LPs when K = R m +) conic form problem in standard form c T s.t. A = b K 0 conic form problem in inequality form c T s.t. F +g K 0 SJTU Ying Cui 46 / 58
Semidefinite prograg (SDP) c T s.t. 1 F 1 + 2 F 2 +...+ n F n +G 0 A = b with G,F i S k and A R p n (note that K is S k +) inequality constraint is a linear matri inequality (LMI) G,F i are all diagonal, then the LMI is equivalent to a set of n linear inequalities, and the SDP reduces to an LP SJTU Ying Cui 47 / 58
Semidefinite prograg (SDP) common to refer to a problem with linear objective, linear equality and inequality constraints, and several LMI constraints, i.e., c T s.t. F (i)() = 1 F (i) 1 + 2 F (i) 2 +...+ n F (i) n +G (i) 0, i = 1,,K G h, A = b as an SDP, as it is readily transformed to an SDP by forg a large block diagonal LMI c T s.t. tr(g h,f (1) (),,F (K) ()) 0 A = b SJTU Ying Cui 48 / 58
Semidefinite prograg (SDP) standard form SDP with C,A i S n inequality form SDP with B,A i S k tr(cx) X s.t. tr(a i X) = b i, i = 1,,p X 0 c T s.t. 1 A 1 + 2 A 2 +...+ n A n B A = b SJTU Ying Cui 49 / 58
LP and SOCP as SDP LP and equivalent SDP LP: c T s.t. A b SDP : c T s.t. diag(a b) 0 (note different interpretation of generalized inequality ) SOCP and equivalent SDP SOCP : f T s.t. A i +b i 2 ci T +d i, i = 1,...,m SDP : s.t. f T [ ] (c T i +d i )I A i +b i (A i +b i ) T ci T 0, i = 1,...,m +d i SJTU Ying Cui 50 / 58
SDP-eamples eigenvalue imization R n λ ma (A()) where A() = A 0 + 1 A 1 +...+ n A n (with given A i S k ) equivalent SDP: t R n,t R s.t. A() ti constraint follows from λ ma (A) t A ti SJTU Ying Cui 51 / 58
SDP-eamples matri norm imization A() 2 = (λ ma (A() T A())) 1/2 R n where A() = A 0 + 1 A 1 +...+ n A n (with given A i R p q ) equivalent SDP: variables R n,t R constraint follows from t R n,t R [ ] ti A() s.t. A() T 0 ti A 2 t A T A t 2 I, t 0 [ ] ti A A T 0 ti SJTU Ying Cui 52 / 58
Vector optimization general vector optimization problem (w.r.t.k) f 0 () s.t. f i () 0, i = 1,...,m h i () = 0, i = 1,...,p with vector objective f 0 : R n R q imized w.r.t. proper cone K R q, and constraint functions f i : R n R, h i : R n R conve vector optimization problem (w.r.t.k) f 0 () s.t. f i () 0, i = 1,...,m A = b with f 0 K-conve, f i conve and h i affine SJTU Ying Cui 53 / 58
Optimal and Pareto optimal points set of achievable objective values O = {f 0 () feasible} R q f 0 () is the imum value of O (f 0 () K f 0 (y), y O): is optimal and f0 () is the optimal value most vector optimization problems do not have an optimal point (value), but this does occur in some special cases f 0 () is a imal value of O (if there eists y O such that f 0 (y) K f 0 (), then f 0 (y) = f 0 ()) is Pareto optimal and f0 () is a Pareto optimal value a vector optimization problem can have many Pareto optimal points (values) O O f0( po ) f0( ) Figure 4.7 The set O of achievable values for a vector optimization with objective values in R 2, with cone K = R 2 +, is shown shaded. In this case, the point labeled f0( ) is the optimal value of the problem, and is an optimal point. The objective value f0( ) can be compared to every other achievable value f0(y), and is better than or equal to f0(y). (Here, better than or equal to means is below and to the left of.) The lightly shaded region is f0( )+K, which is the set of all z R 2 corresponding to objective values worse than (or equal to) f0( ). Figure 4.8 The set O of achievable values for a vector optimization problem with objective values in R 2, with cone K = R 2 +, is shown shaded. This problem does not have an optimal point or value, but it does have a set of Pareto optimal points, whose corresponding values are shown as the darkened curve on the lower left boundary of O. The point labeled f0( po ) is a Pareto optimal value, and po is a Pareto optimal point. The lightly shaded region is f0( po ) K, which is the set of all z R 2 corresponding to objective values better than (or equal to) f0( po ). SJTU Ying Cui 54 / 58
Multicriterion optimization vector optimization problem with K = R q + f 0 () = (F 1 (),...,F q ()) q different scalar objectives F i ; roughly speaking we want all F i s to be small feasible is optimal if f 0 ( ) f 0 (y) for all feasible y if there eists an optimal point, the objectives are noncompeting feasible po is Pareto optimal if that there eists feasible y such that f 0 (y) f 0 ( po ) implies f 0 ( po ) = f 0 (y) if there are multiple Pareto optimal values, there is a trade-off between the objectives SJTU Ying Cui 55 / 58
Multicriterion optimization-eamples regularized least-squares (w.r.t.r 2 +) ( A b 2 2, 2 2) 15 F2() = 2 2 10 5 0 0 5 10 15 F1() = A b 2 2 Figure 4.11 Optimal trade-off curve for a regularized least-squares problem. The shaded set is the set of achievable values ( A b 2 2, 2 2). The optimal trade-off curve, shown darker, is the lower left part of the boundary. eample for A R 100 10 ; heavy line is formed by Pareto optimal points SJTU Ying Cui 56 / 58
Multicriterion optimization-eamples risk return trade-off in portfolio optimization (w.r.t.r 2 +) ( p T, T ) s.t. 1 T = 1, 0 R n is investment portfolio; i is fraction invested in asset i p R n is vector of relative asset price changes; modeled as a random variable with mean p and covariance p T = Er is epected return; T = var r is return var. 15% mean return 10% 5% 0% 1 0% 10% 20% (4) (3) (2) allocation 0.5 (1) 0 0% 10% 20% standard deviation of return Figure 4.12 Top. Optimal risk-return trade-off curve for a simple portfolio optimization problem. The lefthand endpoint corresponds to putting all resources in the risk-free asset, and so has zero standard deviation. The righthand endpoint corresponds to putting all resources in asset 1, which has highest mean return. Bottom. Corresponding optimal allocations. SJTU Ying Cui 57 / 58
Scalarization general vector optimization problems: find Pareto optimal points by choosing λ K 0 and solving scalar problem λ T f 0 () s.t. f i () 0, i = 1,...,m h i () = 0, i = 1,...,p optimal point for the scalar problem is Pareto optimal for vector optimization problem conve vector optimization problems: can find (almost) all Pareto optimal points by varying λ K 0 O f0(1) λ1 f0(3) f0(2) λ2 Figure 4.9 Scalarization. The set O of achievable values for a vector optimization problem with cone K = R 2 +. Three Pareto optimal values f0(1), f0(2), f0(3) are shown. The first two values can be obtained by scalarization: f0(1) imizes λ T 1 u over all u O and f0(2) imizes λ T 2 u, where λ1, λ2 0. The value f0(3) is Pareto optimal, but cannot be found by scalarization. SJTU Ying Cui 58 / 58