Conve Optimization Problems Prof. Daniel P. Palomar The Hong Kong University of Science and Technology (HKUST) MAFS6010R- Portfolio Optimization with R MSc in Financial Mathematics Fall 2018-19, HKUST, Hong Kong
Outline 1 Optimization Problems 2 Conve Optimization 3 Quasi-Conve Optimization 4 Classes of Conve Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers
Outline 1 Optimization Problems 2 Conve Optimization 3 Quasi-Conve Optimization 4 Classes of Conve Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers
Optimization Problem in Standard Form General optimization problem in standard form: f 0 () subject to f i () 0 i = 1,..., m h i () = 0 i = 1,..., p where = ( 1,..., n ) is the optimization variable f 0 : R n R is the objective function f i : R n R, i = 1,..., m are inequality constraint functions h i : R n R, i = 1,..., p are equality constraint functions. Goal: find an optimal solution that s f 0 while satisfying all the constraints. D. Palomar Conve Problems 4 / 48
Optimization Problem in Standard Form Feasibility: a point dom f 0 is feasible if it satisfies all the constraints and infeasible otherwise a problem is feasible if it has at least one feasible point and infeasible otherwise. Optimal value: p = inf {f 0 () f i () 0, i = 1,..., m, h i () = 0, i = 1,..., p} p = if problem infeasible (no satisfies the constraints) p = if problem unbounded below. Optimal solution: such that f ( ) = p (and feasible). D. Palomar Conve Problems 5 / 48
Global and Local Optimality A feasible is optimal if f 0 () = p ; X opt is the set of optimal points. A feasible is locally optimal if is optimal within a ball Eamples: f 0 () = 1/, dom f 0 = R ++ : p = 0, no optimal point f 0 () = 3 3: p =, local optimum at = 1. D. Palomar Conve Problems 6 / 48
Implicit Constraints The standard form optimization problem has an eplicit constraint: m D = dom f i i=0 p dom f i i=1 D is the domain of the problem The constraints f i () 0, h i () = 0 are the eplicit constraints A problem is unconstrained if it has no eplicit constraints Eample: log ( b a T ) is an unconstrained problem with implicit constraint b > a T. D. Palomar Conve Problems 7 / 48
Feasibility Problem Sometimes, we don t really want to any objective, just to find a feasible point: find subject to f i () 0 i = 1,..., m h i () = 0 i = 1,..., p This feasibility problem can be considered as a special case of a general problem: 0 subject to f i () 0 i = 1,..., m h i () = 0 i = 1,..., p where p = 0 if constraints are feasible and p = otherwise. D. Palomar Conve Problems 8 / 48
Outline 1 Optimization Problems 2 Conve Optimization 3 Quasi-Conve Optimization 4 Classes of Conve Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers
Conve Optimization Problem Conve optimization problem in standard form: f 0 () subject to f i () 0 i = 1,..., m A = b where f 0, f 1,..., f m are conve and equality constraints are affine. Local and global optima: any locally optimal point of a conve problem is globally optimal. Most problems are not conve when formulated. Reformulating a problem in conve form is an art, there is no systematic way. D. Palomar Conve Problems 10 / 48
Eample The following problem is nonconve (why not?): 2 1 + 2 2 subject to 1 / ( 1 + 2 2 ) 0 ( 1 + 2 ) 2 = 0 The objective is conve. The equality constraint function is not affine; however, we can rewrite it as 1 = 2 which is then a linear equality constraint. The inequality constraint function is not conve; however, we can rewrite it as 1 0 which again is linear. We can rewrite it as 2 1 + 2 2 subject to 1 0 1 = 2 D. Palomar Conve Problems 11 / 48
Equivalent Reformulations Eliminating/introducing equality constraints: is equivalent to f 0 () subject to f i () 0 i = 1,..., m A = b f 0 (Fz + 0 ) z subject to f i (Fz + 0 ) 0 i = 1,..., m where F and 0 are such that A = b = Fz + 0 for some z. D. Palomar Conve Problems 12 / 48
Equivalent Reformulations Introducing slack variables for linear inequalities: is equivalent to f 0 () subject to ai T b i i = 1,..., m,s f 0 () subject to ai T + s i = b i i = 1,..., m s i 0 D. Palomar Conve Problems 13 / 48
Equivalent Reformulations Epigraph form: a standard form conve problem is equivalent to t,t subject to f 0 () t 0 f i () 0 A = b Minimizing over some variables: is equivalent to where f 0 () = inf y f 0 (, y). i = 1,..., m,y f 0 (, y) subject to f i () 0 i = 1,..., m f 0 () subject to f i () 0 i = 1,..., m D. Palomar Conve Problems 14 / 48
Outline 1 Optimization Problems 2 Conve Optimization 3 Quasi-Conve Optimization 4 Classes of Conve Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers
Quasiconve Optimization Quasi-conve optimization problem in standard form: f 0 () subject to f i () 0 i = 1,..., m A = b where f 0 : R n R is quasiconve and f 1,..., f m are conve. Observe that it can have locally optimal points that are not (globally) optimal: D. Palomar Conve Problems 16 / 48
Quasiconve Optimization Conve representation of sublevel sets of a quasiconve function f 0 : there eists a family of conve functions φ t () for fied t such that Eample: f 0 () t φ t () 0. f 0 () = p () q () with p conve, q concave, and p () 0, q () > 0 on dom f 0. We can choose: φ t () = p () tq () for t 0, φ t () is conve in p () /q () t if and only if φ t () 0. D. Palomar Conve Problems 17 / 48
Quasiconve Optimization Solving a quasiconve problem via conve feasibility problems: the idea is to solve the epigraph form of the problem with a sandwich technique in t: for fied t the epigraph form of the original problem reduces to a feasibility conve problem φ t () 0, f i () 0 i, A b if t is too small, the feasibility problem will be infeasible if t is too large, the feasibility problem will be feasible start with upper and lower bounds on t (termed u and l, resp.) and use a sandwitch technique (bisection method): at each iteration use t = (l + u) /2 and update the bounds according to the feasibility/infeasibility of the problem. D. Palomar Conve Problems 18 / 48
Outline 1 Optimization Problems 2 Conve Optimization 3 Quasi-Conve Optimization 4 Classes of Conve Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers
Linear Programming (LP) LP: subject to c T + d G h A = b Conve problem: affine objective and constraint functions. Feasible set is a polyhedron: D. Palomar Conve Problems 20 / 48
l 1 - and l - Norm Problems as LPs l -norm minimization: is equivalent to the LP subject to G h A = b t, subject to t t1 t1 G h A = b. D. Palomar Conve Problems 21 / 48
l 1 - and l - Norm Problems as LPs l 1 -norm minimization: is equivalent to the LP subject to t, subject to 1 G h A = b i t i t t G h A = b. D. Palomar Conve Problems 22 / 48
Linear-Fractional Programming Linear-fractional programming: ( c T + d ) / ( e T + f ) subject to G h A = b with dom f 0 = { e T + f > 0 }. It is a quasiconve optimization problem (solved by bisection). Interestingly, the following LP is equivalent: y,z subject to c T y + dz Gy hz Ay = bz e T y + fz = 1 z 0 D. Palomar Conve Problems 23 / 48
Quadratic Programming (QP) Quadratic programming: subject to (1/2) T P + q T + r G h A = b Conve problem (assuming P S n 0): conve quadratic objective and affine constraint functions. Minimization of a conve quadratic function over a polyhedron: D. Palomar Conve Problems 24 / 48
Quadratically Constrained QP (QCQP) Quadratically constrained QP: (1/2) T P 0 + q T 0 + r 0 subject to (1/2) T P i + qi T + r i 0 i = 1,..., m A = b Conve problem (assuming P i S n 0): conve quadratic objective and constraint functions. D. Palomar Conve Problems 25 / 48
Second-Order Cone Programming (SOCP) Second-order cone programming: f T subject to A i + b i ci T + d i i = 1,..., m F = g Conve problem: linear objective and second-order cone constraints For A i row vector, it reduces to an LP. For c i = 0, it reduces to a QCQP. More general than QCQP and LP. D. Palomar Conve Problems 26 / 48
Semidefinite Programming (SDP) SDP: subject to c T 1 F 1 + 2 F 2 + + n F n G A = b Inequality constraint is called linear matri inequality (LMI). Conve problem: linear objective and linear matri inequality (LMI) constraints. Observe that multiple LMI constraints can always be written as a single one. D. Palomar Conve Problems 27 / 48
Semidefinite Programming (SDP) subject to LP and equivalent SDP: c T A b SOCP and equivalent SDP: c T subject to diag (A b) 0 f T subject to A i + b i ci T + d i, i = 1,..., m subject to f T [ ( ) c T i + d i I Ai + b i (A i + b i ) T c T i + d i ] 0, i = 1,..., m D. Palomar Conve Problems 28 / 48
Semidefinite Programming (SDP) Eigenvalue minimization: λ ma (A ()) where A () = A 0 + 1 A 1 + + n A n, is equivalent to SDP subject to t A () ti It follows from λ ma (A ()) t A () ti D. Palomar Conve Problems 29 / 48
Outline 1 Optimization Problems 2 Conve Optimization 3 Quasi-Conve Optimization 4 Classes of Conve Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers
Scalarization for Multicriterion Problems Consider a multicriterion optimization with q different objectives: f 0 () = (F 1 (),..., F q ()). To find Pareto optimal points, the positive weighted sum: λ T f 0 () = λ 1 F 1 () + + λ q F q (). Eample: regularized least-squares: A b 2 2 + γ 2 2. D. Palomar Conve Problems 31 / 48
References on Conve Problems Chapter 4 of Stephen Boyd and Lieven Vandenberghe, Conve Optimization. Cambridge, U.K.: Cambridge University Press, 2004. https://web.stanford.edu/~boyd/cvbook/ D. Palomar Conve Problems 32 / 48
Outline 1 Optimization Problems 2 Conve Optimization 3 Quasi-Conve Optimization 4 Classes of Conve Problems: LP, QP, SOCP, SDP 5 Multicriterion Optimization (Pareto Optimality) 6 Solvers
Solvers A solver is an engine for solving a particular type of mathematical problem, such as a conve program. Every programming language (e.g., Matlab, Octave, R, Python, C, C++) has a long list of available solvers to choose from. Solvers typically handle only a certain class of problems, such as LPs, QPs, SOCPs, SDPs, or GPs. They also require that problems be epressed in a standard form. Most problems do not immediately present themselves in a standard form, so they must be transformed into standard form. D. Palomar Conve Problems 34 / 48
Solver Eample: Matlab s linprog A program for solving LPs: = linprog( c, A, b, A_eq, B_eq, l, u ) Problems must be epressed in the following standard form: subject to c T A b A eq = b eq l u D. Palomar Conve Problems 35 / 48
Conversion to Standard Form: Common Tricks Representing free variables as the difference of nonnegative variables: free = +, + 0, 0 Elimininating inequality constraints using slack variables: a T b = a T + s = b, s 0 Splitting equality constraints into inequalities: a T = b = a T b, a T b D. Palomar Conve Problems 36 / 48
Solver Eample: SeDuMi A program for solving LPs, SOCPs, SDPs, and related problems: = sedumi( A, b, c, K ) Solves problems of the form: subject to c T A = b K K 1 K 2 K L where each set K i R n i, i = 1, 2,..., L is chosen from a very short list of cones. The Matlab variable K gives the number, types, and dimensions of the cones K i. D. Palomar Conve Problems 37 / 48
The cones must be arranged in this order, i.e., the free variables first, then the nonnegative orthants, then the second-order cones, then the semidefinite cones. D. Palomar Conve Problems 38 / 48 Solver Eample: SeDuMi Cones supported by SeDuMi: free variables: R n i a nonnegative orthant: R n i + (for linear inequalities) a real or comple second-order cone: Q n {(, y) R n R 2 y} Q n c {(, y) C n R 2 y} a real or comple semidefinite cone: { } S n + X R n n X = X T, X 0 { } H n + X C n n X = X H, X 0
Eample: Norm Approimation Consider the norm approimation problem: A b An optimal value s the residuals r k = a T k b k, k = 1, 2,..., m according to the measure defined by the norm Obviously, the value of depends significantly upon the choice of that norm...... and so does the process of conversion to standard form D. Palomar Conve Problems 39 / 48
Eample: Euclidean or l 2 -Norm Norm approimation problem: ( m ( ) ) A b 2 = k=1 a T 2 1/2 k b k No need to use any solver here: this is a least squares (LS) problem, with an analytic solution: = (A T A) 1 A T b In Matlab or Octave, a single command computes the solution: > > = A \ b Similarly, in R: > > = solve(a, b) D. Palomar Conve Problems 40 / 48
Eample: Chebyshev or l -Norm Norm approimation problem: A b = ma 1 k m a T k b k This can be epressed as a linear program:,t subject to t t1 A b t1 or, equivalently,,t subject to [ 0 T 1 ] [ ] [ ] t [ A 1 A 1 t ] [ b b ] D. Palomar Conve Problems 41 / 48
Eample: Chebyshev or l -Norm Recall the final formulation: [ 0 T 1 ] [ ],t [ ] t [ A 1 subject to A 1 t ] [ b b ] Matlab s linprog call: > > t = linprog( [zeros(n,1); 1],... [A,-ones(m,1); -A,-ones(m,1)],... [b; -b] ) > > = t(1:n) D. Palomar Conve Problems 42 / 48
Eample: Manhattan or l 1 -Norm Norm approimation problem: A b 1 = m k=1 a T k b k This can be epressed as a linear program:,t subject to 1 T t t A b t or, equivalently,,t subject to [ 0 T 1 T ] [ ] [ ] [ t ] A I A I t [ b b ] D. Palomar Conve Problems 43 / 48
Eample: Manhattan or l 1 -Norm Recall the final formulation: [ 0 T 1 T ] [ ],t [ ] [ t ] A I subject to A I t [ b b ] Matlab s linprog call: > > t = linprog( [zeros(n,1); ones(n,1)],... [A,-eye(m,1); -A,-eye(m,1)],... [b; -b] ) > > = t(1:n) D. Palomar Conve Problems 44 / 48
Eample: Constrained Euclidean or l 2 -Norm Constrained norm approimation problem: subject to A b 2 C = d l u This is not a least squares problem, but it is QP and an SOCP. This can be epressed as,y,t,s l,s u subject to t A b = y C = d s l = l + s u = u s l, s u 0 y 2 t D. Palomar Conve Problems 45 / 48
Eample: Constrained Euclidean or l 2 -Norm Equivalently: SeDuMi call: [ 0 T 0 T 0 T 0 T 1 ],y,t,s l,s u A I subject to C I I I I R n R n + R n + Q m > > AA = [ A, zeros(m,n), zeros(m,n), -eye(m), 0; C, zeros(p,n), zeros(p,n) zeros(p,n), 0; eye(n), -eye(n), zeros(n,n), zeros(n,n), 0; eye(n), zeros(n,n), eye(n), zeros(n,n), 0 ] > > bb = [ b; d; l; u ] > > cc = [ zeros(3*n+m,1); 1 ] > > K.f = n; K.l = 2*n; K.q = m + 1; > > syz = sedumi( AA, bb, cc, K ) > > = syz(1:n) D. Palomar Conve Problems 46 / 48 b d l u
Modeling Frameworks: cv A modeling framework simplifies the use of a numerical technology by shielding the user from the underlying mathematical details. Eamples: SDPSOL, YALMIP, cv, etc. cv is designed to support conve optimization or, more specifically, disciplined conve programming (available in Matlab, Python, R, Julia, Octave, etc.) People don t simply write optimization problems and hope that they are conve; instead they draw from a "mental library"of functions and sets with known conveity properties and combine them in ways that conve analysis guarantees will produce conve results. Disciplined conve programming formalizes this methodology. Links: cv: http://cvr.com cv user guide: http://web.cvr.com/cv/doc/cvx.pdf cv for R (cvr): https://github.com/anqif/cvr D. Palomar Conve Problems 47 / 48
Thanks For more information visit: https://www.danielppalomar.com