CS675: Convex and Combinatorial Optimization Fall 2016 Convex Optimization Problems. Instructor: Shaddin Dughmi

CS675: Convex and Combinatorial Optimization Fall 2016 Convex Optimization Problems Instructor: Shaddin Dughmi

Outline 1 Convex Optimization Basics 2 Common Classes 3 Interlude: Positive Semi-Definite Matrices 4 More Convex Optimization Problems

Convex Optimization Basics 0/21 Recall: Convex Optimization Problem A problem of minimizing a convex function (or maximizing a concave function) over a convex set. minimize subject to f(x) x X X R n is convex, and f : R n R is convex Terminology: decision variable(s), objective function, feasible set, optimal solution/value, ɛ-optimal solution/value

Standard Form Convex Optimization Basics 1/21 Instances typically formulated in the following standard form g i is convex minimize f(x) subject to g i (x) 0, for i C 1. a i x = b i, for i C 2. Terminology: equality constraints, inequality constraints, active/inactive at x, feasible/infeasible, unbounded In principle, every convex optimization problem can be formulated in this form (possibly implicitly) Recall: every convex set is the intersection of halfspaces

Local and Global Optimality Fact For a convex optimization problem, every locally optimal feasible solution is globally optimal. Convex Optimization Basics 2/21

Local and Global Optimality Fact For a convex optimization problem, every locally optimal feasible solution is globally optimal. Proof Let x be locally optimal, and y be any other feasible point. The line segment from x to y is contained in the feasible set. By local optimality f(x) f(θx + (1 θ)y) for θ sufficiently close to 1. Convex Optimization Basics 2/21

Convex Optimization Basics 3/21 Representation Typically, by problem we mean a family of instances, each of which is described either explicitly via problem parameters, or given implicitly via an oracle, or something in between. Oracle Representation At their most abstract, convex optimization problems of the following form minimize f(x) subject to x X are described via a separation oracle for X and epi f. Given additional data about instances of the problem, namely a range [L, H] for its optimal value and a ball of volume V containing X, the ellipsoid method returns an ɛ-optimal solution using only poly(n, log( H L ɛ ), log V ) oracle calls.

Representation Typically, by problem we mean a family of instances, each of which is described either explicitly via problem parameters, or given implicitly via an oracle, or something in between. In Between Consider the following fractional relaxation of the Traveling Salesman Problem, described by a network (V, E) and distances d e on e E. min e d ex e s.t. e δ(s) x e 2, S V, S. x 0 Representation of LP is implicit, in the form of a network. Using this representation, separation oracles can be implemented efficiently, and used as subroutines in the ellipsoid method. Convex Optimization Basics 3/21

Equivalence Convex Optimization Basics 4/21 Next up: we look at some common classes of convex optimization problems Technically, not all of them will be convex in their natural representation However, we will show that they are equivalent to a convex optimization problem

Convex Optimization Basics 4/21 Equivalence Next up: we look at some common classes of convex optimization problems Technically, not all of them will be convex in their natural representation However, we will show that they are equivalent to a convex optimization problem Equivalence Loosly speaking, two optimization problems are equivalent if an optimal solution to one can easily be translated into an optimal solution for the other.

Outline 1 Convex Optimization Basics 2 Common Classes 3 Interlude: Positive Semi-Definite Matrices 4 More Convex Optimization Problems

Linear Programming Common Classes 5/21 We have already seen linear programming minimize subject to c x Ax b

Linear Fractional Programming Common Classes 6/21 Generalizes linear programming minimize subject to c x+d e x+f Ax b e x + f 0 The objective is quasiconvex (in fact, quasilinear) over the halfspace where the denominator is nonnegative. Can be reformulated as an equivalent linear program 1 Change variables to y = x e x+f and z = 1 e x+f minimize subject to c y + dz Ay bz z 0 y = x e x+f z = 1 e x+f

Linear Fractional Programming Common Classes 6/21 Generalizes linear programming minimize subject to c x+d e x+f Ax b e x + f 0 The objective is quasiconvex (in fact, quasilinear) over the halfspace where the denominator is nonnegative. Can be reformulated as an equivalent linear program 1 Change variables to y = x e x+f and z = 1 e x+f 2 (y, z) is a solution to the above iff e y + fz = 1 minimize c y + dz subject to Ay bz z 0 y = x e x+f z = 1 e x+f e y + fz = 1

Example: Optimal Production Variant Common Classes 7/21 n products, m raw materials Every unit of product j uses a ij units of raw material i There are b i units of material i available Product j yields profit c j dollars per unit, and requires an investment of e j dollars per unit to produce, with f as a fixed cost Facility wants to maximize Return rate on investment c x e x+f maximize subject to a i x b i, for i = 1,..., m. x j 0, for j = 1,..., n.

Geometric Programming Definition A monomial is a function f : R n + R + of the form where c 0, a i R. f(x) = cx a 1 1 xa 2 2... xan n, A posynomial is a sum of monomials. Common Classes 8/21

Geometric Programming Common Classes 8/21 Definition A monomial is a function f : R n + R + of the form where c 0, a i R. f(x) = cx a 1 1 xa 2 2... xan n, A posynomial is a sum of monomials. A Geometric Program is an optimization problem of the following form minimize f 0 (x) subject to f i (x) b i, for i C 1. h i (x) = b i, for i C 2. x 0 where f i s are posynomials, h i s are monomials, and b i > 0 (wlog 1).

Geometric Programming Definition A monomial is a function f : R n + R + of the form where c 0, a i R. f(x) = cx a 1 1 xa 2 2... xan n, A posynomial is a sum of monomials. A Geometric Program is an optimization problem of the following form minimize f 0 (x) subject to f i (x) b i, for i C 1. h i (x) = b i, for i C 2. x 0 where f i s are posynomials, h i s are monomials, and b i > 0 (wlog 1). Interpretation GP model volume/area minimization problems, subject to constraints. Common Classes 8/21

Example: Designing a Suitcase Common Classes 9/21 A suitcase manufacturer is designing a suitcase Variables: h, w,d Want to minimize surface area 2(hw + hd + wd) (i.e. amount of material used) Have a target volume hwd 5 Practical/aesthetic constraints limit aspect ratio: h/w 2, h/d 3 Constrained by airline to h + w + d 7 minimize 2hw + 2hd + 2wd subject to h 1 w 1 d 1 1 5 hw 1 2 hd 1 3 h + w + d 7 h, w, d 0

Designing a Suitcase in Convex Form Common Classes 10/21 minimize 2hw + 2hd + 2wd subject to h 1 w 1 d 1 1 5 hw 1 2 hd 1 3 h + w + d 7 h, w, d 0

Designing a Suitcase in Convex Form Common Classes 10/21 minimize 2hw + 2hd + 2wd subject to h 1 w 1 d 1 1 5 hw 1 2 hd 1 3 h + w + d 7 h, w, d 0 Change of variables to h = log h, w = log w, d = log d minimize 2e h+ w + 2e h+ d + 2e w+ d subject to e h w d 1 5 e h w 2 e h d 3 e h + e w + e d 7

Geometric Programs in Convex Form Common Classes 11/21 minimize f 0 (x) subject to f i (x) b i, for i C 1. h i (x) = b i, for i C 2. x 0 where f i s are posynomials, h i s are monomials, and b i > 0 (wlog 1). In their natural parametrization by x 1,..., x n R +, geometric programs are not convex optimization problems However, the feasible set and objective function are convex in the variables y 1,..., y n R where y i = log x i

Geometric Programs in Convex Form Common Classes 11/21 minimize f 0 (x) subject to f i (x) b i, for i C 1. h i (x) = b i, for i C 2. x 0 where f i s are posynomials, h i s are monomials, and b i > 0 (wlog 1). Each monomial cx a 1 1 xa 2 2... xa k k function ce a 1y 1 +a 2 y 2 +...+a k y k can be rewritten as a convex Therefore, each posynomial becomes the sum of these convex exponential functions Inequality constraints and objective become convex Equality constraint cx a 1 1 xa 2 2... xa k k = b reduces to an affine constraint a 1 y 1 + a 2 y 2... a k y k = log b c

Outline 1 Convex Optimization Basics 2 Common Classes 3 Interlude: Positive Semi-Definite Matrices 4 More Convex Optimization Problems

Interlude: Positive Semi-Definite Matrices 12/21 Symmetric Matrices A matrix A R n n is symmetric if and only if it is square and A ij = A ji for all i, j. We denote the cone of n n symmetric matrices by S n.

Interlude: Positive Semi-Definite Matrices 12/21 Symmetric Matrices A matrix A R n n is symmetric if and only if it is square and A ij = A ji for all i, j. Fact We denote the cone of n n symmetric matrices by S n. A matrix A R n n is symmetric if and only if it is orthogonally diagonalizable.

Symmetric Matrices A matrix A R n n is symmetric if and only if it is square and A ij = A ji for all i, j. Fact We denote the cone of n n symmetric matrices by S n. A matrix A R n n is symmetric if and only if it is orthogonally diagonalizable. i.e. A = QDQ where Q is an orthogonal matrix and D = diag(λ 1,..., λ n ). The columns of Q are the (normalized) eigenvectors of A, with corresponding eigenvalues λ 1,..., λ n Equivalently: As a linear operator, A scales the space along an orthonormal basis Q The scaling factor λ i along direction q i may be negative, positive, or 0. Interlude: Positive Semi-Definite Matrices 12/21

Interlude: Positive Semi-Definite Matrices 13/21 Positive Semi-Definite Matrices A matrix A R n n is positive semi-definite if it is symmetric and moreover all its eigenvalues are nonnegative. We denote the cone of n n positive semi-definite matrices by S n + We use A 0 as shorthand for A S n + A = QDQ where Q is an orthogonal matrix and D = diag(λ 1,..., λ n ), where λ i 0. As a linear operator, A performs nonnegative scaling along an orthonormal basis Q

Interlude: Positive Semi-Definite Matrices 13/21 Positive Semi-Definite Matrices A matrix A R n n is positive semi-definite if it is symmetric and moreover all its eigenvalues are nonnegative. We denote the cone of n n positive semi-definite matrices by S n + We use A 0 as shorthand for A S n + Note A = QDQ where Q is an orthogonal matrix and D = diag(λ 1,..., λ n ), where λ i 0. As a linear operator, A performs nonnegative scaling along an orthonormal basis Q Positive definite, negative semi-definite, and negative definite defined similarly.

Interlude: Positive Semi-Definite Matrices 14/21 Geometric Intuition for PSD Matrices For A 0, let q 1,..., q n be the orthonormal eigenbasis for A, and let λ 1,..., λ n 0 be the corresponding eigenvalues. The linear operator x Ax scales the q i component of x by λ i When applied to every x in the unit ball, the image of A is an ellipsoid with principal directions q 1,..., q n and corresponding diameters 2λ 1,..., 2λ n When A is positive definite (i.e.λ i > 0), and therefore invertible, the ellipsoid is the set { x : x T A 1 x 1 }

Interlude: Positive Semi-Definite Matrices 15/21 Useful Properties of PSD Matrices If A 0, then x T Ax 0 for all x A has a positive semi-definite square root A 1 2 A 1 2 = Q diag( λ 1,..., λ n )Q A = BB T for some matrix B. Interpretation: PSD matrices encode the pairwise similarity relationships of a family of vectors Interpretation: The quadratic form x T Ax is the length of an affine transformation of x, namely Bx 2 2 The quadratic function x T Ax is convex A can be expressed as a sum of vector outer-products (xx ) E.g., sum of outerproducts of columns of B with A = BB T

Outline 1 Convex Optimization Basics 2 Common Classes 3 Interlude: Positive Semi-Definite Matrices 4 More Convex Optimization Problems

Quadratic Programming More Convex Optimization Problems 16/21 Minimizing a convex quadratic function over a polyhedron. minimize subject to x P x + c x + d Ax b P 0 Objective can be rewritten as (x x 0 ) P (x x 0 ) for some center x 0 Sublevel sets are scaled copies of an ellipsoid centered at x 0

More Convex Optimization Problems 17/21 Examples Constrained Least Squares Given a set of measurements (a 1, b 1 ),..., (a m, b m ), where a i R n is the i th input and b i R is the i th output, fit a linear function minimizing mean square error, subject to known bounds on the linear coefficients. minimize Ax b 2 2 = x A Ax 2b Ax + b b subject to l i x i u i, for i = 1,..., n.

Examples More Convex Optimization Problems 17/21 Distance Between Polyhedra Given two polyhedra Ax b and Cx d, find the distance between them. minimize z 2 2 = z Iz subject to z = y x Ax b By d

Conic Optimization Problems More Convex Optimization Problems 18/21 This is an umbrella term for problems of the following form minimize subject to c x Ax + b K Where K is a convex cone (e.g. R n +, positive semi-definite matrices, etc). Evidently, such optimization problems are convex.

Example: Second Order Cone Programming More Convex Optimization Problems 19/21 We will exhibit an example of a conic optimization problem with K as the second order cone K = {(x, t) : x 2 t}

Example: Second Order Cone Programming More Convex Optimization Problems 19/21 Linear Program with Random Constraints Consider the following optimization problem, where each a i is a gaussian random variable with mean a i and covariance matrix Σ i. minimize c x subject to a i x b i w.p. at least 0.9, for i = 1,..., m. u i := a i x is a univariate normal r.v. with mean u i := a i x and stddev σ i := x Σ i x = Σ 1 2 i x 2 u i b i with probability φ( b i u i σ i ), where φ is the CDF of the standard normal random variable.

Semi-Definite Programming More Convex Optimization Problems 20/21 These are conic optimization problems where the cone in question is the set of positive semi-definite matrices. minimize c x subject to x 1 F 1 + x 2 F 2... x n F n + G 0 Where F 1,..., F n are matrices, and refers to the positive semi-definite cone S n +. Examples Fitting a distribution, say a Gaussian, to observed data. Variable is a positive semi-definite covariance matrix. As a relaxation to combinatorial problems that encode pairwise relationships: e.g. finding the maximum cut of a graph.

More Convex Optimization Problems 21/21 Example: Max Cut Problem Given an undirected graph G = (V, E), find a partition of V into (S, V \ S) maximizing number of edges with exactly one end in S. (i,j) E 1 x i x j 2 maximize subject to x i { 1, 1}, for i V. Vector Program relaxation (i,j) E 1 x i x j 2 maximize subject to x i 2 = 1, for i V. x i R n, for i V.

More Convex Optimization Problems 21/21 Example: Max Cut Problem Given an undirected graph G = (V, E), find a partition of V into (S, V \ S) maximizing number of edges with exactly one end in S. (i,j) E 1 x i x j 2 maximize subject to x i { 1, 1}, for i V. Vector Program relaxation SDP Relaxation (i,j) E 1 x i x j 2 maximize subject to x i 2 = 1, for i V. x i R n, for i V. (i,j) E 1 X ij 2 maximize subject to X ii = 1, for i V. X S+ n