1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

Transcription

1 1. Introduction mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history of convex optimization 1 1

2 Mathematical optimization (mathematical) optimization problem minimize f 0 (x) subject to f i (x) b i, i = 1,...,m x = (x 1,...,x n ): optimization variables f 0 : R n R: objective function f i : R n R, i = 1,...,m: constraint functions optimal solution x has smallest value of f 0 among all vectors that satisfy the constraints Introduction 1 2

3 Examples portfolio optimization variables: amounts invested in different assets constraints: budget, max./min. investment per asset, minimum return objective: overall risk or return variance device sizing in electronic circuits variables: device widths and lengths constraints: manufacturing limits, timing requirements, maximum area objective: power consumption data fitting variables: model parameters constraints: prior information, parameter limits objective: measure of misfit or prediction error Introduction 1 3

4 Solving optimization problems general optimization problem very difficult to solve methods involve some compromise, e.g., very long computation time, or not always finding the solution exceptions: certain problem classes can be solved efficiently and reliably least-squares problems linear programming problems convex optimization problems Introduction 1 4

5 Least-squares minimize Ax b 2 2 solving least-squares problems analytical solution: x = (A T A) 1 A T b reliable and efficient algorithms and software computation time proportional to n 2 k (A R k n ); less if structured a mature technology using least-squares least-squares problems are easy to recognize a few standard techniques increase flexibility (e.g., including weights, adding regularization terms) Introduction 1 5

6 Linear programming solving linear programs no analytical formula for solution minimize c T x subject to a T i x b i, i = 1,...,m reliable and efficient algorithms and software computation time proportional to n 2 m if m n; less with structure a mature technology using linear programming not as easy to recognize as least-squares problems a few standard tricks used to convert problems into linear programs (e.g., problems involving l 1 - or l -norms, piecewise-linear functions) Introduction 1 6

7 Convex optimization problem minimize f 0 (x) subject to f i (x) b i, i = 1,...,m objective and constraint functions are convex: f i (αx+βy) αf i (x)+βf i (y) if α+β = 1, α 0, β 0 includes least-squares problems and linear programs as special cases Introduction 1 7

8 solving convex optimization problems no analytical solution reliable and efficient algorithms computation time (roughly) proportional to max{n 3,n 2 m,f}, where F is cost of evaluating f i s and their first and second derivatives almost a technology using convex optimization often difficult to recognize many tricks for transforming problems into convex form surprisingly many problems can be solved via convex optimization Introduction 1 8

9 Example m lamps illuminating n (small, flat) patches lamp power p j θ kj r kj illumination I k intensity I k at patch k depends linearly on lamp powers p j : I k = m a kj p j, j=1 a kj = r 2 kj max{cosθ kj,0} problem: achieve desired illumination I des with bounded lamp powers minimize max k=1,...,n logi k logi des subject to 0 p j p max, j = 1,...,m Introduction 1 9

10 how to solve? 1. use uniform power: p j = p, vary p 2. use least-squares: minimize n k=1 (I k I des ) 2 round p j if p j > p max or p j < 0 3. use weighted least-squares: n minimize k=1 (I k I des ) 2 + m j=1 w j(p j p max /2) 2 iteratively adjust weights w j until 0 p j p max 4. use linear programming: minimize max k=1,...,n I k I des subject to 0 p j p max, j = 1,...,m which can be solved via linear programming of course these are approximate (suboptimal) solutions Introduction 1 10

11 5. use convex optimization: problem is equivalent to with h(u) = max{u,1/u} minimize f 0 (p) = max k=1,...,n h(i k /I des ) subject to 0 p j p max, j = 1,...,m h(u) f 0 is convex because maximum of convex functions is convex exact solution obtained with effort modest factor least-squares effort u Introduction 1 11

12 additional constraints: does adding 1 or 2 below complicate the problem? 1. no more than half of total power is in any 10 lamps 2. no more than half of the lamps are on (p j > 0) answer: with (1), still easy to solve; with (2), extremely difficult moral: (untrained) intuition doesn t always work; without the proper background very easy problems can appear quite similar to very difficult problems Introduction 1 12

13 Course goals and topics goals 1. recognize/formulate problems (such as the illumination problem) as convex optimization problems 2. develop code for problems of moderate size (1000 lamps, 5000 patches) 3. characterize optimal solution (optimal power distribution), give limits of performance, etc. topics 1. convex sets, functions, optimization problems 2. examples and applications 3. algorithms Introduction 1 13

14 Nonlinear optimization traditional techniques for general nonconvex problems involve compromises local optimization methods (nonlinear programming) find a point that minimizes f 0 among feasible points near it fast, can handle large problems require initial guess provide no information about distance to (global) optimum global optimization methods find the (global) solution worst-case complexity grows exponentially with problem size these algorithms are often based on solving convex subproblems Introduction 1 14

15 Brief history of convex optimization theory (convex analysis): ca algorithms 1947: simplex algorithm for linear programming (Dantzig) 1960s: early interior-point methods (Fiacco & McCormick, Dikin,... ) 1970s: ellipsoid method and other subgradient methods 1980s: polynomial-time interior-point methods for linear programming (Karmarkar 1984) late 1980s now: polynomial-time interior-point methods for nonlinear convex optimization (Nesterov & Nemirovski 1994) applications before 1990: mostly in operations research; few in engineering since 1990: many new applications in engineering (control, signal processing, communications, circuit design,... ); new problem classes (semidefinite and second-order cone programming, robust optimization) Introduction 1 15

16 2. Convex sets Convex Optimization Boyd & Vandenberghe affine and convex sets some important examples operations that preserve convexity generalized inequalities separating and supporting hyperplanes dual cones and generalized inequalities 2 1

17 Affine set line through x 1, x 2 : all points x = θx 1 +(1 θ)x 2 (θ R) x 1 θ = 1.2 θ = 1 θ = 0.6 x 2 θ = 0 θ = 0.2 affine set: contains the line through any two distinct points in the set example: solution set of linear equations {x Ax = b} (conversely, every affine set can be expressed as solution set of system of linear equations) Convex sets 2 2

18 Convex set line segment between x 1 and x 2 : all points x = θx 1 +(1 θ)x 2 with 0 θ 1 convex set: contains line segment between any two points in the set x 1,x 2 C, 0 θ 1 = θx 1 +(1 θ)x 2 C examples (one convex, two nonconvex sets) Convex sets 2 3

19 Convex combination and convex hull convex combination of x 1,..., x k : any point x of the form x = θ 1 x 1 +θ 2 x 2 + +θ k x k with θ 1 + +θ k = 1, θ i 0 convex hull convs: set of all convex combinations of points in S Convex sets 2 4

20 Convex cone conic (nonnegative) combination of x 1 and x 2 : any point of the form x = θ 1 x 1 +θ 2 x 2 with θ 1 0, θ x 1 convex cone: set that contains all conic combinations of points in the set x 2 Convex sets 2 5

21 Hyperplanes and halfspaces hyperplane: set of the form {x a T x = b} (a 0) a x 0 x a T x = b halfspace: set of the form {x a T x b} (a 0) a is the normal vector x 0 a T x b a a T x b hyperplanes are affine and convex; halfspaces are convex Convex sets 2 6

22 Euclidean balls and ellipsoids (Euclidean) ball with center x c and radius r: B(x c,r) = {x x x c 2 r} = {x c +ru u 2 1} ellipsoid: set of the form {x (x x c ) T P 1 (x x c ) 1} with P S n ++ (i.e., P symmetric positive definite) x c other representation: {x c +Au u 2 1} with A square and nonsingular Convex sets 2 7

23 norm: a function that satisfies Norm balls and norm cones x 0; x = 0 if and only if x = 0 tx = t x for t R x+y x + y notation: is general (unspecified) norm; symb is particular norm norm ball with center x c and radius r: {x x x c r} norm cone: {(x,t) x t} Euclidean norm cone is called secondorder cone norm balls and cones are convex t x x Convex sets 2 8

24 Polyhedra solution set of finitely many linear inequalities and equalities Ax b, Cx = d (A R m n, C R p n, is componentwise inequality) a 5 a 1 a2 P a 3 a 4 polyhedron is intersection of finite number of halfspaces and hyperplanes Convex sets 2 9

25 Positive semidefinite cone notation: S n is set of symmetric n n matrices S n + = {X S n X 0}: positive semidefinite n n matrices X S n + z T Xz 0 for all z S n + is a convex cone S n ++ = {X S n X 0}: positive definite n n matrices example: [ x y y z ] S 2 + z y 1 0 x Convex sets 2 10

26 Operations that preserve convexity practical methods for establishing convexity of a set C 1. apply definition x 1,x 2 C, 0 θ 1 = θx 1 +(1 θ)x 2 C 2. show that C is obtained from simple convex sets (hyperplanes, halfspaces, norm balls,... ) by operations that preserve convexity intersection affine functions perspective function linear-fractional functions Convex sets 2 11

27 Intersection the intersection of (any number of) convex sets is convex example: S = {x R m p(t) 1 for t π/3} where p(t) = x 1 cost+x 2 cos2t+ +x m cosmt for m = 2: p(t) π/3 2π/3 π t x S x 1 Convex sets 2 12

28 Affine function suppose f : R n R m is affine (f(x) = Ax+b with A R m n, b R m ) the image of a convex set under f is convex S R n convex = f(s) = {f(x) x S} convex the inverse image f 1 (C) of a convex set under f is convex examples C R m convex = f 1 (C) = {x R n f(x) C} convex scaling, translation, projection solution set of linear matrix inequality {x x 1 A 1 + +x m A m B} (with A i,b S p ) hyperbolic cone {x x T Px (c T x) 2, c T x 0} (with P S n +) Convex sets 2 13

29 Perspective and linear-fractional function perspective function P : R n+1 R n : P(x,t) = x/t, domp = {(x,t) t > 0} images and inverse images of convex sets under perspective are convex linear-fractional function f : R n R m : f(x) = Ax+b c T x+d, domf = {x ct x+d > 0} images and inverse images of convex sets under linear-fractional functions are convex Convex sets 2 14

30 example of a linear-fractional function f(x) = 1 x 1 +x 2 +1 x x2 1 0 C x 1 x2 1 0 f(c) x 1 Convex sets 2 15

31 Generalized inequalities a convex cone K R n is a proper cone if K is closed (contains its boundary) K is solid (has nonempty interior) K is pointed (contains no line) examples nonnegative orthant K = R n + = {x R n x i 0,i = 1,...,n} positive semidefinite cone K = S n + nonnegative polynomials on [0, 1]: K = {x R n x 1 +x 2 t+x 3 t 2 + +x n t n 1 0 for t [0,1]} Convex sets 2 16

32 generalized inequality defined by a proper cone K: x K y y x K, x K y y x intk examples componentwise inequality (K = R n +) x R n + y x i y i, i = 1,...,n matrix inequality (K = S n +) X S n + Y Y X positive semidefinite these two types are so common that we drop the subscript in K properties: many properties of K are similar to on R, e.g., x K y, u K v = x+u K y +v Convex sets 2 17

33 Minimum and minimal elements K is not in general a linear ordering: we can have x K y and y K x x S is the minimum element of S with respect to K if y S = x K y x S is a minimal element of S with respect to K if example (K = R 2 +) x 1 is the minimum element of S 1 x 2 is a minimal element of S 2 y S, y K x = y = x x1 S 2 S 1 x 2 Convex sets 2 18

34 Separating hyperplane theorem if C and D are disjoint convex sets, then there exists a 0, b such that a T x b for x C, a T x b for x D a D a T x b the hyperplane {x a T x = b} separates C and D a T x b C strict separation requires additional assumptions (e.g., C is closed, D is a singleton) Convex sets 2 19

35 Supporting hyperplane theorem supporting hyperplane to set C at boundary point x 0 : {x a T x = a T x 0 } where a 0 and a T x a T x 0 for all x C supporting hyperplane theorem: if C is convex, then there exists a supporting hyperplane at every boundary point of C C x 0 a Convex sets 2 20

36 Dual cones and generalized inequalities dual cone of a cone K: K = {y y T x 0 for all x K} examples K = R n +: K = R n + K = S n +: K = S n + K = {(x,t) x 2 t}: K = {(x,t) x 2 t} K = {(x,t) x 1 t}: K = {(x,t) x t} first three examples are self-dual cones dual cones of proper cones are proper, hence define generalized inequalities: y K 0 y T x 0 for all x K 0 Convex sets 2 21

37 Minimum and minimal elements via dual inequalities minimum element w.r.t. K x is minimum element of S iff for all λ K 0, x is the unique minimizer of λ T z over S x S minimal element w.r.t. K if x minimizes λ T z over S for some λ K 0, then x is minimal x 1 λ 1 S x 2 if x is a minimal element of a convex set S, then there exists a nonzero λ K 0 such that x minimizes λ T z over S λ 2 Convex sets 2 22

38 optimal production frontier different production methods use different amounts of resources x R n production set P: resource vectors x for all possible production methods efficient (Pareto optimal) methods correspond to resource vectors x that are minimal w.r.t. R n + example (n = 2) x 1, x 2, x 3 are efficient; x 4, x 5 are not fuel x 1 x 2 x5 x 4 λ P x 3 labor Convex sets 2 23

39 3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions convexity with respect to generalized inequalities 3 1

40 Definition f : R n R is convex if domf is a convex set and f(θx+(1 θ)y) θf(x)+(1 θ)f(y) for all x,y domf, 0 θ 1 (x,f(x)) (y,f(y)) f is concave if f is convex f is strictly convex if domf is convex and f(θx+(1 θ)y) < θf(x)+(1 θ)f(y) for x,y domf, x y, 0 < θ < 1 Convex functions 3 2

41 Examples on R convex: affine: ax+b on R, for any a,b R exponential: e ax, for any a R powers: x α on R ++, for α 1 or α 0 powers of absolute value: x p on R, for p 1 negative entropy: xlogx on R ++ concave: affine: ax+b on R, for any a,b R powers: x α on R ++, for 0 α 1 logarithm: logx on R ++ Convex functions 3 3

42 Examples on R n and R m n affine functions are convex and concave; all norms are convex examples on R n affine function f(x) = a T x+b norms: x p = ( n i=1 x i p ) 1/p for p 1; x = max k x k examples on R m n (m n matrices) affine function f(x) = tr(a T X)+b = spectral (maximum singular value) norm m i=1 n A ij X ij +b j=1 f(x) = X 2 = σ max (X) = (λ max (X T X)) 1/2 Convex functions 3 4

43 Restriction of a convex function to a line f : R n R is convex if and only if the function g : R R, g(t) = f(x+tv), domg = {t x+tv domf} is convex (in t) for any x domf, v R n can check convexity of f by checking convexity of functions of one variable example. f : S n R with f(x) = logdetx, domf = S n ++ g(t) = logdet(x +tv) = logdetx +logdet(i +tx 1/2 VX 1/2 ) n = logdetx + log(1+tλ i ) where λ i are the eigenvalues of X 1/2 VX 1/2 g is concave in t (for any choice of X 0, V); hence f is concave i=1 Convex functions 3 5

44 Extended-value extension extended-value extension f of f is f(x) = f(x), x domf, f(x) =, x domf often simplifies notation; for example, the condition 0 θ 1 = f(θx+(1 θ)y) θ f(x)+(1 θ) f(y) (as an inequality in R { }), means the same as the two conditions domf is convex for x,y domf, 0 θ 1 = f(θx+(1 θ)y) θf(x)+(1 θ)f(y) Convex functions 3 6

45 First-order condition f is differentiable if domf is open and the gradient f(x) = exists at each x domf ( f(x), f(x),..., f(x) ) x 1 x 2 x n 1st-order condition: differentiable f with convex domain is convex iff f(y) f(y) f(x)+ f(x) T (y x) for all x,y domf (x,f(x)) f(x) + f(x) T (y x) first-order approximation of f is global underestimator Convex functions 3 7

46 Second-order conditions f is twice differentiable if domf is open and the Hessian 2 f(x) S n, exists at each x domf 2 f(x) ij = 2 f(x) x i x j, i,j = 1,...,n, 2nd-order conditions: for twice differentiable f with convex domain f is convex if and only if 2 f(x) 0 for all x domf if 2 f(x) 0 for all x domf, then f is strictly convex Convex functions 3 8

47 Examples quadratic function: f(x) = (1/2)x T Px+q T x+r (with P S n ) f(x) = Px+q, 2 f(x) = P convex if P 0 least-squares objective: f(x) = Ax b 2 2 convex (for any A) f(x) = 2A T (Ax b), quadratic-over-linear: f(x,y) = x 2 /y 2 f(x,y) = 2 y 3 [ y x convex for y > 0 ][ y x ] T 0 2 f(x) = 2A T A f(x,y) y 0 2 x 0 2 Convex functions 3 9

48 log-sum-exp: f(x) = log n k=1 expx k is convex 2 f(x) = 1 1 T z diag(z) 1 (1 T z) 2zzT (z k = expx k ) to show 2 f(x) 0, we must verify that v T 2 f(x)v 0 for all v: v T 2 f(x)v = ( k z kv 2 k )( k z k) ( k v kz k ) 2 ( k z k) 2 0 since ( k v kz k ) 2 ( k z kv 2 k )( k z k) (from Cauchy-Schwarz inequality) geometric mean: f(x) = ( n k=1 x k) 1/n on R n ++ is concave (similar proof as for log-sum-exp) Convex functions 3 10

49 Epigraph and sublevel set α-sublevel set of f : R n R: C α = {x domf f(x) α} sublevel sets of convex functions are convex (converse is false) epigraph of f : R n R: epif = {(x,t) R n+1 x domf, f(x) t} epif f f is convex if and only if epif is a convex set Convex functions 3 11

50 Jensen s inequality basic inequality: if f is convex, then for 0 θ 1, f(θx+(1 θ)y) θf(x)+(1 θ)f(y) extension: if f is convex, then f(ez) Ef(z) for any random variable z basic inequality is special case with discrete distribution prob(z = x) = θ, prob(z = y) = 1 θ Convex functions 3 12

51 Operations that preserve convexity practical methods for establishing convexity of a function 1. verify definition (often simplified by restricting to a line) 2. for twice differentiable functions, show 2 f(x) 0 3. show that f is obtained from simple convex functions by operations that preserve convexity nonnegative weighted sum composition with affine function pointwise maximum and supremum composition minimization perspective Convex functions 3 13

52 Positive weighted sum & composition with affine function nonnegative multiple: αf is convex if f is convex, α 0 sum: f 1 +f 2 convex if f 1,f 2 convex (extends to infinite sums, integrals) composition with affine function: f(ax+b) is convex if f is convex examples log barrier for linear inequalities m f(x) = log(b i a T i x), domf = {x a T i x < b i,i = 1,...,m} i=1 (any) norm of affine function: f(x) = Ax+b Convex functions 3 14

53 Pointwise maximum if f 1,..., f m are convex, then f(x) = max{f 1 (x),...,f m (x)} is convex examples piecewise-linear function: f(x) = max i=1,...,m (a T i x+b i) is convex sum of r largest components of x R n : f(x) = x [1] +x [2] + +x [r] is convex (x [i] is ith largest component of x) proof: f(x) = max{x i1 +x i2 + +x ir 1 i 1 < i 2 < < i r n} Convex functions 3 15

54 Pointwise supremum if f(x,y) is convex in x for each y A, then is convex examples g(x) = supf(x,y) y A support function of a set C: S C (x) = sup y C y T x is convex distance to farthest point in a set C: f(x) = sup x y y C maximum eigenvalue of symmetric matrix: for X S n, λ max (X) = sup y T Xy y 2 =1 Convex functions 3 16

55 Composition with scalar functions composition of g : R n R and h : R R: f is convex if f(x) = h(g(x)) g convex, h convex, h nondecreasing g concave, h convex, h nonincreasing proof (for n = 1, differentiable g,h) f (x) = h (g(x))g (x) 2 +h (g(x))g (x) note: monotonicity must hold for extended-value extension h examples expg(x) is convex if g is convex 1/g(x) is convex if g is concave and positive Convex functions 3 17

56 Vector composition composition of g : R n R k and h : R k R: f(x) = h(g(x)) = h(g 1 (x),g 2 (x),...,g k (x)) f is convex if g i convex, h convex, h nondecreasing in each argument g i concave, h convex, h nonincreasing in each argument proof (for n = 1, differentiable g,h) examples f (x) = g (x) T 2 h(g(x))g (x)+ h(g(x)) T g (x) m i=1 logg i(x) is concave if g i are concave and positive log m i=1 expg i(x) is convex if g i are convex Convex functions 3 18

57 Minimization if f(x,y) is convex in (x,y) and C is a convex set, then is convex examples g(x) = inf y C f(x,y) f(x,y) = x T Ax+2x T By +y T Cy with [ A B B T C ] 0, C 0 minimizing over y gives g(x) = inf y f(x,y) = x T (A BC 1 B T )x g is convex, hence Schur complement A BC 1 B T 0 distance to a set: dist(x,s) = inf y S x y is convex if S is convex Convex functions 3 19

58 Perspective the perspective of a function f : R n R is the function g : R n R R, g(x,t) = tf(x/t), domg = {(x,t) x/t domf, t > 0} g is convex if f is convex examples f(x) = x T x is convex; hence g(x,t) = x T x/t is convex for t > 0 negative logarithm f(x) = logx is convex; hence relative entropy g(x,t) = tlogt tlogx is convex on R 2 ++ if f is convex, then g(x) = (c T x+d)f ( (Ax+b)/(c T x+d) ) is convex on {x c T x+d > 0, (Ax+b)/(c T x+d) domf} Convex functions 3 20

59 the conjugate of a function f is The conjugate function f (y) = sup (y T x f(x)) x dom f f(x) f is convex (even if f is not) will be useful in chapter 5 (0, f (y)) xy x Convex functions 3 21

60 examples negative logarithm f(x) = logx f (y) = sup(xy +logx) = x>0 { 1 log( y) y < 0 otherwise strictly convex quadratic f(x) = (1/2)x T Qx with Q S n ++ f (y) = sup(y T x (1/2)x T Qx) x = 1 2 yt Q 1 y Convex functions 3 22

61 Quasiconvex functions f : R n R is quasiconvex if domf is convex and the sublevel sets S α = {x domf f(x) α} are convex for all α β α a b c f is quasiconcave if f is quasiconvex f is quasilinear if it is quasiconvex and quasiconcave Convex functions 3 23

62 x is quasiconvex on R Examples ceil(x) = inf{z Z z x} is quasilinear logx is quasilinear on R ++ f(x 1,x 2 ) = x 1 x 2 is quasiconcave on R 2 ++ linear-fractional function is quasilinear distance ratio f(x) = at x+b c T x+d, domf = {x ct x+d > 0} f(x) = x a 2 x b 2, domf = {x x a 2 x b 2 } is quasiconvex Convex functions 3 24

63 internal rate of return cash flow x = (x 0,...,x n ); x i is payment in period i (to us if x i > 0) we assume x 0 < 0 and x 0 +x 1 + +x n > 0 present value of cash flow x, for interest rate r: PV(x,r) = n (1+r) i x i internal rate of return is smallest interest rate for which PV(x,r) = 0: IRR(x) = inf{r 0 PV(x,r) = 0} IRR is quasiconcave: superlevel set is intersection of open halfspaces n IRR(x) R (1+r) i x i > 0 for 0 r < R i=0 i=0 Convex functions 3 25

64 Properties modified Jensen inequality: for quasiconvex f 0 θ 1 = f(θx+(1 θ)y) max{f(x),f(y)} first-order condition: differentiable f with cvx domain is quasiconvex iff f(y) f(x) = f(x) T (y x) 0 f(x) x sums of quasiconvex functions are not necessarily quasiconvex Convex functions 3 26

65 Log-concave and log-convex functions a positive function f is log-concave if logf is concave: f(θx+(1 θ)y) f(x) θ f(y) 1 θ for 0 θ 1 f is log-convex if logf is convex powers: x a on R ++ is log-convex for a 0, log-concave for a 0 many common probability densities are log-concave, e.g., normal: f(x) = 1 (2π)n detσ e 1 2 (x x)t Σ 1 (x x) cumulative Gaussian distribution function Φ is log-concave Φ(x) = 1 2π x e u2 /2 du Convex functions 3 27

66 Properties of log-concave functions twice differentiable f with convex domain is log-concave if and only if f(x) 2 f(x) f(x) f(x) T for all x domf product of log-concave functions is log-concave sum of log-concave functions is not always log-concave integration: if f : R n R m R is log-concave, then g(x) = f(x,y) dy is log-concave (not easy to show) Convex functions 3 28

67 consequences of integration property convolution f g of log-concave functions f, g is log-concave (f g)(x) = f(x y)g(y)dy if C R n convex and y is a random variable with log-concave pdf then is log-concave f(x) = prob(x+y C) proof: write f(x) as integral of product of log-concave functions p is pdf of y f(x) = g(x+y)p(y)dy, g(u) = { 1 u C 0 u C, Convex functions 3 29

68 example: yield function Y(x) = prob(x+w S) x R n : nominal parameter values for product w R n : random variations of parameters in manufactured product S: set of acceptable values if S is convex and w has a log-concave pdf, then Y is log-concave yield regions {x Y(x) α} are convex Convex functions 3 30

69 Convexity with respect to generalized inequalities f : R n R m is K-convex if domf is convex and f(θx+(1 θ)y) K θf(x)+(1 θ)f(y) for x, y domf, 0 θ 1 example f : S m S m, f(x) = X 2 is S m +-convex proof: for fixed z R m, z T X 2 z = Xz 2 2 is convex in X, i.e., z T (θx +(1 θ)y) 2 z θz T X 2 z +(1 θ)z T Y 2 z for X,Y S m, 0 θ 1 therefore (θx +(1 θ)y) 2 θx 2 +(1 θ)y 2 Convex functions 3 31

70 Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization geometric programming generalized inequality constraints semidefinite programming vector optimization 4 1

71 Optimization problem in standard form minimize f 0 (x) subject to f i (x) 0, i = 1,...,m h i (x) = 0, i = 1,...,p x R n is the optimization variable f 0 : R n R is the objective or cost function f i : R n R, i = 1,...,m, are the inequality constraint functions h i : R n R are the equality constraint functions optimal value: p = inf{f 0 (x) f i (x) 0, i = 1,...,m, h i (x) = 0, i = 1,...,p} p = if problem is infeasible (no x satisfies the constraints) p = if problem is unbounded below Convex optimization problems 4 2

72 Optimal and locally optimal points x is feasible if x domf 0 and it satisfies the constraints a feasible x is optimal if f 0 (x) = p ; X opt is the set of optimal points x is locally optimal if there is an R > 0 such that x is optimal for minimize (over z) f 0 (z) subject to f i (z) 0, i = 1,...,m, h i (z) = 0, i = 1,...,p z x 2 R examples (with n = 1, m = p = 0) f 0 (x) = 1/x, domf 0 = R ++ : p = 0, no optimal point f 0 (x) = logx, domf 0 = R ++ : p = f 0 (x) = xlogx, domf 0 = R ++ : p = 1/e, x = 1/e is optimal f 0 (x) = x 3 3x, p =, local optimum at x = 1 Convex optimization problems 4 3

73 Implicit constraints the standard form optimization problem has an implicit constraint x D = m i=0 domf i p i=1 domh i, we call D the domain of the problem the constraints f i (x) 0, h i (x) = 0 are the explicit constraints a problem is unconstrained if it has no explicit constraints (m = p = 0) example: minimize f 0 (x) = k i=1 log(b i a T i x) is an unconstrained problem with implicit constraints a T i x < b i Convex optimization problems 4 4

74 Feasibility problem find x subject to f i (x) 0, i = 1,...,m h i (x) = 0, i = 1,...,p can be considered a special case of the general problem with f 0 (x) = 0: minimize 0 subject to f i (x) 0, i = 1,...,m h i (x) = 0, i = 1,...,p p = 0 if constraints are feasible; any feasible x is optimal p = if constraints are infeasible Convex optimization problems 4 5

75 Convex optimization problem standard form convex optimization problem minimize f 0 (x) subject to f i (x) 0, i = 1,...,m a T i x = b i, i = 1,...,p f 0, f 1,..., f m are convex; equality constraints are affine problem is quasiconvex if f 0 is quasiconvex (and f 1,..., f m convex) often written as minimize f 0 (x) subject to f i (x) 0, i = 1,...,m Ax = b important property: feasible set of a convex optimization problem is convex Convex optimization problems 4 6

76 example minimize f 0 (x) = x 2 1+x 2 2 subject to f 1 (x) = x 1 /(1+x 2 2) 0 h 1 (x) = (x 1 +x 2 ) 2 = 0 f 0 is convex; feasible set {(x 1,x 2 ) x 1 = x 2 0} is convex not a convex problem (according to our definition): f 1 is not convex, h 1 is not affine equivalent (but not identical) to the convex problem minimize x 2 1+x 2 2 subject to x 1 0 x 1 +x 2 = 0 Convex optimization problems 4 7

77 Local and global optima any locally optimal point of a convex problem is (globally) optimal proof: suppose x is locally optimal and y is optimal with f 0 (y) < f 0 (x) x locally optimal means there is an R > 0 such that z feasible, z x 2 R = f 0 (z) f 0 (x) consider z = θy +(1 θ)x with θ = R/(2 y x 2 ) y x 2 > R, so 0 < θ < 1/2 z is a convex combination of two feasible points, hence also feasible z x 2 = R/2 and f 0 (z) θf 0 (x)+(1 θ)f 0 (y) < f 0 (x) which contradicts our assumption that x is locally optimal Convex optimization problems 4 8

78 Optimality criterion for differentiable f 0 x is optimal if and only if it is feasible and f 0 (x) T (y x) 0 for all feasible y X f 0 (x) x if nonzero, f 0 (x) defines a supporting hyperplane to feasible set X at x Convex optimization problems 4 9

79 unconstrained problem: x is optimal if and only if x domf 0, f 0 (x) = 0 equality constrained problem minimize f 0 (x) subject to Ax = b x is optimal if and only if there exists a ν such that x domf 0, Ax = b, f 0 (x)+a T ν = 0 minimization over nonnegative orthant x is optimal if and only if minimize f 0 (x) subject to x 0 x domf 0, x 0, { f0 (x) i 0 x i = 0 f 0 (x) i = 0 x i > 0 Convex optimization problems 4 10

80 Equivalent convex problems two problems are (informally) equivalent if the solution of one is readily obtained from the solution of the other, and vice-versa some common transformations that preserve convexity: eliminating equality constraints is equivalent to minimize f 0 (x) subject to f i (x) 0, i = 1,...,m Ax = b minimize (over z) f 0 (Fz +x 0 ) subject to f i (Fz +x 0 ) 0, i = 1,...,m where F and x 0 are such that Ax = b x = Fz +x 0 for some z Convex optimization problems 4 11

81 introducing equality constraints is equivalent to minimize f 0 (A 0 x+b 0 ) subject to f i (A i x+b i ) 0, i = 1,...,m minimize (over x, y i ) f 0 (y 0 ) subject to f i (y i ) 0, i = 1,...,m y i = A i x+b i, i = 0,1,...,m introducing slack variables for linear inequalities is equivalent to minimize f 0 (x) subject to a T i x b i, i = 1,...,m minimize (over x, s) f 0 (x) subject to a T i x+s i = b i, i = 1,...,m s i 0, i = 1,...m Convex optimization problems 4 12

82 epigraph form: standard form convex problem is equivalent to minimize (over x, t) t subject to f 0 (x) t 0 f i (x) 0, i = 1,...,m Ax = b minimizing over some variables is equivalent to where f 0 (x 1 ) = inf x2 f 0 (x 1,x 2 ) minimize f 0 (x 1,x 2 ) subject to f i (x 1 ) 0, i = 1,...,m minimize f0 (x 1 ) subject to f i (x 1 ) 0, i = 1,...,m Convex optimization problems 4 13

83 Quasiconvex optimization minimize f 0 (x) subject to f i (x) 0, i = 1,...,m Ax = b with f 0 : R n R quasiconvex, f 1,..., f m convex can have locally optimal points that are not (globally) optimal (x,f 0 (x)) Convex optimization problems 4 14

84 convex representation of sublevel sets of f 0 if f 0 is quasiconvex, there exists a family of functions φ t such that: φ t (x) is convex in x for fixed t t-sublevel set of f 0 is 0-sublevel set of φ t, i.e., example f 0 (x) t φ t (x) 0 f 0 (x) = p(x) q(x) with p convex, q concave, and p(x) 0, q(x) > 0 on domf 0 can take φ t (x) = p(x) tq(x): for t 0, φ t convex in x p(x)/q(x) t if and only if φ t (x) 0 Convex optimization problems 4 15

85 quasiconvex optimization via convex feasibility problems φ t (x) 0, f i (x) 0, i = 1,...,m, Ax = b (1) for fixed t, a convex feasibility problem in x if feasible, we can conclude that t p ; if infeasible, t p Bisection method for quasiconvex optimization given l p, u p, tolerance ǫ > 0. repeat 1. t := (l + u)/2. 2. Solve the convex feasibility problem (1). 3. if (1) is feasible, u := t; else l := t. until u l ǫ. requires exactly log 2 ((u l)/ǫ) iterations (where u, l are initial values) Convex optimization problems 4 16

86 Linear program (LP) minimize c T x+d subject to Gx h Ax = b convex problem with affine objective and constraint functions feasible set is a polyhedron c P x Convex optimization problems 4 17

87 Examples diet problem: choose quantities x 1,..., x n of n foods one unit of food j costs c j, contains amount a ij of nutrient i healthy diet requires nutrient i in quantity at least b i to find cheapest healthy diet, piecewise-linear minimization equivalent to an LP minimize c T x subject to Ax b, x 0 minimize max i=1,...,m (a T i x+b i) minimize t subject to a T i x+b i t, i = 1,...,m Convex optimization problems 4 18

88 Chebyshev center of a polyhedron Chebyshev center of P = {x a T i x b i, i = 1,...,m} is center of largest inscribed ball x cheb B = {x c +u u 2 r} a T i x b i for all x B if and only if sup{a T i (x c +u) u 2 r} = a T i x c +r a i 2 b i hence, x c, r can be determined by solving the LP maximize r subject to a T i x c+r a i 2 b i, i = 1,...,m Convex optimization problems 4 19

89 Linear-fractional program linear-fractional program minimize f 0 (x) subject to Gx h Ax = b f 0 (x) = ct x+d e T x+f, domf 0(x) = {x e T x+f > 0} a quasiconvex optimization problem; can be solved by bisection also equivalent to the LP (variables y, z) minimize c T y +dz subject to Gy hz Ay = bz e T y +fz = 1 z 0 Convex optimization problems 4 20

90 generalized linear-fractional program f 0 (x) = max i=1,...,r c T i x+d i e T i x+f, domf 0 (x) = {x e T i x+f i > 0, i = 1,...,r} i a quasiconvex optimization problem; can be solved by bisection example: Von Neumann model of a growing economy maximize (over x, x + ) min i=1,...,n x + i /x i subject to x + 0, Bx + Ax x,x + R n : activity levels of n sectors, in current and next period (Ax) i, (Bx + ) i : produced, resp. consumed, amounts of good i x + i /x i: growth rate of sector i allocate activity to maximize growth rate of slowest growing sector Convex optimization problems 4 21

91 Quadratic program (QP) minimize (1/2)x T Px+q T x+r subject to Gx h Ax = b P S n +, so objective is convex quadratic minimize a convex quadratic function over a polyhedron P x f 0 (x ) Convex optimization problems 4 22

92 Examples least-squares minimize Ax b 2 2 analytical solution x = A b (A is pseudo-inverse) can add linear constraints, e.g., l x u linear program with random cost minimize c T x+γx T Σx = Ec T x+γvar(c T x) subject to Gx h, Ax = b c is random vector with mean c and covariance Σ hence, c T x is random variable with mean c T x and variance x T Σx γ > 0 is risk aversion parameter; controls the trade-off between expected cost and variance (risk) Convex optimization problems 4 23

93 Quadratically constrained quadratic program (QCQP) minimize (1/2)x T P 0 x+q T 0x+r 0 subject to (1/2)x T P i x+q T i x+r i 0, i = 1,...,m Ax = b P i S n +; objective and constraints are convex quadratic if P 1,...,P m S n ++, feasible region is intersection of m ellipsoids and an affine set Convex optimization problems 4 24

94 Second-order cone programming minimize f T x subject to A i x+b i 2 c T i x+d i, i = 1,...,m Fx = g (A i R n i n, F R p n ) inequalities are called second-order cone (SOC) constraints: (A i x+b i,c T i x+d i ) second-order cone in R n i+1 for n i = 0, reduces to an LP; if c i = 0, reduces to a QCQP more general than QCQP and LP Convex optimization problems 4 25

95 Robust linear programming the parameters in optimization problems are often uncertain, e.g., in an LP there can be uncertainty in c, a i, b i minimize c T x subject to a T i x b i, i = 1,...,m, two common approaches to handling uncertainty (in a i, for simplicity) deterministic model: constraints must hold for all a i E i minimize c T x subject to a T i x b i for all a i E i, i = 1,...,m, stochastic model: a i is random variable; constraints must hold with probability η minimize c T x subject to prob(a T i x b i) η, i = 1,...,m Convex optimization problems 4 26

96 deterministic approach via SOCP choose an ellipsoid as E i : E i = {ā i +P i u u 2 1} (ā i R n, P i R n n ) center is ā i, semi-axes determined by singular values/vectors of P i robust LP minimize c T x subject to a T i x b i a i E i, i = 1,...,m is equivalent to the SOCP minimize c T x subject to ā T i x+ PT i x 2 b i, i = 1,...,m (follows from sup u 2 1(ā i +P i u) T x = ā T i x+ PT i x 2) Convex optimization problems 4 27

97 stochastic approach via SOCP assume a i is Gaussian with mean ā i, covariance Σ i (a i N(ā i,σ i )) a T i x is Gaussian r.v. with mean āt i x, variance xt Σ i x; hence ( ) prob(a T b i ā T i i x b i ) = Φ x Σ 1/2 i x 2 where Φ(x) = (1/ 2π) x e t2 /2 dt is CDF of N(0,1) robust LP minimize c T x subject to prob(a T i x b i) η, i = 1,...,m, with η 1/2, is equivalent to the SOCP minimize c T x subject to ā T i x+φ 1 (η) Σ 1/2 i x 2 b i, i = 1,...,m Convex optimization problems 4 28

98 Geometric programming monomial function f(x) = cx a 1 1 xa 2 2 xa n n, domf = R n ++ with c > 0; exponent α i can be any real number posynomial function: sum of monomials f(x) = K c k x a 1k k=1 geometric program (GP) 1 xa 2k with f i posynomial, h i monomial 2 x a nk n, domf = R n ++ minimize f 0 (x) subject to f i (x) 1, i = 1,...,m h i (x) = 1, i = 1,...,p Convex optimization problems 4 29

99 Geometric program in convex form change variables to y i = logx i, and take logarithm of cost, constraints monomial f(x) = cx a 1 1 xa n n transforms to logf(e y 1,...,e y n ) = a T y +b (b = logc) posynomial f(x) = K k=1 c kx a 1k 1 xa 2k 2 x a nk n ( K logf(e y 1,...,e y n ) = log k=1 e at k y+b k transforms to ) (b k = logc k ) geometric program transforms to convex problem ( K ) minimize log k=1 exp(at 0k y +b 0k) ( K ) subject to log k=1 exp(at ik y +b ik) 0, i = 1,...,m Gy +d = 0 Convex optimization problems 4 30

100 Design of cantilever beam segment 4 segment 3 segment 2 segment 1 F N segments with unit lengths, rectangular cross-sections of size w i h i given vertical force F applied at the right end design problem minimize total weight subject to upper & lower bounds on w i, h i upper bound & lower bounds on aspect ratios h i /w i upper bound on stress in each segment upper bound on vertical deflection at the end of the beam variables: w i, h i for i = 1,...,N Convex optimization problems 4 31

101 objective and constraint functions total weight w 1 h 1 + +w N h N is posynomial aspect ratio h i /w i and inverse aspect ratio w i /h i are monomials maximum stress in segment i is given by 6iF/(w i h 2 i ), a monomial the vertical deflection y i and slope v i of central axis at the right end of segment i are defined recursively as F v i = 12(i 1/2) Ew i h 3 +v i+1 i F y i = 6(i 1/3) Ew i h 3 +v i+1 +y i+1 i for i = N,N 1,...,1, with v N+1 = y N+1 = 0 (E is Young s modulus) v i and y i are posynomial functions of w, h Convex optimization problems 4 32

102 formulation as a GP note minimize w 1 h 1 + +w N h N subject to w 1 maxw i 1, w min w 1 i 1, i = 1,...,N h 1 maxh i 1, h min h 1 i 1, i = 1,...,N S 1 maxw 1 i h i 1, S min w i h 1 i 1, i = 1,...,N 6iFσ 1 maxw 1 i h 2 i 1, i = 1,...,N y 1 maxy 1 1 we write w min w i w max and h min h i h max w min /w i 1, w i /w max 1, h min /h i 1, h i /h max 1 we write S min h i /w i S max as S min w i /h i 1, h i /(w i S max ) 1 Convex optimization problems 4 33

103 Minimizing spectral radius of nonnegative matrix Perron-Frobenius eigenvalue λ pf (A) exists for (elementwise) positive A R n n a real, positive eigenvalue of A, equal to spectral radius max i λ i (A) determines asymptotic growth (decay) rate of A k : A k λ k pf as k alternative characterization: λ pf (A) = inf{λ Av λv for some v 0} minimizing spectral radius of matrix of posynomials minimize λ pf (A(x)), where the elements A(x) ij are posynomials of x equivalent geometric program: variables λ, v, x minimize λ subject to n j=1 A(x) ijv j /(λv i ) 1, i = 1,...,n Convex optimization problems 4 34

104 Generalized inequality constraints convex problem with generalized inequality constraints minimize f 0 (x) subject to f i (x) Ki 0, i = 1,...,m Ax = b f 0 : R n R convex; f i : R n R k i K i -convex w.r.t. proper cone K i same properties as standard convex problem (convex feasible set, local optimum is global, etc.) conic form problem: special case with affine objective and constraints minimize c T x subject to Fx+g K 0 Ax = b extends linear programming (K = R m +) to nonpolyhedral cones Convex optimization problems 4 35

105 Semidefinite program (SDP) with F i, G S k minimize c T x subject to x 1 F 1 +x 2 F 2 + +x n F n +G 0 Ax = b inequality constraint is called linear matrix inequality (LMI) includes problems with multiple LMI constraints: for example, x 1ˆF1 + +x nˆfn +Ĝ 0, x 1 F 1 + +x n Fn + G 0 is equivalent to single LMI [ ˆF1 0 x 1 0 F1 ] [ ˆF2 0 +x 2 0 F2 ] + +x n [ ˆFn 0 0 Fn ] [ Ĝ G ] 0 Convex optimization problems 4 36

106 LP and SOCP as SDP LP and equivalent SDP LP: minimize c T x subject to Ax b SDP: minimize c T x subject to diag(ax b) 0 (note different interpretation of generalized inequality ) SOCP and equivalent SDP SOCP: minimize f T x subject to A i x+b i 2 c T i x+d i, i = 1,...,m SDP: minimize [ f T x (c T subject to i x+d i )I A i x+b i (A i x+b i ) T c T i x+d i ] 0, i = 1,...,m Convex optimization problems 4 37

107 Eigenvalue minimization minimize λ max (A(x)) where A(x) = A 0 +x 1 A 1 + +x n A n (with given A i S k ) equivalent SDP variables x R n, t R follows from minimize t subject to A(x) ti λ max (A) t A ti Convex optimization problems 4 38

108 Matrix norm minimization minimize A(x) 2 = ( λ max (A(x) T A(x)) ) 1/2 where A(x) = A 0 +x 1 A 1 + +x n A n (with given A i R p q ) equivalent SDP minimize t[ ] ti A(x) subject to A(x) T 0 ti variables x R n, t R constraint follows from A 2 t A T A t 2 I, t 0 [ ] ti A A T 0 ti Convex optimization problems 4 39

109 Vector optimization general vector optimization problem minimize (w.r.t. K) f 0 (x) subject to f i (x) 0, i = 1,...,m h i (x) 0, i = 1,...,p vector objective f 0 : R n R q, minimized w.r.t. proper cone K R q convex vector optimization problem minimize (w.r.t. K) f 0 (x) subject to f i (x) 0, i = 1,...,m Ax = b with f 0 K-convex, f 1,..., f m convex Convex optimization problems 4 40

110 Optimal and Pareto optimal points set of achievable objective values O = {f 0 (x) x feasible} feasible x is optimal if f 0 (x) is the minimum value of O feasible x is Pareto optimal if f 0 (x) is a minimal value of O O O f 0 (x po ) f 0 (x ) x is optimal x po is Pareto optimal Convex optimization problems 4 41

111 Multicriterion optimization vector optimization problem with K = R q + f 0 (x) = (F 1 (x),...,f q (x)) q different objectives F i ; roughly speaking we want all F i s to be small feasible x is optimal if y feasible = f 0 (x ) f 0 (y) if there exists an optimal point, the objectives are noncompeting feasible x po is Pareto optimal if y feasible, f 0 (y) f 0 (x po ) = f 0 (x po ) = f 0 (y) if there are multiple Pareto optimal values, there is a trade-off between the objectives Convex optimization problems 4 42

112 Regularized least-squares minimize (w.r.t. R 2 +) ( Ax b 2 2, x 2 2) F2(x) = x O F 1 (x) = Ax b 2 2 example for A R ; heavy line is formed by Pareto optimal points Convex optimization problems 4 43

113 Risk return trade-off in portfolio optimization minimize (w.r.t. R 2 +) ( p T x,x T Σx) subject to 1 T x = 1, x 0 x R n is investment portfolio; x i is fraction invested in asset i p R n is vector of relative asset price changes; modeled as a random variable with mean p, covariance Σ p T x = Er is expected return; x T Σx = varr is return variance example mean return 15% 10% 5% 0% 0% 10% 20% standard deviation of return allocation x x(4) x(3) x(2) x(1) 0% 10% 20% standard deviation of return Convex optimization problems 4 44

114 Scalarization to find Pareto optimal points: choose λ K 0 and solve scalar problem if x is optimal for scalar problem, then it is Pareto-optimal for vector optimization problem minimize λ T f 0 (x) subject to f i (x) 0, i = 1,...,m h i (x) = 0, i = 1,...,p f 0 (x 1 ) O f λ 0 (x 3 ) 1 f 0 (x 2 ) λ 2 for convex vector optimization problems, can find (almost) all Pareto optimal points by varying λ K 0 Convex optimization problems 4 45

115 Scalarization for multicriterion problems to find Pareto optimal points, minimize positive weighted sum λ T f 0 (x) = λ 1 F 1 (x)+ +λ q F q (x) examples regularized least-squares problem of page 4 43 take λ = (1,γ) with γ > 0 minimize Ax b 2 2+γ x 2 2 for fixed γ, a LS problem x γ = Ax b 2 2 Convex optimization problems 4 46

116 risk-return trade-off of page 4 44 minimize p T x+γx T Σx subject to 1 T x = 1, x 0 for fixed γ > 0, a quadratic program Convex optimization problems 4 47

117 Convex Optimization Boyd & Vandenberghe 5. Duality Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized inequalities 5 1

118 Lagrangian standard form problem (not necessarily convex) minimize f 0 (x) subject to f i (x) 0, i = 1,...,m h i (x) = 0, i = 1,...,p variable x R n, domain D, optimal value p Lagrangian: L : R n R m R p R, with doml = D R m R p, L(x,λ,ν) = f 0 (x)+ m λ i f i (x)+ i=1 p ν i h i (x) i=1 weighted sum of objective and constraint functions λ i is Lagrange multiplier associated with f i (x) 0 ν i is Lagrange multiplier associated with h i (x) = 0 Duality 5 2

119 Lagrange dual function Lagrange dual function: g : R m R p R, g(λ,ν) = inf L(x,λ,ν) x D ( = inf x D f 0 (x)+ g is concave, can be for some λ, ν m λ i f i (x)+ i=1 lower bound property: if λ 0, then g(λ,ν) p proof: if x is feasible and λ 0, then ) p ν i h i (x) i=1 f 0 ( x) L( x,λ,ν) inf L(x,λ,ν) = g(λ,ν) x D minimizing over all feasible x gives p g(λ,ν) Duality 5 3

120 Least-norm solution of linear equations dual function minimize x T x subject to Ax = b Lagrangian is L(x,ν) = x T x+ν T (Ax b) to minimize L over x, set gradient equal to zero: x L(x,ν) = 2x+A T ν = 0 = x = (1/2)A T ν plug in in L to obtain g: a concave function of ν g(ν) = L(( 1/2)A T ν,ν) = 1 4 νt AA T ν b T ν lower bound property: p (1/4)ν T AA T ν b T ν for all ν Duality 5 4

121 Standard form LP minimize c T x subject to Ax = b, x 0 dual function Lagrangian is L is affine in x, hence L(x,λ,ν) = c T x+ν T (Ax b) λ T x g(λ,ν) = inf x L(x,λ,ν) = = b T ν +(c+a T ν λ) T x { b T ν A T ν λ+c = 0 otherwise g is linear on affine domain {(λ,ν) A T ν λ+c = 0}, hence concave lower bound property: p b T ν if A T ν +c 0 Duality 5 5

122 Equality constrained norm minimization dual function minimize x subject to Ax = b g(ν) = inf x ( x νt Ax+b T ν) = where v = sup u 1 u T v is dual norm of { b T ν A T ν 1 otherwise proof: follows from inf x ( x y T x) = 0 if y 1, otherwise if y 1, then x y T x 0 for all x, with equality if x = 0 if y > 1, choose x = tu where u 1, u T y = y > 1: x y T x = t( u y ) as t lower bound property: p b T ν if A T ν 1 Duality 5 6

123 Two-way partitioning minimize x T Wx subject to x 2 i = 1, i = 1,...,n a nonconvex problem; feasible set contains 2 n discrete points interpretation: partition {1,...,n} in two sets; W ij is cost of assigning i, j to the same set; W ij is cost of assigning to different sets dual function g(ν) = inf x (xt Wx+ i ν i (x 2 i 1)) = inf x xt (W +diag(ν))x 1 T ν { 1 = T ν W +diag(ν) 0 otherwise lower bound property: p 1 T ν if W +diag(ν) 0 example: ν = λ min (W)1 gives bound p nλ min (W) Duality 5 7

124 Lagrange dual and conjugate function dual function g(λ,ν) = minimize f 0 (x) subject to Ax b, Cx = d ( inf f0 (x)+(a T λ+c T ν) T x b T λ d T ν ) x domf 0 = f 0( A T λ C T ν) b T λ d T ν recall definition of conjugate f (y) = sup x domf (y T x f(x)) simplifies derivation of dual if conjugate of f 0 is known example: entropy maximization f 0 (x) = n x i logx i, f0(y) = i=1 n i=1 e y i 1 Duality 5 8

125 The dual problem Lagrange dual problem maximize g(λ, ν) subject to λ 0 finds best lower bound on p, obtained from Lagrange dual function a convex optimization problem; optimal value denoted d λ, ν are dual feasible if λ 0, (λ,ν) domg often simplified by making implicit constraint (λ, ν) dom g explicit example: standard form LP and its dual (page 5 5) minimize c T x subject to Ax = b x 0 maximize b T ν subject to A T ν +c 0 Duality 5 9

126 weak duality: d p Weak and strong duality always holds (for convex and nonconvex problems) can be used to find nontrivial lower bounds for difficult problems for example, solving the SDP maximize 1 T ν subject to W +diag(ν) 0 gives a lower bound for the two-way partitioning problem on page 5 7 strong duality: d = p does not hold in general (usually) holds for convex problems conditions that guarantee strong duality in convex problems are called constraint qualifications Duality 5 10

127 Slater s constraint qualification strong duality holds for a convex problem if it is strictly feasible, i.e., minimize f 0 (x) subject to f i (x) 0, i = 1,...,m Ax = b x intd : f i (x) < 0, i = 1,...,m, Ax = b also guarantees that the dual optimum is attained (if p > ) can be sharpened: e.g., can replace intd with relintd (interior relative to affine hull); linear inequalities do not need to hold with strict inequality,... there exist many other types of constraint qualifications Duality 5 11

128 Inequality form LP primal problem minimize c T x subject to Ax b dual function g(λ) = inf x dual problem ( (c+a T λ) T x b T λ ) { b = T λ A T λ+c = 0 otherwise maximize b T λ subject to A T λ+c = 0, λ 0 from Slater s condition: p = d if A x b for some x in fact, p = d except when primal and dual are infeasible Duality 5 12

129 primal problem (assume P S n ++) Quadratic program minimize x T Px subject to Ax b dual function g(λ) = inf x dual problem ( x T Px+λ T (Ax b) ) = 1 4 λt AP 1 A T λ b T λ maximize (1/4)λ T AP 1 A T λ b T λ subject to λ 0 from Slater s condition: p = d if A x b for some x in fact, p = d always Duality 5 13

130 A nonconvex problem with strong duality A 0, hence nonconvex minimize x T Ax+2b T x subject to x T x 1 dual function: g(λ) = inf x (x T (A+λI)x+2b T x λ) unbounded below if A+λI 0 or if A+λI 0 and b R(A+λI) minimized by x = (A+λI) b otherwise: g(λ) = b T (A+λI) b λ dual problem and equivalent SDP: maximize b T (A+λI) b λ subject to A+λI 0 b R(A+λI) maximize t [ λ A+λI b subject to b T t strong duality although primal problem is not convex (not easy to show) ] 0 Duality 5 14

131 Geometric interpretation for simplicity, consider problem with one constraint f 1 (x) 0 interpretation of dual function: g(λ) = inf (u,t) G (t+λu), where G = {(f 1(x),f 0 (x)) x D} λu + t = g(λ) t p G g(λ) λu+t = g(λ) is (non-vertical) supporting hyperplane to G hyperplane intersects t-axis at t = g(λ) u t p d G u Duality 5 15