Convex functions I basic properties and I operations that preserve convexity I quasiconvex functions I log-concave and log-convex functions IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 1 Definition f : R n! R is convex if dom f is a convex set and f ( x +(1 )y) apple f (x)+(1 )f (y) for all x, y 2 dom f,0apple apple1 ments (x, f(x)) (y, f(y)) I f is concave if f is convex I f is strictly convex if dom f is convex and f ( x +(1 )y) < f (x)+(1 )f (y) for x, y 2 dom f, x 6= y, 0< < 1 IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 2
Examples on R convex: I a ne: ax + b on R, foranya, b 2 R I exponential: e ax,foranya 2 R I powers: x on R ++,for 1or apple 0 I powers of absolute value: x p on R, forp 1 I negative entropy: x log x on R ++ concave: I a ne: ax + b on R, foranya, b 2 R I powers: x on R ++,for0apple apple 1 I logarithm: log x on R ++ IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 3 Examples on R n and R m n a ne functions are convex and concave; all norms are convex on R n I a ne function f (x) =a T x + b I norms: kxk p =( P n i=1 x i p ) 1/p for p 1; kxk 1 =max k x k on R m n (m n matrices) I a ne function f (X )=tr(a T X )+b = mx nx A ij X ij + b i=1 j=1 I spectral (maximum singular value) norm f (X )=kx k 2 = max (X )=( max (X T X )) 1/2 IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 4
Restriction of a convex function to a line f : R n! R is convex if and only if the function g : R! R, g(t) =f (x + tv), dom g = {t x + tv 2 dom f } is convex (in t) foranyx 2 dom f, v 2 R n So, can check convexity of f by checking convexity of functions of one variable example. f : S n! R with f (X ) = log det X, dom f = S n ++ g(t) = log det(x + tv ) = log det X + log det(i + tx 1/2 VX 1/2 ) nx = log det X + log(1 + t i ) i=1 where i are the eigenvalues of X 1/2 VX 1/2 g is concave in t (for any choice of X 0, V ); hence f is concave IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 5 First-order condition f is di erentiable if dom f is open and the gradient @f (x) @f (x) @f (x) rf (x) =,,..., @x 1 @x 2 @x n exists at each x 2 dom f 1st-order condition: di erentiable f with convex domain is convex i f (y) f (x)+rf (x) T (y x) for all x, y 2 dom f (gradient inequality) s f(y) f(x) + f(x) T (y x) (x, f(x)) first-order approximation of f is global underestimator IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 6
Second-order conditions f is twice di erentiable if dom f is open and the Hessian r 2 f (x) 2 S n, r 2 f (x) ij = @2 f (x) @x i @x j, i, j =1,...,n, exists at each x 2 dom f 2nd-order conditions: for twice di erentiable f with convex domain I f is convex if and only if r 2 f (x) 0 for all x 2 dom f I if r 2 f (x) 0forallx 2 dom f,thenf is strictly convex IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 7 Examples quadratic function: f (x) =(1/2)x T Px + q T x + r (with P 2 S n ) rf (x) =Px + q, r 2 f (x) =P convex i P 0 least-squares objective: f (x) =kax bk 2 2 rf (x) =2A T (Ax b), r 2 f (x) =2A T A convex (for any A) PSfrag replacements quadratic-over-linear: f (x, y) =x 2 /y r 2 f (x, y) = 2 apple y 3 convex for y > 0 y x apple y x T 0 f(x, y) 2 1 0 2 1 y 0 2 x 0 2 IOE Convex 611: Nonlinear functions Programming, Fall 2017 3. Convex functions Page 3 8 3 9
Examples log-sum-exp: f (x) = log P n k=1 exp x k is convex r 2 f (x) = 1 1 T z diag(z) 1 (1 T z) 2 zzt (z k =expx k ) to show r 2 f (x) 0, we must verify that v T r 2 f (x)v 0forallv: v T r 2 f (x)v = (P k z kv 2 k )(P k z k) ( P k v kz k ) 2 ( P k z k) 2 0 since ( P k v kz k ) 2 apple ( P k z kv 2 k )(P k z k) (from Cauchy-Schwarz inequality) geometric mean: f (x) =( Q n k=1 x k) 1/n on R n ++ is concave (similar proof as for log-sum-exp) IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 9 Sublevel set and Epigraph -sublevel set of f : R n! R: C = {x 2 dom f f (x) apple } sublevel sets of convex functions are convex (converse is false) epigraph of f : R n! R: epi f = {(x, t) 2 R n+1 x 2 dom f, f (x) apple t} epi f s f f is convex if and only if epi f is a convex set IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 10
Jensen s inequality basic inequality: if f is convex, then for 0 apple apple 1, f ( x +(1 )y) apple f (x)+(1 )f (y) extension: if f is convex, then f (E z) apple E f (z) for any random variable z s.t. P(z 2 dom f )=1 basic inequality is special case with discrete distribution prob(z = x) =, prob(z = y) =1 IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 11 Operations that preserve convexity practical methods for establishing convexity of a function: first, show it has convex domain; then, 1. verify definition (often simplified by restricting to a line) 2. for twice di erentiable functions, show r 2 f (x) 0 8x 2 dom f 3. show that f is obtained from simple convex functions by operations that preserve convexity I nonnegative weighted sum I composition with a ne function I pointwise maximum and supremum I composition I minimization I perspective Proofs that these operations are convexity preserving are often based on analysis of epigraphs and their convexity-preserving transformations IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 12
Positive weighted sum & composition with a ne function nonnegative multiple: f is convex if f is convex, 0 sum: f 1 + f 2 convex if f 1, f 2 convex (extends to infinite sums, integrals) composition with a ne function: f (Ax + b) is convex if f is convex I log barrier for linear inequalities f (x) = mx log(b i i=1 a T i x), dom f = {x a T i x < b i, i =1,...,m} I (any) norm of a ne function: f (x) =kax + bk IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 13 Pointwise maximum if f 1,...,f m are convex, then f (x) =max{f 1 (x),...,f m (x)} is convex proof: epi max{f 1 (x),...,f m (x)} = \ m i=1 epi f i I piecewise-linear function: f (x) =max i=1,...,m (a T i x + b i )is convex I sum of r largest components of x 2 R n : f (x) =x [1] + x [2] + + x [r] is convex (x [i] is ith largest component of x) proof: f (x) =max{x i1 + x i2 + + x ir 1 apple i 1 < i 2 < < i r apple n} IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 14
Pointwise supremum if f (x, y) is convex in x for each y 2 A, then is convex g(x) =supf (x, y) y2a I support function of a set C: S C (x) =sup y2c y T x is convex I distance to farthest point in a set C: f (x) =supkx y2c yk I maximum eigenvalue of symmetric matrix: for X 2 S n, max(x )= sup y T Xy kyk 2 =1 IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 15 Extended-value extension extended-value extension of f is f : R n! R [ {1} f (x) =f (x), x 2 dom f, f (x) =1, x 62 dom f often simplifies notation; for example, the condition 0 apple apple 1 =) f ( x +(1 )y) apple f (x)+(1 ) f (y) (as an inequality in R [ {1}), means the same as the two conditions I dom f is convex I for x, y 2 dom f, 0 apple apple 1 =) f ( x +(1 )y) apple f (x)+(1 )f (y) IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 16
Composition with scalar functions composition of g : R n! R and h : R! R: f is convex if f (x) =h(g(x)) g convex, h convex, h nondecreasing g concave, h convex, h nonincreasing I note: monotonicity must hold for extended-value extension h I proof (for n = 1, twice di erentiable g, h defined everywhere) f 00 (x) =h 00 (g(x))g 0 (x) 2 + h 0 (g(x))g 00 (x) I exp g(x) is convex if g is convex I 1/g(x) is convex if g is concave and positive IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 17 Vector composition composition of g : R n! R k and h : R k! R: f (x) =h(g(x)) = h(g 1 (x), g 2 (x),...,g k (x)) f is convex if g i s convex, h convex, h nondecreasing in each argument g i s concave, h convex, h nonincreasing in each argument proof (for n = 1, twice di erentiable g, h defined everywhere) f 00 (x) =g 0 (x) T r 2 h(g(x))g 0 (x)+rh(g(x)) T g 00 (x) I P m i=1 log g i(x) is concave if g i are concave and positive I log P m i=1 exp g i(x) is convex if g i are convex IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 18
Minimization if f (x, y) is convex in (x, y) andc is a convex set, then is convex g(x) = inf f (x, y) y2c I f (x, y) =x T Ax +2x T By + y T Cy with apple A B B T C 0, C 0 minimizing over y gives g(x) =inf y f (x, y) =x T (A BC 1 B T )x g is convex, hence Schur complement A BC 1 B T 0 I distance to a set: dist(x, S) =inf y2s kx convex yk is convex if S is IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 19 Perspective the perspective of a function f : R n! R is the function g : R n R! R: g(x, t) =tf (x/t), dom g = {(x, t) x/t 2 dom f, t > 0} g is convex if f is convex I f (x) =x T x is convex on R n ;henceg(x, t) =x T x/t is convex for t > 0 I f (x) = log x is convex of R ++ ; hence relative entropy g(x, t) =t log t t log x is convex on R 2 ++ I if f is convex, then g(x) =(c T x + d)f (Ax + b)/(c T x + d) is convex on {x c T x + d > 0, (Ax + b)/(c T x + d) 2 dom f } IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 20
Quasiconvex functions f : R n! R is quasiconvex if dom f is convex and the sublevel sets S = {x 2 dom f f (x) apple } are convex for all nts β α a b c I f is quasiconcave if f is quasiconvex I f is quasilinear if it is quasiconvex and quasiconcave IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 21 Examples I p x is quasiconvex on R I ceil(x) =inf{z 2 Z z I log x is quasilinear on R ++ x} is quasilinear I f (x 1, x 2 )=x 1 x 2 is quasiconcave on R 2 ++ I linear-fractional function f (x) = at x + b c T x + d, dom f = {x ct x + d > 0} is quasilinear I distance ratio f (x) = kx ak 2 kx bk 2, dom f = {x kx ak 2 applekx bk 2 } is quasiconvex IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 22
Internal rate of return I cash flow x =(x 0,...,x n ); x i is payment in period i (to us if x i > 0) I we assume x 0 < 0andx 0 + x 1 + + x n > 0 I present value of cash flow x, forinterestrater: PV(x, r) = nx (1 + r) i x i i=0 I internal rate of return is smallest interest rate for which PV(x, r) = 0: IRR(x) =inf{r 0 PV(x, r) =0} IRR is quasiconcave: superlevel set is intersection of halfspaces IRR(x) R () nx (1 + r) i x i 0for0apple r apple R i=0 IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 23 Properties modified Jensen inequality: for quasiconvex f 0 apple apple 1 =) f ( x +(1 )y) apple max{f (x), f (y)} first-order condition: di erentiable f with convex domain is quasiconvex i f (y) apple f (x) =) rf (x) T (y x) apple 0 x f(x) s sums of quasiconvex functions are not necessarily quasiconvex IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions Page 3 24