Convex Functions Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong
Outline of Lecture Definition convex function Examples on R, R n, and R n n Restriction of a convex function to a line First- and second-order conditions Operations that preserve convexity Quasi-convexity, log-convexity, and convexity w.r.t. inequalities generalized (Acknowledgement to Stephen Boyd for material for this lecture.) Daniel P. Palomar 1
Definition of Convex Function A function f : R n R is said to be convex if the domain, dom f, is convex and for any x, y dom f and 0 θ 1, f (θx + (1 θ) y) θf (x) + (1 θ) f (y). f is strictly convex if the inequality is strict for 0 < θ < 1. f is concave if f is convex. Daniel P. Palomar 2
Convex functions: affine: ax + b on R Examples on R powers of absolute value: x p on R, for p 1 (e.g., x ) powers: x p on R ++, for p 1 or p 0 (e.g., x 2 ) exponential: e ax on R negative entropy: x log x on R ++ Concave functions: affine: ax + b on R powers: x p on R ++, for 0 p 1 logarithm: log x on R ++ Daniel P. Palomar 3
Examples on R n Affine functions f (x) = a T x + b are convex and concave on R n. Norms x are convex on R n (e.g., x, x 1, x 2 ). Quadratic functions f (x) = x T P x + 2q T x + r are convex R n if and only if P 0. The geometric mean f (x) = ( n i=1 x i) 1/n is concave on R n ++. The log-sum-exp f (x) = log i ex i used to approximate max x i). i=1,...,n is convex on R n (it can be Quadratic over linear: f (x, y) = x 2 /y is convex on R n R ++. Daniel P. Palomar 4
Affine functions: (prove it!) Examples on R n n f (X) = Tr (AX) + b are convex and concave on R n n. Logarithmic determinant function: (prove it!) f (X) = logdet (X) is concave on S n = {X R n n X 0}. Maximum eigenvalue function: (prove it!) is convex on S n. f (X) = λ max (X) = sup y 0 y T Xy y T y Daniel P. Palomar 5
Epigraph The epigraph of f if the set epi f = { (x, t) R n+1 x dom f, f (x) t }. Relation between convexity in sets and convexity in functions: f is convex epi f is convex Daniel P. Palomar 6
Restriction of a Convex Function to a Line f : R n R is convex if and only if the function g : R R g (t) = f (x + tv), dom g = {t x + tv dom f} is convex for any x dom f, v R n. In words: a function is convex if and only if it is convex when restricted to an arbitrary line. Implication: we can check convexity of f by checking convexity of functions of one variable! Example: concavity of logdet (X) follows from concavity of log (x). Daniel P. Palomar 7
Example: concavity of logdet (X) : g (t) = logdet (X + tv ) = logdet (X) + logdet (I + tx 1/2 V X 1/2) = logdet (X) + n i=1 log (1 + tλ i ) where λ i s are the eigenvalues of X 1/2 V X 1/2. The function g is concave in t for any choice of X 0 and V ; therefore, f is concave. Daniel P. Palomar 8
First and Second Order Condition Gradient (for differentiable f): f (x) = [ f(x) x 1 f(x) x n ] T R n. Hessian (for twice differentiable f): ( 2 2 ) f (x) f (x) = x i x j ij R n n. Taylor series: f (x + δ) = f (x) + f (x) T δ + 1 2 δt 2 f (x) δ + o ( δ 2). Daniel P. Palomar 9
First-order condition: convex if and only if a differentiable f with convex domain is f (y) f (x) + f (x) T (y x) x, y dom f Interpretation: first-order approximation if a global underestimator. Second-order condition: a twice differentiable f with convex domain is convex if and only if 2 f (x) 0 x dom f Daniel P. Palomar 10
Examples Quadratic function: f (x) = (1/2) x T P x + q T x + r (with P S n ) f (x) = P x + q, 2 f (x) = P is convex if P 0. Least-squares objective: f (x) = Ax b 2 2 f (x) = 2A T (Ax b), 2 f (x) = 2A T A is convex. Daniel P. Palomar 11
Quadratic-over-linear: f (x, y) = x 2 /y is convex for y > 0. 2 f (x, y) = 2 y 3 [ y x ] [ y x ] 0 Daniel P. Palomar 12
Operations that Preserve Convexity How do we establish the convexity of a given function? 1. Applying the definition. 2. With first- or second-order conditions. 3. By restricting to a line. 4. Showing that the functions can be obtained from simple functions by operations that preserve convexity: nonnegative weighted sum composition with affine function (and other compositions) pointwise maximum and supremum, minimization perspective Daniel P. Palomar 13
Nonnegative weighted sum: if f 1, f 2 are convex, then α 1 f 1 +α 2 f 2 is convex, with α 1, α 2 0. Composition with affine functions: if f is convex, then f (Ax + b) is convex (e.g., y Ax is convex, logdet ( I + HXH T ) is concave). Pointwise maximum: if f 1,..., f m are convex, then f (x) = max {f 1,..., f m } is convex. Example: sum of r largest components of x R n : f (x) = x [1] + x [2] + + x [r] where x [i] is the ith largest component of x. Proof: f (x) = max {x i1 + x i2 + + x ir 1 i 1 < i 2 < < i r n}. Daniel P. Palomar 14
Pointwise supremum: if f (x, y) is convex in x for each y A, then g (x) = sup f (x, y) y A is convex. Example: distance to farthest point in a set C: f (x) = sup y C x y. Example: maximum eigenvalue of symmetric matrix: for X S n, λ max (X) = sup y 0 y T Xy y T y. Daniel P. Palomar 15
Composition with scalar functions: let g : R n R and h : R R, then the function f (x) = h (g (x)) satisfies: f (x) is convex if g convex, h convex nondecreasing g concave, h convex nonincreasing Minimization: if f (x, y) is convex in (x, y) and C is a convex set, then g (x) = inf f (x, y) y C is convex (e.g., distance to a convex set). Example: distance to a set C: is convex if C is convex. f (x) = inf y C x y Daniel P. Palomar 16
Perspective: if f (x) is convex, then its perspective g (x, t) = tf (x/t), dom g = { (x, t) R n+1 x/t dom f, t > 0 } is convex. Example: f (x) = x T x is convex; hence g (x, t) = x T x/t is convex for t > 0. Example: the negative logarithm f (x) = log x is convex; hence the relative entropy function g (x, t) = t log t t log x is convex on R 2 ++. Daniel P. Palomar 17
Quasi-Convexity Functions A function f : R n R is quasi-convex if dom f is convex and the sublevel sets S α = {x dom f f (x) α} are convex for all α. f is quasiconcave if f is quasiconvex. Daniel P. Palomar 18
Examples: x is quasiconvex on R ceil (x) = inf {z Z z x} is quasilinear log x is quasilinear on R ++ f (x 1, x 2 ) = x 1 x 2 is quasiconcave on R 2 ++ the linear-fractional function is quasilinear f (x) = at x + b c T x + d, dom f = { x c T x + d > 0 } Daniel P. Palomar 19
Log-Convexity A positive function f is log-concave is log f is concave: f (θx + (1 θ) y) f (x) θ f (y) 1 θ for 0 θ 1. f is log-convex if log f is convex. Example: x a on R ++ is log-convex for a 0 and log-concave for a 0 Example: many common probability densities are log-concave Daniel P. Palomar 20
Convexity w.r.t. Generalized Inequalities f : R n R m is K-convex if dom f is convex and for any x, y dom f and 0 θ 1, f (θx + (1 θ) y) K θf (x) + (1 θ) f (y). Example: f : S m S m, f (X) = X 2 is S m +-convex. Daniel P. Palomar 21
References Chapter 3 of Stephen Boyd and Lieven Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge University Press, 2004. http://www.stanford.edu/ boyd/cvxbook/ Daniel P. Palomar 22