Convex Optimization Convex Analsis - Functions p. 1
A function f : K R n R is convex, if K is a convex set and x, K,x, λ (,1) we have f(λx+(1 λ)) λf(x)+(1 λ)f(). (x, f(x)) (,f()) x <,,> - strictl convex, concave, strictl concave f : K R n R is convex iff x, K,x, λ (,1) is convex. g(λ) = f(λx+(1 λ)) p. 2
Sub-level sets If f : R n R is convex then α R is S α = {x f(x) α} convex and closed. and if there exists α such that S α = {x f(x) α } is nonempt and bounded, then S α is bounded for an α > α. Minima of a convex function Ever local minimum of a convex function is a global minimum. The set of all minima of a convex function is a convex set. Strictl convex function has at most one minimum. p. 3
Continuit Assume K R n is an open and convex set and f : K R is convex. Then f is Lipschitz continuous on an compact subset of U K, that is there exists a constant L such that x, U f(x) f() L x. Convex function f : K R n R defined on an open convex set K is continuous on K. Convex function f : K R n R defined on a closed convex set K is upper semi-continuous on K. Convex function defined on a closed convex set K is not necessaril continuous. f(x) a b p. 4
First order conditions The function f : K R n R defined on an convex set K is convex iff (First order Talor approximation las below the graph of f) (Monotone gradient) x,, x : f(x) f()+ f() T (x ) x,, x : ( f(x) f()) T (x ) >,,< - strictl convex/ concave/ strictl concave f ( x) * f ( ) f '( )( x ) x p. 5
Second order conditions Th function f : K R n R defined on an convex set K is convex iff (Positive semidefinite Hessian matrix) x : 2 f(x) - concave,, - strictl convex/ strictl concave - onl one implication holds - f(x) = x 4 p. 6
Second order conditions f ( x) qx ( ) qx ( ) * x Convex function f(x) and its second order Talor approximation at the point - convex quadratic function q(x) = f()+ f() T (x )+ 1 2 (x )T 2 f()(x ) p. 7
Epigraph of a function f : K R n R is the set epif = {(x,t) x K,f(x) t} The function is convex its epigraph is a convex set. First order condition interpretation: the hperplane defined b the normal vector ( f(), 1) is the supporting hperplane epif at the boundar point (,f()) (x,t) epif ( f() 1 ) T ( x t ) ( f() 1 ) T ( f() ) epi f (, f()) (f (), 1) p. 8
Operations preserving convexit nonnegative linear combination: if f 1,...,f m are convex w 1,...,w m - then g(x) = w 1 f 1 (x)+ +w m f m (x) is convex. affine transformation of variables: if f is convex, then g(x) = f(ax+b) is convex point-wise maximum: if f 1,...,f m are convex, then is convex g(x) = max{f 1 (x),...,f m (x)} p. 9
Operations preserving convexit supremum: if C is f(x,) convex in x and sup C f(x,) <, then is convex. g(x) = sup C f(x,) Example: distance (of the point x) to the farthest point of the set C: g(x) = sup C x p. 1
Operations preserving convexit infimum: if f(x,) is convex in (x,), the set C is convex and inf C f(x,) >, then is convex g(x) = inf C f(x,) Example: distance of the point x from the convex set C: dist(x,c) = inf C x p. 11
Operations preserving convexit If f : R n R is convex, then the perspective function ( x g : R n R ++ R, g(x,t) = tf t) is convex 4 4 3.5 3 3 2.5 2 2 1.5 1 1.5-2 -1.5-1 -.5.5 1 1.5 2 2 1 x -1-2 2 1.5 1.5 p. 12
Composition h(t) : R R, f(x) : R n R, H(x) = h(f(x)) : R n R f h h H ր ց ր ց 3 25 1 2.8 15.6 1.4 5 2 1-1 -2-2 -1 x 1 2.2 1 8 6 4 x 2 8 1 6 4 2 f(x 1,x 2 ) = exp(x 2 1 +x 2 2) f(x 1,x 2 ) = 1/ x 1 x 2 p. 13
! the reverse implication does not hold! H(x 1,x 2 ) = x 1 x 2 is concave on R 2 ++, h(t) = t is concave and increasing on R ++, but f(x 1,x 2 ) = x 1 x 2 is not concave (and not convex) Vector composition h() : R m R, f i (x) : R n R, i = 1,...,m, H : R n R, H(x) = h(f 1 (x),...,f m (x)) f i, i h h H րրր ցցց րրր ցցց p. 14
Quasi-convex functions The function f : K R n R is called quasi-convex, if the set K is convex and x, K, λ [,1] it holds f(λx+(1 λ)) max{f(x),f()}. <,,> - strictl quasi-convex, quasi-concave, strictl quasi-convex quasilinear functions Equivalent definition: The function f : K R n R is called quasi-convex, if the set K is convex and α R n are the sub-level sets S α = {x f(x) α} convex. f : K R n R is quasi-convex x, K, λ [,1] is the function quasi-convex. g(λ) = f(λx+(1 λ)) p. 15
Quasi-convex functions f( x) f( x) f( x) x2 x1 x x 1 x2 x a b x a x b x2 x1 x x 1 2 a x b x f( x) f( x) f( x) a x b x a x b x a x b x p. 16
Operations preserving quasi-convexit weighted maximum: if f 1,...,f m are quasi-convex, w 1,...,w m - then g(x) = max{w 1 f 1 (x),...,w m f m (x)} is quasi-convex supremum: if C is the function f(x, ) quasi-convex in x and sup C f(x,) <, then g(x) = sup C f(x,) is quasi-convex infimum: if f(x,) is quasi-convex in (x,), the set C is convex and inf C f(x,) >, then is quasi-convex g(x) = inf C f(x,) p. 17
First order conditions The function f : K R n R defined on a convex set K is quasi-convex iff x,, x : f(x) f() f() T (x ) Geometric interpretation: if f(), then f() is the normal of the supporting hperplane of the sub-level set at the point. S = {x f(x) f()} grad f() p. 18
Second order conditions If f is quasi-convex, then x, it holds T f(x) = T 2 f(x) If the function f satisfies x, then f is quasi-convex. T f(x) = T 2 f(x) >, x : f(x) = is 2 f(x) positive (semi)definite If f(x), then 2 f(x) is positive semidefinite on the subspace f(x) - the matrix ( ) 2 f(x) f(x) H(x) = f(x) T has exactl one negative eigenvalue. p. 19
Strong convexit The function f : R n R is called strong convex, if there exists β > such that x and λ (,1) it holds f(λx+(1 λ))+βλ(1 λ) x 2 λf(x)+(1 λ)f(). q(x) = βx T x = β x 2 2, β > is the weakest strong convex function f(x) is strong convex there exists β > such that the function h(x) = f(x) q(x) is convex. If f(x) is strong convex, α are the sub-level sets convex and compact. S α = {x f(x) α} p. 2
5 1 4 8 h(x 1,x 2 ) 3 2 f(x 1,x 2 ) 6 4 1 2 1 x 2-1 -.5 x 1.5 1 x 2-1 -.5 x 1.5 Convex function h(x 1,x 2 ) = e x 1+x 2 2 and strong convex function f(x) = h(x)+βx T x p. 21
First and second order conditions The function f(x) is strong convex if there exists β > such that f(x) f()+ f() T (x )+β x 2 ( f(x) f()) T (x ) 2β x 2 2 f(x) βi p. 22
Generalized convexit The cone K is called proper if it has the following properties: K is convex; K is closed; K is solid - int(k) ; K is pointed - x K x K x = Example: R n +, S n +, C 2 The partial ordering associated with the cone K: x K x K, x K x int(k) p. 23
Properties of the generalized inequalities propert K K invariant x K, u K v x K, u K v x+u K +v x+u K +v x K,α αx K α x K,α > αx K α reflexive x K x! x K x transitive x K, K z x K, K z x K x K z antismmetric x K, K x x = x K, u,v small enough: x+u K +v x i K i, i, x i x, i x K p. 24
K = R n + x K x i i i = 1,2,...,n K = S n + Löwner partial ordering of smmetric matrices : - A = QΛQ T αi - spectrum of the matrix A is bounded above with the constant α - for the positive semidefinite matrices - the inequalit A B implies h(a) h(b) and det(a) det(b) p. 25
Generalized convexit: Let K R m be a proper cone and K is the associated generalized inequalit. Then the function f : R n R m is called K-convex if x, R n and λ [,1] it holds f(λx+(1 λ)) K λf(x)+(1 λ)f(). Example: Matrix convexit The function f : R n S m is called matrix convex if f(λx+(1 λ)) λf(x)+(1 λ)f(). x, R n and λ [,1]. Equivalent definition: the function z T f(x)z is convex z R m. E. g. the function f : R n m S n,f(x) = XX T is matrix convex since for fixed z is the function z T XX T z = X T z 2 convex quadratic. p. 26