IE 521 Convex Optimization

Transcription

1 Lecture 5: Convex II 6th February 2019

2 Convex Local Lipschitz Outline Local Lipschitz 1 / 23

3 Convex Local Lipschitz Convex Function: f : R n R is convex if dom(f ) is convex and for any λ [0, 1], x, y dom(f ), f (λx + (1 λ)y) λf (x) + (1 λ)f (y). Remark. (Extended-value function) f can be extended to a function from R n R {+ } by setting f (x) = +, if x / dom(f ). We say f is convex if for any λ [0, 1], x, y, f (λx + (1 λ)y) λf (x) + (1 λ)f (y). 2 / 23

4 General Convex Inequality Proposition. If f is convex, then λ i 0, m i=1 λ i = 1, ( m ) f λ i x i i=1 m λ i f (x i ). i=1 Convex Local Lipschitz Remark. (Jensen s Inequality) Let ξ be a random variable and f be convex, then f (E[ξ]) E[f (ξ)]. 3 / 23

5 Convex Local Lipschitz Level set One-dimensional property First order condition Second order condition 4 / 23

6 What can you tell from these sets? Convex Local Lipschitz Figure: Four sets 5 / 23

7 Definition. The epigraph of a function f is epi(f ) = { (x, t) R n+1 : f (x) t } Convex Local Lipschitz Figure: Proposition. f is convex iff epi(f ) is a convex set. Q. Is epi(f ) always closed? 6 / 23

8 Convex Local Lipschitz Proposition. f is convex iff epi(f ) is a convex set. Proof. (if part) First, dom(f ) is convex since it is the projection of epi(f ). Second, since (x 1, f (x 1 )), (x 2, f (x 2 )) epi(f ), then (λx 1 + (1 λ)x 2, λf (x 1 ) + (1 λ)f (x 2 )) epi(f ), λ [0, 1] f (λx 1 + (1 λ)x 2 ) λf (x 1 ) + (1 λ)f (x 2 ), λ [0, 1]. (only if part) Let (x 1, t 1 ), (x 2, t 2 ) epi(f ), f (λx 1 + (1 λ)x 2 ) λf (x 1 ) + (1 λ)f (x 2 ), λ [0, 1] f (λx 1 + (1 λ)x 2 ) λt 1 + (1 λ)t 2, λ [0, 1] λ(x 1, t 1 ) + (1 λ)(x 2, t 2 ) epi(f ), λ [0, 1] epi(f )is a convex set 7 / 23

9 Convex Local Lipschitz Example. If f i (x), i I are convex, then so is g(x) := max i I f i (x). Note epi(g) = i I epi(f i ) is convex. Example. If f is convex, then so is the perspective function g(x, t) = tf (x/t), where dom(g) = {(x, t) : x/t dom(f ), t > 0}. Note epi(g) = P 1 (epi(f )) is convex, where P is the perspective mapping (x, t, s) (x/t, s/t). (x, t, s) epi(g) tf (x/t) s (x/t, s/t) epi(f ) 8 / 23

10 Definition. For any t R, the level set of f at level t is lev t (f ) = {x dom(f ) : f (x) t}. Convex Local Lipschitz Figure: Level set Proposition. If f is convex, then every level set is convex. Q. Is the reverse still true? 9 / 23

11 What are the level sets at t 1 and t 2? Are they convex? Convex Local Lipschitz Figure: Level set 10 / 23

12 Quasi-convex Definition. f is quasi-convex if all level sets are convex. Convex Proposition. f is quasi-convex iff f (λx + (1 λ)y) max {f (x), f (y)}, λ [0, 1] Remark. f is quasi-concave iff f (λx + (1 λ)y) min {f (x), f (y)}, λ [0, 1] Local Lipschitz Example. f (x) = log(x) is both quasi-convex and quasi-concave; f (x 1, x 2 ) = x 1 x 2 is quasi-convex on R 2 +; f (x) = x 0 is quasi-concave on R n +; f (X ) = rank(x ) is quasi-concave on S n / 23

13 Convex Local Lipschitz Proposition. f is convex if and only is its restriction on any line is convex, i.e., x, h R n, φ(t) = f (x + th) is convex on its domain dom(φ) = {t x + th dom(f )}. Remark. Checking convexity in R n boils down to check convexity of one-dimensional function on the axis. 12 / 23

14 Convex Local Lipschitz Example. The negative log-determinant function f (X ) = log(det(x )) is convex on S n ++. Sufficient to show that φ(t) = log(det(x + th)) is convex for any given X, H R n n. Once can compute that φ(t) = ln ( det ( ) ( ) ( X 1 2 det I + tx 1 2 HX 1 1 )) 2 det X 2 n = ln(1 + tλ i ) + φ(0) i=1 where {λ i } n i=1 are eigenvalues of X 1 2 HX / 23

15 Proposition. Assume f is differentiable, then f is convex if and only if dom(f ) is convex and f (x) f (x 0 ) + f (x 0 ) T (x x 0 ), x, x 0. Convex Local Lipschitz Figure: First-order condition Remark. ( f (x 0 ), 1) is a supporting hyperplane of epi(f ). 14 / 23

16 Convex Local Lipschitz Proposition. Assume f is differentiable, then f is convex if and only if dom(f ) is convex and Proof. f (x) f (x 0 ) + f (x 0 ) T (x x 0 ), x, x 0. (if part) λ [0, 1], let x 0 = λx + (1 λ)y, then f (x) f (x 0 ) + f (x 0 ) T (x x 0 ) f (y) f (x 0 ) + f (x 0 ) T (y x 0 ) λf (x) + (1 λ)f (y) f (x 0 ) = f (λx + (1 λ)y). (only if part) By convexity, we have for all λ [0, 1], f ((1 λ)x 0 + λx) (1 λ)f (x 0 ) + λf (x) f (x 0 + λ(x x 0 )) λ f (x 0) λ + f (x) f (x 0) f (x) f (x 0 ) + f (x + λ(x x 0)) f (x 0 ), λ [0, 1] λ f (x) f (x 0 ) + f (x 0 ) T (x x 0 ), letting λ 0 15 / 23

17 Convex Local Lipschitz Proposition. Assume f is twice-differentiable, then f is convex if and only if dom(f ) is convex and 2 f (x) 0, x dom(f ). Example. Quadratic function: f (x) = 1 2 x T Qx + b T x + c is convex if and only if Q 0. Log-sum-exp: f (x) = log( n i=1 ex i ) is convex on R n. Geometric mean: f (x) = ( n i=1 x i) 1/n is concave on R n / 23

18 Convex Local Lipschitz Proposition. Assume f is twice-differentiable, then f is convex if and only if dom(f ) is convex and Proof. 2 f (x) 0, x dom(f ). (if part) Any one dimensional restriction φ(t) = f (x+th) is convex since φ (t) = h T 2 f (x+th)h 0. Hence, f is convex. (only if part) x, h, φ(t) = f (x + th) is convex on the axis φ (t) = h T 2 f (x + th)h 0 φ (0) = h T 2 f (x)h 0, h. Hence 2 f (x) / 23

19 Convex Local Lipschitz Theorem. If f is convex, f is continuous on rint(dom(f )). Remark. Convex functions are almost everywhere continuous; f needs not to be continuous on dom(f ). E.g., 1, x = 0 f (x) = 0, x > 0 +, o.w. Corollary. Let f be convex and X rint(dom(f )) be a closed, bounded set. Then f is bounded on X. 18 / 23

20 Convex Local Lipschitz Proof. W.l.o.g, assume dim(dom(f )) = n, 0 int(dom(f )) and {x : x 2 1} dom(f ). Consider the continuity at point 0. Let {x k } 0 with x k 2 1. (a) lim sup k f (x k ) f (0). Observe x k = (1 x k 2 ) 0 + x k 2 y k, where y k = x k x k 2 dom(f ). By convexity of f, we have f (x k ) (1 x k 2 ) f (0) + x k 2 f (y k ) Therefore, lim sup k f (x k ) f (0). (b) lim inf k f (x k ) f (0). Observe 1 0 = x k x 2+1 k + x k 2 x k z 2+1 k, where z k = x k x k 2 dom(f ). By the convexity of f, we have f (0) 1 x k f (x k) + x k 2 x k f (z k) Therefore, lim inf k f (x k ) f (0). 19 / 23

21 Local Lipscthiz Theorem. Let f be convex and X rint(dom(f )) be a closed, bounded set. Then f is Lipschitz continuous on X. Proof: Skipped. Convex Local Lipschitz Remark.. All three conditions are essential (1) closedness (2) boundedness (3)relative interior f (x) = 1/x, X = (0, 1], not Lipschitz continuous f (x) = x 2, X = R, not Lipschitz continuous f (x) = x, X = [0, 1], not Lipschitz continuous 20 / 23

22 Closed Definition. A function is closed if epi(f ) is a closed set. Remark. Any continuous function is closed. A closed function is not necessarily continuous. Convex Local Lipschitz Proposition. The following are equivalent: (i) f (x) is closed; (ii) f (x) is lower-semicontinuous (l.s.c), i.e., x R n, {x k } x, f (x) lim inf k f (x k). (iii) Every level set is closed. 21 / 23

23 Convex Local Lipschitz Definition. A function is closed convex if epi(f ) is a closed and convex set. A convex is closed if it is lower-semicontinuous. Example. { +, x / C The indicator function I C (x) = is closed 0, x C convex if C is a closed convex set; { 1, x = 0 f (x) = is convex but not closed. 0, x > 0 Remark. A closed convex function f can be viewed as the pointwise supremum of all affine minorants of f (affine functions that underestimate f. 22 / 23

24 References Convex Boyd & Vandenberghe, Chapter 3.1 Ben-Tal & Nemirovski, Chapter Local Lipschitz 23 / 23