Convex Sets. Prof. Dan A. Simovici UMB

Convex Sets Prof. Dan A. Simovici UMB 1 / 57

Outline 1 Closures, Interiors, Borders of Sets in R n 2 Segments and Convex Sets 3 Properties of the Class of Convex Sets 4 Closure and Interior Points of Convex Sets 5 Proximal Point in a Convex Set 6 Separation of Convex Sets 7 Theorems of Alternatives 2 / 57

Closures, Interiors, Borders of Sets in R n Open and Closed Spheres Definition An open sphere of radius r centered in x 0 is the set B(x 0, r) = {x R n x x 0 < r}. A closed sphere of radius r centered in x 0 is the set B[x 0, r] = {x R n x x 0 r}. 3 / 57

Closures, Interiors, Borders of Sets in R n Closure of a Set Definition Let S be a subset of R n. A point x is in the closure of a set S if S B(x, r) for every r > 0. The closure of S is denoted by K(S). If S = K(S), then S is said to be closed. 4 / 57

Closures, Interiors, Borders of Sets in R n Interior of a Set Definition Let S be a subset of R n. A point x is in the interior of a set S if B(x, r) S for some r > 0. The interior of S is denoted by I (S). If S = I (S), then S is said to be open. 5 / 57

Closures, Interiors, Borders of Sets in R n Boundary of a Set Definition Let S be a subset of R n. A point x is in the border of a set S if we have both B(x, r) S and B(x, r) (R n S) for every r > 0. The border of S is denoted by (S). 6 / 57

Closures, Interiors, Borders of Sets in R n Example The set S = B[0 2, 1] = {x R 2 x1 2 + x 2 2 S = K(S). The interior I (S) is 1} is closed, that is, while the border of S is B(0 2, 1) = {x R 2 x 2 1 + x 2 2 < 1}, S = {x R 2 x 2 1 + x 2 2 = 1}. 7 / 57

Segments and Convex Sets Segments Let x, y R n. The closed segment determined by x and y is the set [x, y] = {ax + (1 a)y 0 a 1}. The half-closed segments determined by x and y are the sets [x, y) = {ax + (1 a)y 0 < a 1}, and (x, y] = {ax + (1 a)y 0 a < 1}. The open segment determined by x and y is (x, y) = {ax + (1 a)y 0 < a < 1}. 8 / 57

Segments and Convex Sets Definition A subset C of R n is convex if, for all x, y C we have [x, y] C. Note that the empty subset and every singleton {x} of R n is convex. 9 / 57

Segments and Convex Sets Example Every linear subspace T of R n is convex. Example The set R n 0 of all vectors of Rn having non-negative components is a convex set called the non-negative orthant of R n. Example The convex subsets of (R, +, ) are the intervals of R. Regular polygons are convex subsets of R 2. 10 / 57

Segments and Convex Sets Example An open sphere C(x 0, r) R n is convex, where R n is equipped with the Euclidean norm. Indeed, suppose that x, y C(x 0, r), that is, x x 0 < r and y x 0 < r. Let a [0, 1] and let z = ax + (1 a)y. We have so z C(x 0, r). x 0 z = x 0 ax (1 a)y = a(x 0 x) + (1 a)(x 0 y) a x 0 x +(1 a) x 0 y) r. 11 / 57

Segments and Convex Sets Definition Let U be a subset of R n and let x 1,..., x k U. A linear combination of U, a 1 x 1 + + a k x k is an affine combination of U if k i=1 a i = 1; a non-negative combination of U if a i 0 for 1 i k; a positive combination of U if a i > 0 for 1 i k; a convex combination of U if it is a non-negative combination of U and a 1 + + a k = 1. 12 / 57

Segments and Convex Sets Theorem A subset C of R n is convex if and only if any convex combination of elements of C belongs to C. 13 / 57

Properties of the Class of Convex Sets Theorem Let C 1,..., C k be convex subsets of R n. If b 1,..., b k R, then is a convex set. {y = b 1 x 1 + + b k x k x i C i for 1 i k} 14 / 57

Properties of the Class of Convex Sets Theorem If C is a convex subset of R m and f : R m R n is an affine mapping, then the set f (C) is a convex subset of R n. If D is a convex subset of R n, then f 1 (D) = {x R n f (x) D} is a convex subset of R m. Proof: Since f is an affine mapping we have f (x) = Ax + b, where A R n m and b R n for x R m. Therefore, if y 1, y 2 f (C) we can write y 1 = Ax 1 + b and y 2 = Ax 2 + b. This, in turn, allows us to write for a [0, 1]: ay 1 + (1 a)y 2 = a(ax 1 + b) + (1 a)(ax 2 + b) = A(ax 1 + (1 a)x 2 ) + b = h(ax 1 + (1 a)x 2 ). The convexity of C implies ax 1 + (1 a)x 2 C, so ay 1 + (1 a)y 2 f (C), which shows that f (C) is convex. 15 / 57

Properties of the Class of Convex Sets Definition A subset D of R n is affine if, for all x, y C and all a R, we have ax + (1 a)y D. In other words, D is an affine set if every point on the line determined by x and y belongs to C. Note that D is a subspace of R n if 0 D and D is an affine set. 16 / 57

Properties of the Class of Convex Sets Theorem Let D be a non-empty affine set in R n. There exists translation t u and a unique subspace L of R n such that D = t u (L). 17 / 57

Properties of the Class of Convex Sets Proof Let L = {x y x, y D} and let x 0 D. We have 0 = x 0 x 0 L and it is immediate that L is an affine set. Therefore, L is a subspace. Suppose that D = t u (L) = t v (K), where both L and K are subspaces of R n. Since 0 K, it follows that there exists w L such that u + w = v. Similarly, since 0 L, it follows that there exists t K such that u = v + t. Consequently, since w + t = 0, both w and t belong to both subspaces L and K. If s L, it follows that u + s = v + z for some z K. Therefore, s = (v u) + z K because w = v u K. This implies L K. The reverse inclusion can be shown similarly. 18 / 57

Properties of the Class of Convex Sets Theorem Let A R m n and let b R m. The set S = {x R n Ax = b} is an affine subset of R n. Conversely, every affine subset of R n is the set of solutions of a system of the form Ax = b. 19 / 57

Properties of the Class of Convex Sets Proof It is immediate that the set of solutions of a linear system is affine. Conversely, let S be an affine subset of R n and let L be the linear subspace such that S = u + L. Let {a 1,..., a m } be a basis of L. We have L = {x R n a ix = 0 for 1 i m} = {x R n Ax = 0}, where A is a matrix whose rows are a 1,..., a m. By defining b = Au we have S = {u + x Ax = 0} = {y R n Ay = b}. 20 / 57

Properties of the Class of Convex Sets Theorem The intersection of any collection of convex (affine) sets in R n is a convex (affine) set. This allows us to define the convex closure K conv (S) of a subset S of R n as the intersection of all convex sets that contain S. This is the least convex set that contains S. Simlarly, K aff (S), the intersection of all affine sets that contain S is the least affine subset of R n that contains S. 21 / 57

Closure and Interior Points of Convex Sets Theorem Let S be a convex set in R n with I (S). If x 1 K(S) and x 2 I (S), then ax 1 + (1 a)x 2 S for a (0, 1). 22 / 57

Closure and Interior Points of Convex Sets Proof Since x 2 I (S) there exists ɛ > 0 such that B(x 2, ɛ) S. Let y = ax 1 + (1 a)x 2. To show that y I (S) it is sufficient to show that B(y, (1 a)ɛ) S. z 2 z x 2 I (S) z 1 y x 1 K(S) 23 / 57

Closure and Interior Points of Convex Sets Proof (cont d) Since x 1 K(S) we have ( ) (1 a)ɛ z y B x 1, S. a In particular, there exists z 1 S such that z 1 x 1 < (1 a)ɛ z y. a z 2 z x 2 I (S) y z 1 24 / 57

Closure and Interior Points of Convex Sets Proof (cont d) Define z 1 = z az 1 1 a. This allows us to write z 2 x 2 = z az 1 1 a x 2 = 1 1 a (z y) + a(x 1 z 1 ) 1 1 a ( (z y) +a x 1 z 1 < ɛ, so z 2 S. By the definition of z 2 note that z = az 1 + (1 a)z 2. Since z 1, z 2 S, we have z S. Therefore, y I (S). 25 / 57

Closure and Interior Points of Convex Sets Corollary For a convex set S, I (S) is convex. Corollary If S is a convex set and I (S), then K(S) is convex 26 / 57

Closure and Interior Points of Convex Sets Proof Let x 1, x 2 K(S) and let z I (S). By the theorem on slide 22, ax 2 + (1 a)z I (S) for each a (0, 1). For b (0, 1) we have bx 1 + (1 b)(ax 2 + (1 a)z)i (S) S. Since lim a 1 bx 1 + (1 b)(ax 2 + (1 a)z) = bx 1 + (1 b)x 2 K(S). 27 / 57

Closure and Interior Points of Convex Sets Corollary Let S be a set with I (S). Then, K(I (S)) = K(S). 28 / 57

Closure and Interior Points of Convex Sets Proof It is clear that K(I (S)) K(S). Let x K(S) and y I (S) (since I (S) ). Then, ax + (1 a)y I (S) for each a (0, 1). Since x = lim a 1 ax + (1 a)y K(I (S)), the equality follows. 29 / 57

Closure and Interior Points of Convex Sets Corollary Let S be a set with I (S). Then I (K(S)) = I (S). 30 / 57

Closure and Interior Points of Convex Sets Proof We have I (S) I (K(S)). Let x 1 I (K(S)). There exists ɛ > 0 such that B(x 1, ɛ) K(S). Let x 2 x 1 that belongs to I (S) and let y = (1 + b)x 1 bx 2, ɛ where b = 2 x 2 x 1. Since y x 1 = ɛ 2, we have y K(S). But x 1 = cy + (1 c)x 2, where c = 1 1+b (0, 1). Since y K(S) and x 2 I (S), then x 1 I (S). 31 / 57

Proximal Point in a Convex Set The Proximal Point Lemma Let C be a nonempty and closed convex set, C R n and let x 0 C. There exits a unique point u C such that u x 0 is the minimal distance from x 0 to C. 32 / 57

Proximal Point in a Convex Set Proof Let µ = min{ x x 0 x C}. There exists a sequence of elements in C, (z n ) such that lim n z n x 0 = µ. By the law of the parallelogram, z k z m 2 = 2 z k x 0 2 +2 z m x 0 2 4 x k +x m 2 x 0 2. Since C is convex, we have x k +x m 2 C; the definition of µ implies that x k + x m 2 x 0 µ 2, 2 so z k z m 2 2 z k x 0 2 +2 z m x 0 2 4µ 2. 33 / 57

Proximal Point in a Convex Set Proof (cont d) Since lim n z n x 0 = µ, for every ɛ > 0, there exists n ɛ such that k, m > n ɛ imply z k x 0 < µɛ and z m x 0 < µɛ. Therefore, if k, m > n ɛ, it follows that z k z m 2 4µ 2 (ɛ 2 1). Thus, (z n ) is a Cauchy sequence. If lim n z n = u, then u C because C is a closed set. 34 / 57

Proximal Point in a Convex Set Suppose v C with v v and v x 0 = u x 0. Since C is convex, w = 1 2 (u + v) C and we have 1 2 (u + v) x 1 0 2 u x 0 + 1 2 v x 0 = µ. If 1 2 (u + v) x 0 < µ, the definition of µ is violated. Therefore, we have 1 2 (u + v) x 0 = µ, which implies u x 0 = k(v x 0 ) for some k R. This, in turn, implies k = 1. If k = 1 we would have u x 0 = v x 0, so u = v, which is a contradiction. Therefore, a = 1 and this implies x 0 = 1 2 (u + v) C, which is again a contradiction. This implies that u is indeed unique. 35 / 57

Proximal Point in a Convex Set The point u whose existence and uniqueness is was established is the C-proximal point to x 0. u 1 1 C x 1 x 0 36 / 57

Proximal Point in a Convex Set Lemma Let C be a nonempty and closed convex set, C R n and let x 0 C. Then u C is the C-proximal point to x 0 if and only if for all x C we have (x u) (u x 0 ) 0. 37 / 57

Proximal Point in a Convex Set Proof Let x C. Since x x 0 2 = x u + u x 0 2 = x u 2 + u x 0 2 +(x u) (u x 0 ), u x 0 2 0 and (x u) (u x 0 ) 0, it follows that x x 0 x u, which means that u is the closest point in C to x 0, and the condition of the lemma is sufficient. 38 / 57

Proximal Point in a Convex Set Proof (cont d) Conversely, suppose that u is the proximal point in C to x 0, that is, x x 0 x 0 u for x C. If t is positive and sufficiently small, then u + t(x u) C because x C. Consequently, Since x 0 u t(x u) 2 x 0 u 2. x 0 u t(x u) 2 = x 0 u 2 2t(x 0 u) (x u) + t 2 x u 2 it follows that 2t(x 0 u) (x u) + t 2 x u 2 0, which implies (x u) (u x 0 ) 0, when we divide the previous equality by a 0. 39 / 57

Separation of Convex Sets Definition Let S 1, S 2 be two subsets of R n and let H w,a be a hyperplane in R n. H w,a separates S 1 and S 2 if w x a for x S 1 and w x a for x S 2 ; strictly separates S 1 and S 2 if w x > a for x S 1 and w x < a for x S 2 ; strongly separates S 1 and S 2 if w x > a + ɛ for x S 1 and w x < a for x S 2 and some ɛ > 0. 40 / 57

Separation of Convex Sets Separation between a Convex Set and a Point Theorem Let S be a non-empty convex set in R n and y S. There exists w 0 n and a R such that w y > a and w x a for x S. 41 / 57

Separation of Convex Sets Proof Since S is non-empty and closed and y S there exists a unique closest point x 0 S such that (x x 0 ) (y x 0 ) 0 for each x S. Equivalently, Since for w = y x 0 0 n we have x 0(y x 0 ) x (y x 0 ). y x 0 2 = (y x 0 ) (y x 0 ) = y (y x 0 ) x 0(y x 0 ) y (y x 0 ) x (y x 0 ) = (y x) (y x 0 ), w (y x) y x 0 2, so w y w x+ y x 0 2. If a = sup{w x x S} we have the desired inequalities. 42 / 57

Separation of Convex Sets A variation of the previous theorem, where C is just a convex set (not necessarily closed) is given next. Theorem Let C be a nonempty convex set, C R n and let x 0 C. There exists w R n {0 n } and a R such that w (x x 0 ) 0 for x K(C). 43 / 57

Separation of Convex Sets Proof Since x 0 C, there exists a sequence (z m ) such that z m K(C) and lim m z m = x 0. By Theorem on slide 41, for each m N there exists w m R n {0 n } such that w mz m > w mx for each x K(C). Without loss of generality we may assume that w m = 1. Since the sequence (w m ) is bounded, it contains a convergent subsequence w ip such that lim p w ip = w and we have w i p z ip > w i p x for each x K(C). Taking p we obtain w x 0 > w x for x K(C). 44 / 57

Separation of Convex Sets Theorem Let C be a nonempty convex set, C R n and let x 0 C. There exists w R n {0 n } and a R such that w (x x 0 ) 0 for x K(C). Proof: If x K(C), the statement follows from the Theorem on slide 41. Otherwise, x 0 K(C) C C, so x 0 C and the statement is a consequence of Theorem from slide 43. 45 / 57

Separation of Convex Sets Theorem Let C R n be a closed and convex set. Then, C equals the intersection of all half-spaces that contain C. Proof: It is immediate that C is included in the intersection of all half-spaces that contain C. Conversely, suppose that z be a point contained in all halfspaces that contain C such that z C. There exists a half-space that contains C but not z, which contradicts the definition of z. Thus, the intersection of all half-spaces that contain C equals C. 46 / 57

Separation of Convex Sets Definition Let S be a non-empty subset of R n and let x 0 (S). A supporting hyperplane of S at x 0 is a hyperplane H w,a such that either S H + w,a where w (x x 0 ) 0 for each x S, or S H w,a where w (x x 0 ) 0 for each x S. Equivalently, H w,a is a supporting hyperplane at x 0 partial(s) if either w x 0 = inf{w x x S}, or w x 0 = sup{w x x S}. 47 / 57

Separation of Convex Sets Theorem Let C R n be a nonempty convex set and let x 0 C. There exists a supporting hyperplane of C at x 0. Proof: Since x 0 C, there exists a sequence (z n ) of elements of R n C such that lim n z n = x 0. For each z n there exists w n such that w nz n > a and w nx a for x C. Without loss of generality we may assume that w n = 1. Since the sequence (w n ) is bounded, it contains a convergent subsequence (w im ) such that lim m w im = w. For this subsequence we have w z im > a and w x a. Taking m we obtain w x 0 > a and w x a for all x C, which means that H w,a is a support plane of C at x 0. 48 / 57

Separation of Convex Sets Theorem Let S, T be two non-empty convex subsets of R n that are disjoint. There exists w R n {0 n } such that inf{w s s S} sup{w t t T }. Proof: It is easy to see that the set S T defined by S T = {s t s S and t T } is convex. Furthermore 0 n S T because the sets S and T are disjoint. Thus, there exists in S T a proximal point w to 0 n, for which we have (x w) w 0 for every x S T, that is, (s t w) w 0, which is equivalent to s w t w+ w 2 for s S and t T. This implies the inequality of the theorem. 49 / 57

Separation of Convex Sets Corollary For any two non-empty convex subsets that are disjoint, there exists a non-zero vector w R n such that inf{w s s S} sup{w t t K(T )}. 50 / 57

Theorems of Alternatives Notation For x, y R n we write if x i > y i for 1 i n, if x i y i for 1 i n, and x > y x y x y if x i y i for 1 i n and at least of these inequalities is strict, that is, there exists i such that x i > y i. 51 / 57

Theorems of Alternatives Separation results have two important consequences for optimization theory, namely Farkas and Gordan s alternative theorems. Theorem (Farkas Alternative Theorem) Let A R m n and let c R n. Exactly one of the following linear systems has a solution: (i) Ax 0 m and c x > 0; (ii) A y = c and y 0 m. 52 / 57

Theorems of Alternatives Proof First System Ax 0 m c x > 0 Second System A y = c y 0 m If the second system has a solution, then A y = c and y 0 m for some y R m. Suppose that x is a solution of the first system. Then, c x = y Ax 0, which contradicts the inequality c x > 0. Thus, if the second system has a solution, the first system has no solution. 53 / 57

Theorems of Alternatives First System Ax 0 m c x > 0 Second System A y = c y 0 m Suppose now that the second system has no solution. Note that the set S = {x R n x = A y, y 0 m } is a closed convex set. Furthermore, c S because, otherwise, c would be a solution of the second system. Thus, there exists w R n {0 n } and a R such that w c > a and w x a for x S. In particular, since 0 n S we have a 0 and, therefore, w c > 0. Also, for y 0 m we have a w A y = y Aw. Since y can be made arbitrarily large we must have Aw 0 m. Then w is a solution of the first system. 54 / 57

Theorems of Alternatives Theorem (Gordan s Alternative Theorem) Let A R m n be a matrix. Exactly one of the following linear systems has a solution: Ax < 0 m for x R n ; A y = 0 n and y 0 m for y R m. 55 / 57

Theorems of Alternatives Proof First System Ax < 0 m Second System A y = 0 n y 0 m Let A be a matrix such that the first system, Ax < 0 m has a solution x 0. Suppose that a solution y 0 of the second system exists. Since Ax 0 < 0 m and y 0 0 m (which implies that at least one component of y 0 is positive) it follows that y 0 Ax 0 < 0, which is equivalent to x 0 A y < 0. This contradicts the assumption that A y = 0 n. Thus, the second system cannot have a solution if the first has one. 56 / 57

Theorems of Alternatives Proof (cont d) First System Second System Ax < 0 m A y = 0 n y 0 m Suppose now that the first system has no solution and consider the non-empty convex subsets S, T of R m defined by S = {s R m s = Ax, x R n } and T = {t R m t < 0 m }. These sets are disjoint by the previous supposition. Then, there exists w 0 m such that w As w t for s S and t K(T ). This implies that w 0 m because otherwise the components of t that correspond to a negative component of w could be made arbitrarily negative (and large in absolute value) and this would contradict the above inequality. Thus, w 0 m. Since 0 m K(T ), we also have w As 0 for every s R m. In particular, for s = A w we obtain w A( A w) = A w 2 = 0, so A w = 0 m, which means that the second system has a solution. 57 / 57