IE 521: Convex Optimization Instructor: Niao He Lecture 1: Convex Sets January 23 Spring 2017, UIUC Scribe: Niao He Courtesy warning: These notes do not necessarily cover everything discussed in the class. Please email TA (swang157@illinois.edu) if you find any typos or mistakes. In this lecture, we cover the following topics Topology review Convex sets (convex/conic/affine hulls) Examples of convex sets Calculus of convex sets Some nice topological properties of convex sets. 1.1 Topology Review Let X be a nonempty set in R n. A point x 0 is called an interior point if X contains a small ball around x 0, i.e. r > 0, such that B(x 0, r) := {x : x x 0 2 r} X. A point x 0 is called a limit point if there exits a convergent sequence in X that converges to x 0, i.e. {x n } X, such that x n x 0 as n. The interior of X, denoted as int(x), is the set of all interior point of X. The closure of X, denoted as cl(x), is the set of all limit points of X. The boundary of X, denoted as (X) = cl(x)/int(x), is the set of points that belongs to the closure but not in the interior. X is closed if cl(x) = X; X is open if int(x) = X. Here are some basic facts: int(x) X cl(x); A set X is closed if and only if its complement X c = R n /X is open; The intersection of arbitrary number of closed sets is closed, i.e., α A X α is closed if X α is closed for all α A. The union of finite number of closed sets is closed, i.e., n i=1 X i is closed if X i is closed for i = 1,..., n. 1-1
Lecture 1: Convex Sets January 23 1-2 1.2 Convex Sets Definition 1.1 (Convex set) A set X R n is convex if x, y X, λx + (1 λ)y X for any λ [0, 1]. In another word, the line segment that connects any two elements lies entirely in the set. (a) convex set (b) non-convex set Figure 1.1: Examples of convex and non-convex sets Given any elements x 1,..., x k, the combination λ 1 x 1 + λ 2 x 1 +... + λ k x k is called Covex: if λ i 0, i = 1,..., k and λ 1 + λ 2 +... + λ k = 1; Conic: if λ i 0, i = 1,..., k; Affine: if λ 1 + λ 2 +... + λ k = 1; Linear: if λ i R, i = 1,..., k. Consequently, we have A set is convex if all convex combinations of its elements are in the set; A set is a convex cone if all conic combinations of its elements are in the set; A set is a linear subspace if all affine combinations of its elements are in the set; A set is a linear subspace if all linear combinations of its elements are in the set. Clearly, a linear or affine subspace is always a convex cone; a convex cone is always a convex set.
Lecture 1: Convex Sets January 23 1-3 Definition 1.2 (Convex hull) A convex hull of a set X R n is the set of all convex combination of its elements, denoted as { k Conv(X) = λ ix i : k N, λ i 0, } k λ i = 1, x i X, i = 1,..., k. i=1 i=1 Figure 1.2: Examples of convex hulls Similarly, one can define the conic hull and affine hull of a set. Cone(X) = Aff(X) = { k { k } λ ix i : k N, x i X, λ i 0, i = 1,..., k. i=1 λ ix i : k N, x i X, } k λ i = 1, i = 1,..., k. i=1 i=1 Proposition 1.3 We have the following 1. A convex hull is always convex. 2. If X is convex, then conv(x) = X. 3. For any set X, conv(x) is the smallest convex set that contains X. Proof: 1. By definition, for any x, y Conv(X), we can write x = i λ ix i and y = i µ ix i where λ i, µ i 0 and i λ i = i µ i = 1. Hence, for any α [0, 1], we have αx + (1 α)y = α i λ i x i + (1 α) i µ i x i = i ξ i x i where ξ i = αλ i + (1 α)µ i, i. Note that ξ i 0 and i ξ i = α i λ i + (1 α) i µ i = 1. Therefore, αx + (1 α)y Conv(X). Hence, Conv(X) is convex.
Lecture 1: Convex Sets January 23 1-4 2. First of all, based on definition of convex hull, it is straightforward to see that X Conv(X). Next, we show that Conv(X) X by induction on k. The baseline is when k = 1, which is trivial. Now assuming that any convex combination with k entries is in X, we want to show that any convex combination of k + 1 entries is still in X. Consider the convex combination below given by λ 1,..., λ k+1 with λ i 0, i = 1,..., k + 1 and k+1 i=1 λ i = 1. ( ) λ 1 λ k λ 1 x 1 +... + λ k+1 x k+1 = (1 λ k+1 ) x 1 +... + x k +λ k+1 x k+1 1 λ k+1 1 λ }{{ k+1 } z Based on induction, we can see that z X since z is a convex combination of k entries in X. By convexity of X, we further have λ 1 x 1 +... + λ k+1 x k+1 X. 3. Suppose Y is convex and Y X, we want to show that Y Conv(X). From previous argument, if Y contains X, then Y should contain all convex combinations of X, i.e. Y Conv(X). Examples of Convex Sets Example 1. Some simple convex sets: Hyperplane: {x R n : a T x = b} Halfspace: {x R n : a T x b} Affine space: {x R n : Ax = b} Polyhedron: {x R n : Ax b} Simplex: {x R n : x 0, n i=1 x i = 1} = conv(e 1,..., e n ). Example 2. Euclidean balls: {x R n : x 2 r} where 2 is the Euclidean norm defined on R n. Example 3. Ellipsoid: {x R n : (x a) T Q(x a) r 2 } where Q 0 and is symmetric.
Lecture 1: Convex Sets January 23 1-5 1.3 Calculus of Convex Sets The following operators preserve the convexity of sets, which can be easily verified based on the definition. 1. Intersection: If X α, α A are convex sets, then α A X α 2. Direct product: If X i R n i, i = 1,..., k are convex sets, then X 1 X k := {(x 1,..., x k ) : x i X i, i = 1,..., k} 3. Weighted summation: If X i R n, i = 1,..., k are convex sets, then α 1 X 1 + + α k X k := {α 1 x 1 +... + α k x k : x i X i, i = 1,..., k} 4. Affine image: If X R n is a convex set and A(x) : x Ax + b is an affine mapping from R n to R k, then A(X) := {Ax + b : x X} 5. Inverse affine image: If X R n is a convex set and A(y) : y Ay +b is an affine mapping from R k to R n, then A 1 (X) := {y : Ay + b X} Proof: 1. Let x, y α A X α, then x, y X α, α A. Since X α is convex, for any λ [0, 1], λx + (1 λ)y X α, α A. Hence, λx + (1 λ)y α A X α. 2. Let x = (x 1,..., x k ) X 1 X k, y = (y 1,..., y k ) X 1 X k,. Since X i is convex, for λ [0, 1], λx i + (1 λ)y i X i, for i = 1,..., k. Hence λx + (1 λ)y = (λx 1 + (1 λ)y 1,..., λx k + (1 λ)y k ) X 1 X k. 3. Let x, y α 1 X 1 + + α k X k, by definition, there exists x i, y i X i, i = 1,..., k, such that x = α 1 x 1 +... + α k x k, y = α 1 y 1 +... + α k y k. Hence, for all λ [0, 1] λx + (1 λ)y = α 1 z 1 +... + α k z k α 1 X 1 + + α k X k because z i = λx i + (1 λ)y i X i, for all i = 1,..., k.
Lecture 1: Convex Sets January 23 1-6 4. Let y 1, y 2 A(X), then there exits x 1, x 2 X such that y 1 = Ax 1 + b and y 2 = Ax 2 + b. Therefore, for any λ [0, 1], we have λy 1 +(1 λ)y 2 = A(λx 1 +(1 λ)x 2 )+b A(X) because λx 1 + (1 λ)x 2 X. 5. Let y 1, y 2 A 1 (X), then there exits x 1, x 2 X such that x 1 = Ay 1 + b and x 2 = Ay 2 + b. Therefore, for any λ [0, 1], we have A(λy 1 + (1 λ)y 2 ) + b λx 1 + (1 λ)x 2 X, this implies that λy 1 + (1 λ)y 2 A 1 (X). 1.4 Nice Topological Properties of Convex Sets Convex sets are special because of their nice geometric properties. Proposition 1.4 If X be a convex set with nonempty interior, then int(x) is dense in cl(x). Proof: Let x 0 int(x) and x cl(x). We can construct a convergence sequence y n = 1 n x 0+(1 1 n x such that y n x. We only need to show that y n int(x). Therefore, it suffices to prove the following claim: Claim 1.5 If x 0 int(x) and x cl(x), then [x 0, x) int(x), namely, for any α [0, 1), the point z := αx 0 + (1 α)x int(x). This can be proved as follows. Since x 0 int(x), there exits r > 0 such that B(x 0, r) X. Since x cl(x), there exits a sequence {x n } X such that x n x. Let z n = αx 0 + (1 α)x n, then z n z. When n is large enough, z n z 2 αr 2. Since B(x 0, r) X and x n X, then B(z n, αr) = αb(x 0, r) + (1 α)x n X. Hence, B(z, αr 2 ) B(z n, αr) X. This is because for any z B(z, αr 2 ), z z αr 2, z z n 2 z z 2 + z n z 2 αr 2 + αr 2 = αr. Remark. Note that in general, for any set X, int(x) X cl(x), but int(x) and cl(x) can differ dramatically. For instance, let X be the set of all irrational numbers in (0, 1), then int(x) =, cl(x) = [0, 1]. The proposition implies that a convex set is perfectly well characterized by its closure or interior if nonempty.