SWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm. Convex Optimization. Computing and Software McMaster University

Transcription

1 SWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm Convex Optimization Computing and Software McMaster University

2 General NLO problem (NLO : Non Linear Optimization) (N LO) min f(x) s.t. h i (x) = 0, i I = {1,, p} g j (x) 0, j J = {1,, m} x C where x IR n, C IR n is a given set, f, h 1,..., h p, g 1,..., g m are functions defined on C. Set of feasible solutions: F = {x C h i (x) = 0, i and g j (x) 0, j}.

3 Convex sets Definition 1. Given two points x 1, x 2 IR n and 0 λ 1, the point x = λx 1 + (1 λ)x 2 is a convex combination of x 1 and x 2. The set C IR n is called convex if all convex combinations of any two points x 1, x 2 C are again in C. Convex and nonconvex sets in the plane.

4 Convex functions Definition 2. A function f : C R defined on a convex set C is convex if, for all x 1, x 2 C and 0 λ 1, the following inequality holds: f(λx 1 + (1 λ)x 2 ) λf(x 1 ) + (1 λ)f(x 2 ). Definition 3. The epigraph of a function f : C R is the (n + 1)- dimensional set {(x, τ) : f(x) τ, x C, τ IR}. The epigraph of a convex function f. f C

5 Convex hull For any set S IR n, we can associate a convex set, the convex hull of S, in the following way. Definition 4. Given a set S IR n, its convex hull is defined by: conv(s) := {x x = λx 1 + (1 λ)x 2, x 1, x 2 S conv(s), λ (0, 1)} conv(s) is the smallest convex set containing S.

6 Affine hull If L is a (linear) subspace and a IR n, then a+l is an affine subspace of IR n. The dimension of a + L is the dimension of L. Definition 5. The smallest affine space a + L containing a convex set C IR n is the affine hull of C and is denoted by aff(c). The dimension of C is the dimension of aff(c). Definition 6. Given two points x 1, x 2 IR n and λ IR, the point x = λx 1 + (1 λ)x 2 is an affine combination of x 1 and x 2.

7 Convex cones Definition 7. The set C IR n is a convex cone if C is convex set and λx C for all x C and 0 λ. The following set C is a convex cone in IR 2 S = {(x 1, x 2 ) IR 2 x 2 2x 1, x x 1} The following set C is a convex cone in IR 3 : C = {(x 1, x 2, x 3 ) IR 3 x x2 2 x2 3, x 3 0}. C C

8 Convex cones Definition 8. A convex cone is pointed if it does not contain any subspace except the origin. A pointed closed convex cone could be defined equivalently as a convex cone not containing any line. Lemma 1. A convex cone C is pointed if and only if the origin 0 is an extremal point of C.

9 Recession cone Lemma 2. Assume that C is a closed and unbounded convex set, then: (i) for each x C there is a vector z IR n such that x + λz C for all λ 0, i.e. the set R(x) = {z x + λz C, λ 0} is not empty; (ii) the set R(x) is a closed convex cone (called the recession cone at x); (iii) the cone R(x) = R is independent of x, thus it is the recession cone of the convex set C; (iv) R is a pointed cone if and only if C has at least one extremal point. Corollary 1. The nonempty closed convex set C is bounded if and only if its recession cone R consists only of the zero vector.

10 Recession cone Let C be the epigraph of f(x) = 1 x. Every point on x 2 = 1 x 1 is an extreme point of C. For x = (x 1, x 2 ) the recession cone is given by R(x) = {z IR 2 z 1, z 2 0}. C x + R(x) x

11 Relative interior Definition 9. Given a convex set C, the point x C is in the relative interior of C if, for all x C, there exists x C and 0 < λ < 1 such that x = λx + (1 λ) x. The set of all the relative interior points of C is denoted by C 0. Let C = {x IR 3 x x2 2 1, x 3 = 1} and L = {x IR 3 x 3 = 0}, then C aff(c) = (0, 0, 1) + L. Hence, dim(c) = 2 and C 0 = {x IR 3 x x2 2 < 1, x 3 = 1}. C

12 Relative interior Lemma 3. Given a convex set C IR n, for each x C 0, y C, and 0 < λ 1 we have z = λx + (1 λ)y C 0 C. Corollary 2. The relative interior C 0 of a convex set C IR n is convex. Lemma 4. Given a convex set C, if C is nonempty then its relative interior C 0 is nonempty as well. Moreover, (C 0 ) 0 = C 0.

13 Convex functions Lemma 5. Let f be a convex function defined on the convex set C. Then f is continuous on the relative interior C 0 of C. Lemma 6 (Jensen s inequality). Let f be a convex function defined on a convex set C IR n. Given k points x 1,, x k C and λ 1,, λ k 0 with k i=1 λ i = 1, then: f( k i=1 λ i x i ) k i=1 λ i f(x i ). Lemma 7. Let f 1,, f k be convex functions defined on a convex set C IR n, then the function f(x) = is convex for all λ 1,, λ k 0; the function is convex. k i=1 λ i f i (x) f(x) = max 1 i k f i (x)

14 Convex functions Definition 10. The function h : IR IR is monotonically non-decreasing if h(t 1 ) h(t 2 ) for all t 1 < t 2 IR; strictly monotonically increasing if h(t 1 ) < h(t 2 ) for all t 1 < t 2 IR. Lemma 8. Let f be a convex function on the convex set C IR n and h : IR IR a convex monotonically non-decreasing function, then the composite function h(f(x)) : C IR is convex. Definition 11. Given a convex function f : C IR defined on the convex set C and α IR an arbitrary number, the set D α = {x C f(x) α} is called a level set of the function f. Lemma 9. Given a convex function f on C, the level set D α (possibly empty) convex set for all α IR. is a

15 Convex functions Definition 12. Given x IR n and a direction (vector) s IR n, the directional derivative δf(x, s) of the function f at the point x and in the direction s is defined as: δf(x, s) = lim λ 0 f(x + λs) f(x) λ if the above limit exists. Lemma 10. If the function f is continuously differentiable then, for all s IR n, δf(x, s) = f(x) T s. The Hessian matrix is defined as: ( 2 f(x)) ij = 2 f(x) x i x j for all i, j = 1,, n.

16 Convex functions Lemma 11. Let f be a function defined on a convex set C IR n. The function f is convex if and only if the function φ(λ) = f(x + λs) is convex on the interval [0, 1] for all x C and x + s C. Lemma 12. Let f be a continuously differentiable function on the open convex set C IR n, then the following statements are equivalent: 1. the function f is convex on C, 2. for any two vectors x, x C f(x) T (x x) f(x) f(x) f(x) T (x x), 3. for any x, x + s C the function φ(λ) = f(x + λs) is continuously differentiable on the open interval (0, 1) and φ (λ) = s T f(x + λs) is a monotonically non-decreasing function.

17 Proof of the lemma We first prove that 1 implies 2. Let 0 λ 1 and x, x C. Then the convexity of f implies f(λx + (1 λ)x) λf(x) + (1 λ)f(x). This can be rewritten as f(x + λ(x x)) f(x) λ f(x) f(x). Taking the limit with λ 0 and applying Lemma 10 yields the lefthand-side inequality of 2. Similarly, interchanging the role x and x yields the right-hand-side inequality.

18 Proof of the lemma Now we prove that 2 implies 3. Let x, x + s C and 0 λ 1, λ 2 1. Applying the inequalities of 2 with the points x + λ 1 s and x + λ 2 s yields the following relations: (λ 2 λ 1 ) f(x + λ 1 s) T s f(x + λ 2 s) f(x + λ 1 s) (λ 2 λ 1 ) f(x + λ 2 s) T s, hence (λ 2 λ 1 )φ (λ 1 ) φ(λ 2 ) φ(λ 1 ) (λ 2 λ 1 )φ (λ 2 ). Assuming λ 1 < λ 2 we have φ (λ 1 ) φ(λ2 ) φ(λ 1 ) λ 2 λ 1 φ (λ 2 ) which proves that the function φ (λ) is monotonically non-decreasing. (Need to still show the function φ(λ) = f(x + λs) is continuously differentiable on the open interval (0, 1) and φ (λ) = s T f(x + λs))

19 Proof of the lemma We prove finally that 3 implies 1. We only have to show that φ(λ) is convex if φ (λ) is monotonically non-decreasing. Consider 0 < λ 1 < λ 2 < 1 with φ (λ 1 ) < φ (λ 2 ). Then, for 0 α 1, (1 α)φ(λ 1 ) + αφ(λ 2 ) φ((1 α)λ 1 + αλ 2 ) = α[φ(λ 2 ) φ(λ 1 )] [φ((1 α)λ 1 + αλ 2 ) φ(λ 1 )] = α(λ 2 λ 1 ) 0. 1 ( 1 0 φ (λ 1 + t(λ 2 λ 1 ))dt 0 φ (λ 1 + tα(λ 2 λ 1 ))dt )

20 Illustration of the lemma f(x) f(x) f(x) f(x) f(x) T (x x) f(x) T (x x) Lemma 13. Let f be a twice continuously differentiable function on the open convex set C IR n. The function f is convex if and only if its Hessian 2 f(x) is positive semi-definite (PSD) for all x C. Furthermore, φ (λ) = s T 2 f(x + λs)s.

21 Consider the problem Optimality Unconstrained minimization minimize f(x) where x IR n and f : IR n IR is a differentiable function. define local and global minima of the above problem. We first Definition 13. Given a function f : IR n IR, a point x IR n is a local minimum of the function f if there is an ɛ > 0 such that, for all x IR n, f(x) f(x) if x x ɛ; a point x IR n is a strict local minimum of the function f if there is an ɛ > 0 such that, for all x IR n, f(x) < f(x) if x x ɛ; a point x IR n is a global minimum of the function f if f(x) f(x) for all x IR n ; a point x IR n is a strict global minimum of the function f if f(x) < f(x) for all x IR n.

22 Optimality Unconstrained minimization Lemma 14. Any (strict) local minimum of a convex function is a (strict) global minimum as well. Lemma 15. Let f be continuously differentiable. If the point x IR n is a minimum of the function f then f(x) = 0. Lemma 16. Let f be a continuously differentiable convex function. The point x IR n is a minimum of the function f if and only if f(x) = 0. Lemma 17. Let f be a twice continuously differentiable function. Given a point x IR n, if f(x) = 0 and 2 f(x) is positive semidefinite in an ɛ neighbourhood (ɛ > 0) of x, then x is a local minimum of the function f. Corollary 3. Let f be a twice continuously differentiable function. Given a point x IR n, if the gradient f(x) = 0 and the Hessian 2 f(x) is positive definite, then x is a strict local minimum of the function f.

23 Constrained Optimization Theorem 1. Consider the convex optimization problem min{ f(x) : x C} where C is a relatively open convex set and f is a convex differentiable function. The point x is an optimal solution of this problem if and only if f(x) T s = 0 for all s L where L denotes the linear subspace with aff(c) = x + L for any x C. (Note that aff(c) denotes the affine hull of C). Proof. Let s L. If x is a minimum, then f(x) f(x + λs) if x + λs C. Note that for any s L, x + λs C for a λ sufficiently small, since L is a relatively open set. Bringing f(x) to the right hand side and dividing by λ yields: f(x + λs) f(x) 0. λ Taking the limit as λ 0, we obtain: 0 δf(x, s) = f(x) T s for all s L. As s L is arbitrary, we can conclude that f(x) T s = 0 for all s L. On the other hand, if f is a convex function and f(x) T s = 0 for all s L, then since s = (x x) L. f(x) f(x) f(x) T (x x) = 0

24 Feasible directions Definition 14. The vector s IR n is called a feasible direction at a point x F if there is a λ 0 > 0 such that x + λs F for all 0 λ λ 0. The set of feasible directions at the feasible point x F is denoted by FD(x) x + FD(x) x Lemma 18. For any convex set F and for any x F the set of feasible directions FD(x) is a convex cone. Theorem 2. A feasible point x F is an optimal solution of the convex optimization problem if and only if δf(x, s) 0 for all s FD(x).