1 Convexity, Convex Relaxations, and Global Optimization

Transcription

1 1 Conveity, Conve Relaations, and Global Optimization Algorithms Minima Consider the nonlinear optimization problem in a Euclidean space: where the feasible region. min f(), R n Definition (global minimum). A point is a global minimum of f if ynonyms: problem). f() f( ). optimal solution, global optimal solution, or solution (to the optimization Definition (local minimum). A point is a local minimum if there eists an ε > 0 f() f( ) < ε. Remark. Trivially, global minimum local minimum. f() f() local and global minimum local min. (not global) global min. What is so special about the function on the left that the local minimum is also the global minimum? CONVEXITY! The property of conveity is so important because its definition implies that every local minimum is also a global minimum. 1 c PIB 1/2/02 1

2 1.2 Conve sets Definition (conve set). A set R n is conve if for every, y, λ + (1 λ)y λ [0, 1]. Remark. In words, every point on the straight line connecting any pair of points in must also be in. 2 2 y y Conve et 1 Nonconve et 1 N.B. Must hold for all pairs of points in the set. The intersection of two conve sets is conve. number of conve sets is conve. By induction, the intersection of a finite 1.3 Conve functions Definition (conve function). Let f : R, where is a nonempty conve set in R n. The function is said to be conve on if for each, y and for each λ (0, 1). f(λ + (1 λ)y) λf() + (1 λ)f(y) Geometrically: Function always lies below straight line connecting any two points. 2

3 λf() + (1 λ)f(y) f(z) y [ ] f(λ + (1 λ)y) Conve Function z Must be satisfied for all pairs of points in. f(z) [ y ] z Nonconve Function 1 Conveity depends on the set. nonconve on. In the above eample, the function is conve on 1 but 3

4 Lemma. Let be a nonempty conve set in R n and let g : R be a conve function. Then, the set { : g() α, α R} is conve. 1.4 Conve Optimization Theorem. Let be a nonempty conve set in R n and let f : R be conve on. Consider the problem to minimize f() subject to. uppose is a local minimum. Then is a global minimum. Proof. By hypothesis, is a local minimum of f. Thus, by definition, there eists an ε > 0 f() f( ) B ε ( ) where B ε ( ) is the open ball defined by B ε ( ) = { : < ε}. Assume the contrary. That is, suppose that is not a global minimum. This implies the eistence of an ˆ f(ˆ) < f( ). (1) Choose λ (0, 1) sufficiently small λˆ + (1 λ) B ε ( ). The eistence of such a λ is clear by the Archimedean property of the real numbers. By the conveity of f and the Inequality (1), we have f(λˆ + (1 λ) ) λf(ˆ) + (1 λ)f( ) < λf( ) + (1 λ)f( ) = f( ). However, this contradicts that is a local minimum of f. In other words, for conve optimization (minimization of a conve function on a conve set): Why is this useful? local minimum global minimum. Eample. Consider the unconstrained minimization: From elementary calculus, we know: min f(). =R n Theorem. uppose that f : R n R is differentiable at. If is a local minimum, then f( ) = f( ) = 0. 4

5 Remark. f() = 0 are known as stationary points. Assume that we have an algorithm that can reliably find points f ( ) = 0. Differentiable conve functions satisfy the following theorem. Theorem. Let be a nonempty open conve set in R n, and let f : R be differentiable on. Then f is conve on if and only if for any ˆ we have f() f(ˆ) + f(ˆ) T ( ˆ). Now suppose f() is differentiable and conve on = R n. If f( ) = 0 then f() f( ). Or, is a global minimum on. Check: R n is open and conve, so satisfies the hypotheses. This illustrates the general strategy: For conve optimization problems, if we have a reliable procedure for locating local minima, then we always get the global minimum. In fact, an infallible procedure to locate local minima of conve optimization problems appears very difficult to implement in practice, even if algorithms eist that promise this property theoretically. N.B. Might search for local minima directly, not stationary points. 1.5 Nonconve Optimization f() stationary point local minimum global minimum For nonconve optimization problems, the strategy of finding local minima or stationary points fails. In the figure, 5

6 All three points indicated are stationary points, one of which is actually a local maimum. Of the two local minima, only one is a global minimum. A simple descent strategy started to the left of the maimum will most likely locate the suboptimal local minimum. A deterministic global optimization algorithm is designed to guarantee locating the global minimum objective function value within some ε tolerance with a finite number of iterations. There are many approaches to the design of deterministic global optimization algorithms but here we will only discuss the branch-and-bound approach (B&B). B&B relies on the notion of a conve relaation of a nonconve function. Definition (conve relaation). Let f : R where R n is a nonempty conve set. Then a conve function u : R is a conve relaation of f if u() f(). f() u() [ ] Definition (conve envelope). Let f : R where R n is a nonempty conve set. The conve envelope of f over (denoted f ) is a conve relaation for any other conve relaation u of f on, we have f () u(). Remark. The conve envelope is the tightest possible conve relaation of a nonconve function. For a univariate concave function, the conve envelope is the secant joining the end points of the set, as shown in the figure. 6

7 f() f () [ ] The conve envelopes of many functions are known. However, in general, finding the conve envelope of an arbitrary function is as hard as finding the global minimum. On the other hand, a number of polynomial algorithms eist for constructing conve relaations of quite general classes of functions. uppose we have the nonconve optimization problem min f() g() 0 where R n is a nonempty conve compact set, f : R and g : R m are continuous and potentially nonconve. We can construct a conve optimization problem that is a relaation of this problem via conve relaations u and h of f and g respectively on : min u() h() 0. N.B. The set { : h() 0} is conve, so every local minimum will also be a global minimum. Consider the minimum of the nonconve and conve problems, and ˆ, respectively. Both and ˆ will be feasible for the conve problem because h( ) g( ) 0. 7

8 Moreover, u(ˆ) u( ) f( ). i.e., the minimum for the conve problem is a lower bound on the minimum of the nonconve problem. Hence, the term conve underestimating problem is frequently used. These preliminaries enable us to describe the B&B procedure for nonconve optimization. f() UBD LBD u() [ ] Partition the space : f() UBD 1 LBD 1 UBD 2 LBD 2 u 2 () u 1 () [ ][ ] 1 2 8

9 Now, because LBD 1 UBD 2, the minimum cannot be attained in the set 1. o, 1 is ecluded from further considerations, or fathomed. 2 is further partitioned, fathoming as necessary until LBD on all sets comes within ε of the UBD. The partitioning procedure can be illustrated by the following branch-and-bound tree: etc. The labels denote the partition over which the LBD and UBD are generated. In order to have finite termination for ε tolerance, it is also necessary to have convergence of the conve relaation to the nonconve function as the size of the partition tends toward zero. 1.6 Constructing conve relaations using AD Eample. Consider the following nonconve function: f() = 1 [ 2 3 ( which can be represented by the following binary tree: etc. )] 1/3 / w 5 1 w 4 w w 2 3 / w

10 or the elementary operation list: w 1 = ( )/2 w 2 = 3 w 1 w 3 = 2 w 2 w 4 = w 1/3 3 w 5 = 1 /w 4 These representations indicate that the optimization problem min f() X can be reformulated as min X,w W w 5 w 1 = ( )/2 w 2 = 3 w 1 w 3 = 2 w 2 w 4 = w 1/3 3 w 5 = 1 /w 4 A conve relaation of this problem may be constructed via the known conve and concave envelopes of the RH of the equality constraints. For eample: linear constraints such as always define a conve set. w 1 = ( )/2 The conve and concave envelopes for any bilinear term such as 3 w 1 leading to the following relaation: are known, w 2 L 3 w w1 L L 3 w1 L w 2 U 3 w w1 U U 3 w1 U w 2 L 3 w w1 U L 3 w1 U w 2 U 3 w w1 L U 3 w1 L where L 3, U 3 are lower and upper bounds on 3 in the partition of interest, and w1 L and w1 U can be inferred by interval analysis from the elementary functions and the bounds on the i and other w i 10