Optimization Problems with Constraints - introduction to theory, numerical Methods and applications

Size: px

Start display at page:

Download "Optimization Problems with Constraints - introduction to theory, numerical Methods and applications"

Alexandra Terry
5 years ago
Views:

1 Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)

2 Optimization problems with constraints A general form of a constrained optimization problem is (NLP) min f (x) x s.t. h i (x) = 0, i = 1, 2,..., p; g j (x) 0, j = 1, 2,..., m. where f, g i, h j : R n R are at least one-time differentiable functions and x R n. Feasible set of NLP: x R n is feasible point of NLP if h i (x) = 0, i = 1, 2,..., p and g j (x) 0, j = 1, 2,..., m. The set of all feasible points of NLP is the set which we represent by F := { x R n h i (x) = 0, i = 1,..., p; g j (x) 0, j = 1,..., m }. known as the feasible set of NLP.

3 Optimization problems with constraints... Some times it is convenient to write the constraints of NLP in a compact form as h(x) = 0, g(x) 0, where Example: h 1 (x) g 1 (x) h 2 (x) g 2 (x) h(x) = and g(x) =.. h p (x) g m (x) (NLP1) { } 1 min x 2 x2 1 + x 1x2 2 s.t. x 1 x = 0, x x 2 0, x 2 0.

4 Optimization problems with constraints... In the example problem NLP1 above: there is only one equality constraint h 1 (x) = x 1 x and two inequality constraints g 1 (x) = x x 2 0 and g 2 (x) = x 2 0. Observe that the point x = (1, 1) is a feasible point; while the point (0, 0) is not feasible (or infeasible); i.e., x = (0, 0) does not belong to the feasible set {( ) x1 F = x 2 } R 2 x 1 x2 2 1 = 0, x2 1 + x 2 0, x 2 0. Optimal Solution A point x R n is an optimal solution of the constrained problem NLP if and only if (i) if x is feasible to NLP; i.e. x F and (ii) f (x) f (x ) for all x F.

5 Optimization problems with constraints... For the example problem NLP1, the point x = (1, 1) is an optimal solution. In general, it is not easy to determine an optimal solution of a constrained optimization problem like NLP. Question: (Q1) How do we verify that a give point x R n is an optimal solution of NLP? ( This about optimality criteria. ) (Q2) What methods are available to find an optimal solution of NLP? ( This about computation of optimal solutions. ) Definition (Active Constraints) Let x be a feasible point of NLP. An inequality constraint g i (x) 0 of NLP is said to be active at x if g i (x) = 0. The set A(x) = {i {1, 2,..., m} g i (x) = 0} denotes index set of active constraints at x.

6 Optimization with constraints... Optimality criteria For the example problem NLP1, the constraint g 1 = x x 2 is active at the point x = (1, 1); while g 2 = x 2 is active at x. Hence, A = {1} descent direction A vector d is a descent direction for the objective function f at the point x if f (x + d) f (x) Moving from point x in the direction of d decreases the function f. Any vector d that satisfies d f (x) 0 is a descent direction. To verify this, recall the 1st order Taylor approximation of f at the point x. It follows that f (x + d) f (x) + d f (x). f (x + d) f (x) = d f (x) 0 f (x + d) f (x) 0. Hence, f (x + d) f (x). Therefore, d is a descent direction. NB: If d is a decent direction, then d = αd, α > 0, is also a descent direction.

7 Optimization with constraints... Optimality criteria feasible direction Let x a feasible point (i.e. x F) and d is a vector in R n. If (i) h i (x + d) = 0, i = 1,..., p and (ii) g j (x + d) 0, j = 1,..., m, then d is said to be to be a feasible direction for the NLP. Let x a feasible point. If a vector d satisfies d h i (x) = 0, i = 1,..., m and d g j (x) < 0, j A(x), then d = αd is feasible direction at x, for some α > 0. To verify this,use the 1st order Taylor approximations at the point x. i = 1,..., p : h i (x + αd) h i (x) +α d h i (x); h i (x + d) = 0, }{{} } {{ } =0 =0 j A(x) : g j (x + αd) g j (x) +α d g j (x) g j (x + d) 0; }{{} } {{ } 0 <0 j / A(x) : g j (x + αd) g j (x) +αd g j (x) g j (x + d) 0, for 0 < α g j (x) }{{}}{{} d if d g j (x) > 0. g j (x) non active constraints <0

8 Optimization with constraints... Optimality criteria Optimality condition If x is an optimal solution of NLP, then there is no vector d R n which is both a descent direction for f and feasible direction at x. That is, if x an optimal solution of NLP, then system of inequalities d f (x ) < 0, d h i (x ) = 0, i = 1,..., p; d g j (x ) < 0, j A(x ) equivalently [ f (x )] d > 0, [ h i (x ) ] d = 0, i = 1,..., p; [ g j (x )] d < 0, j A(x ) (1) has no solution d. Farkas Theorem Given any set of vectors c, a i, b j R n, i = 1,..., m; j = 1,..., l. Then exactly only one of the following two systems has a solution System I: c d > 0, ai d = 0, i = 1,..., p; bj d < 0, j = 1,..., m p l System II: c = λ i a i + µ j b j, µ > 0. i=1 j=1 Now if we let c = f (x ), a i = h i (x ), i = 1,..., p and b j = g j (x ), j A(x ), then, since x an optimal point, then only system II has a solution.

9 Optimization with constraints... Optimality criteria Hence, if x is an optimal solution of NLP, then there exist vectors λ R m, λ = ( λ 1, λ 2,..., λm) and µ R l, µ = ( µ 1, µ 2,..., µ l ) > 0 such that m l f (x ) = λ i h i (x ) + µ j g j (x ) i=1 j=1 where l = #A(x ). If we let µ j = 0 for for j {1,..., m} \ A(x ), then we can write f (x ) = m i=1 λ i h i (x ) + m j=1 µ j g j (x ). The Karush-Kuhn-Tucker (KKT) optimality condition If x is a minimum point of NLP, then there is λ R p and µ R m, µ 0 such that the following hold true: m m f (x ) + λ i h i (x ) + µ j g j (x ) = 0 (Optimality) i=1 j=1 h(x ) = 0 (feasibility) g(x ) 0 µ 0 (Nonnegativity) µ j g j (x ) = 0, j = 1,..., m. (Complementarity )

10 Optimization with constraints... Optimality criteria The function L(x, λ, µ) = f (x) + m λ i h i (x) + i=1 m µ j g j (x) j=1 called the Lagrange function associated to the NLP. Example: Solve the following optimization problem: { (NLP2) min f (x) = x 2 x 1 x2 2 } (2) s.t. (3) x 1 + 2x = 0 (4) x 1 x 2 3. (5) Solution: Lagrange function L(x, λ, µ) = (x 2 1 x 2 2 ) + λ(x 1 + 2x 2 + 1) + µ(x 1 x 2 3).

11 Optimization with constraints... Optimality criteria Optimality condition: Feasibility L = 0 2x 1 + λ + µ = 0 x 1 = 1 (λ + µ) x 1 2 (6) L = 0 2x 2 + 2λ µ = 0 x 2 = 1 (2λ µ) x 2 2 (7) h(x) = 0 x 1 + 2x = 0 1 (λ + µ) + (2λ µ) + 1 = 0 2 λ = µ 2 3 Complementarity µg(x) = 0 µ(x 1 x 2 3) = 0 µ ( 12 (λ + µ) 12 ) (2λ µ) 3 = 0 ( µ 1 [(µ 23 ] 2 ) + µ [(µ 23 ] ) ) µ 3 = 0 µ [ µ 2] = 0 µ = 0 or µ = 2. µ = 2 < 0 is not allowed. Hence, µ = 0 is the only possibility. As a result λ = µ 2 3 = 2 3.

12 Optimization with constraints... Optimality criteria Now using µ = 0 and λ = 1 x 1 = 1 2 (λ + µ ) = 1 2 ( ) = 1 3 (8) x 2 = 1 2 (2λ µ ) = 1 2 (2 ( 2 3 ) 0) = 2 3. (9) The point x = ( 1 3, 2 3 ) is a candidate for optimality. Observe that: the inequality constraint g(x) = x 1 x is not active at x. In general, the KKT conditions above are only necessary optimality conditions.

13 Optimization with constraints... Optimality criteria Sufficient optimality conditions Suppose the functions f, h i, i =,..., p; g j, j = 1,..., m are twice differentiable. Let x be a feasible point of NLP. If there are Lagrange multipliers λ and µ 0 such that: (i) the KKT conditions are satisfied at (x, µ, λ ); and (ii) and the hessian of the Lagrangian p m x L(x, λ, µ ) = 2 f (x ) + λ i 2 h j (x ) + µ j 2 g j (x ) i=1 j=1 is positive definite (i.e. d x L(x, λ, µ )d > 0) for d from the subspace S = { d R n d h i (x ) = 0, i = 1,..., p; d g j (x ) = 0, µ j > 0, j A(x ) }, then x is an optimal solution of NLP. For the problem NLP2, since g(x ) < 0 we have ( S = {d R 2 d h(x ) = 0} = {d R 2 1 (d 1, d 2 ) = 0} = {d R 2) 2 d 1 = 2d 2 }. ( ) x L(x, λ, µ 2 0 ) =. Let d 0 2 = (d 1.d 2 ) S, d 0 (note that if d 1 0, then d 2 0 and vice-versa.) ( 2 0 (d 1, d 2 ) 0 2 ) ( d1 ( ) Therefore, x = 13, 2 is an optimal solution of NLP2. 3 ) ( ) ( ) 2 0 2d2 = ( 2d d 2, d 2 ) = 2d d 2 > 0. 2

14 Numerical methods for constrained optimization problems Unless in some trivial cases, solution of a constrained optimization problem by hand is not a trivial task. As a result, currently we find several numerical methods to determine an approximate solution(s) for a constrained optimization problem. Some well known gradient-based methods are (I) The sequential quadratic programming (SQP) method (II) The Interior Point Method (III) Penalty function methods (IV) The Augmented Lagrangian method, etc SQP based algorithms are highly favored for the solution of constrained nonlinear optimization problems. In the SQP method: a sequence of iterates x k+1 = x k + α k d k are generated by solving quadratic optimization (programming) problems to determine the d k s.

15 Quadratic optimization (programming) problems Quadratic optimization problems appear in several engineering applications. Eg: Least square approximations (regression), Model predictive control, in SQP method for NLP, etc. (I) Quadratic programming problems with equality constraints (QP) min x { } f (x) = x Qx + q x (10) s.t. (11) Ax = b, (12) where: Q R n n a symmetric (i.e. Q = Q) positive definite matrix. x R n the (unknown) optimization variable; q R n a given vector. A R m n and b R m are given; i.e. there are m equality constraint. rank(a) = m.

16 Quadratic optimization... The Lagrange function for (QP): Karush-Kuhn-Tucker Conditions: L(x, λ) = x Qx + q x + λ (Ax b). L x = 0 Qx + A λ = q (13) L = 0 Ax = b. (14) λ This implies [ ] [ [ ] Q A x q = A 0 λ] c [ ] Q A If Q is positive definite and rank(a) = m, then is invertible and A 0 [ [ ] x Q A 1 [ ] q = λ] A 0 c x = Q 1 q + Q 1 A ( AQ 1 A ) 1 ( b + AQ 1 q ) ( λ = AQ 1 A ) 1 ( b + AQ 1 q ).

17 Quadratic optimization (programming) problems Example: A material flow from two reservoirs R1 and R2 flows into a third reservoir R3. The rate of flow from R1 is measured to be F1 M = 6.5m 3 /h and from R2 is F2 M = 14.7m 3 /h. As a result of these two inflows, the volume in R3 is measured to change at a rate F3 M = 19.8m 3 /h. Flow rate measurement devices for each reservoirs are known to incur measurement errors each with standard-deviation: σ 1 = 0.1m 3 /h, σ 2 = 0.2m 3 /h and σ 3 = 0.3m 3 /h. State an optimization problem to determine the actual flow rates. Solution: Let F 1, F 2 and F 3 be the true flow rates. Note that F i = Fi M ± η i σ i, i = 1, 2, 3; measurement ±η i σ i = F i Fi M represents measurement error. To minimize all these measurement error {( F1 F1 M min (F 1,F 2,F 3 ) σ 1 s.t. F 1 + F 2 = F 3. ) ( F2 F2 M + σ 22 ) ( F3 F3 M + σ 3 )}

18 Quadratic optimization (programming) problems (II) Quadratic programming problems with equality and inequality constraints (QP) { } min x f (x) = x Qx + q x (15) s.t. (16) a i x = b i, i = 1, 2,..., p (17) a i x b i, i = p + 1,..., m. (18) where: Q is symmetric positive definite. If we set A = [a 1, a 2,..., a p], a = (b 1, b 2,..., b p), B = [a p+1, a p+2,..., a m] and b = (b p+1, b p+2,..., b m) Then (QP) can be compactly written as: (QP) { } min x f (x) = x Qx + q x (19) s.t. (20) Ax = a (21) Bx b. (22) In general, inequality constrained (QP)s may not be easy to be solved analytically.

19 Quadratic optimization... Active set Method Active Set Method Strategy: Start from an arbitrary point x 0 Find the next iterate by setting x k+1 = x k + α k d k, where α k is a step-length and d k is search direction. Question How to determine the search direction d k? How to determine the step-length α k? (A) Determination of the search direction: At the current iterate x k determine the index set of active constraints A k = {i a i x k b i = 0, i = p + 1,..., m} {1, 2,..., p}.

20 Quadratic optimization... Active set Method Active Set Method Strategy: Start from an arbitrary point x 0 Find the next iterate by setting x k+1 = x k + α k d k, where α k is a step-length and d k is search direction. Question How to determine the search direction d k? How to determine the step-length α k? (A) Determination of the search direction: At the current iterate x k determine the index set of active constraints A k = {i a i x k b i = 0, i = p + 1,..., m} {1, 2,..., p}.

21 Quadratic optimization... Active set Method Solve the direction finding problem { } 1 min d 2 (xk + d) Q(x k + d) + q (x k + d) s.t. a i (x k + d) = b i, i A k. Expand 2 1 (xk + d) Q(x k + d) + q (x k + d) = 1 2 d Qd d Qx k (xk ) Qd (xk ) Qx k + q d + q x k ( ) a i x k + d = b i a i d = b i ai x k = 0. Simplify these expressions and drop constants to obtain: { 1 [ ] } min d 2 d Q + d + Qx k + q d s.t. a i d = 0, i A k. Set a 1 A = a2, i Ak.. Solve this problem using the problem using the KKT conditions to obtain d k.

22 Quadratic optimization... Active set Method (A) Determination of the step length: Once you obtained d k compute α k as α k = min [1, b i ai x k ] ai d k, i / A k

23 Quadratic optimization... Active set Method Example: Application Problem A parallel flow heat exchanger of a given length to be designed. The conducting tubes, all of the same diameter, are enclosed in an outer shell of diameter D = 1m (see Fugure). The diameter of each of the conducting tubes should not exceed 60mm. Determine the number of tubes and the diameter for the largest surface area.

Constrained Optimization

1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange