Chapter 11 Optimization with Equality Constraints

Transcription

1 Ch. - Optimization with Equalit Constraints Chapter Optimization with Equalit Constraints Albert William Tucker arold William Kuhn 95 oseph-ouis Giuseppe odovico comte de arane 76-. General roblem Now a constraint is added to the optimization problem: ma u s.t p + p = I where p p are eoenous prices and I is eoenous income. Different methods to solve this problem: Substitution Total differential approach arane Multiplier

2 Ch. - Optimization with Equalit Constraints. Substitution Approach Eas to use in simple sstems. sin the constraint substitute into objective function and optimize as usual. Eample: Solve for Substitutin into F.o.c.: Check s.o.c.: d d d d ; maimum ; Calculate maimum Value for. : s.t. 6 and ;. Total Differential Approach Total differentiation of objective function and constraints: f f f ; s.t. B d f d f d ; f d d f d ; d d f f ; ; db d d f d db d d d f Equation alon with the restriction form the basis to solve this optimization problem.

3 Ch. - Optimization with Equalit Constraints. Total Differential Approach Eample: = + s.t. 6= + Takin first-order differentials of and budet constraint B: d d d 6 ; d d ; d d d ; db d d 5. Total-differential approach Graph for tilit function and budet constraint: 6

4 Ch. - Optimization with Equalit Constraints 7 ' ' ' B f f. arane-multiplier Approach To avoid workin with possibl zero denominators let denote the common value in. Rewritin and addin the budet constraint we are left with a sstem of equations: There is a convenient function that produces and as a set of f.o.c.: The aranian function which includes the objective function and the constraint: The constraint is multiplied b a variable called the arane multiplier M. B f 7. arane-multiplier Approach 5 f f B B f Once we form the aranian function the arane function becomes the new objective function.

5 Ch. - Optimization with Equalit Constraints 5 9. M Approach 5 Note that = = =. Then If the constraints are linear the essian of the aranian can be seen as the essian of the oriinal objective function bordered b the first derivatives of the constraints. This new essian is called bordered essian. B B B constraint budet the Subject to where tilit Maimize. M Approach: Eample

6 Ch. - Optimization with Equalit Constraints. M Approach: Interpretation From B f.o.c. B B Note: From the f.o.c. at the optimal point: - = marinal utilit of ood i scaled b its price. - Scaled marinal utilities are equal across oods. - The ratio of marinal utilities MRS equals relative prices.. M Approach: Interpretation Now we total differentiate. We et similar results d d B d d Budet B d marinal r ate of substituti on MRS constraint and budet d d line Note: Aain at the optimal point the ratio of marinal utilities MRS is equal to the ratio of prices. 6

7 Ch. - Optimization with Equalit Constraints. M Approach: Second-order conditions has no effect on the value of because the constraint equals zero but A new set of second-order conditions are needed The constraint chanes the criterion for a relative ma. or min. 5 a a α h h iff solve b α β the s. t. a β h b constraint b β α β a β h b for. M Approach: Second-order conditions aβ α β h bα a h - aβ α β h - bα h b is positive definite s.t. iff a h min h b 7

8 Ch. - Optimization with Equalit Constraints ; ; 7 5 ; 6 ; 6 ; 9 ; 7 ; Form the aranianfunction 6 B s.t. - FOC. M Approach: Eample 5 maimum is definite neative is ; 6 ; conditions order 6 function aranian 6 constraint Budet 6 B function tilit d st. M Approach: Eample - Cramer s rule 6

9 Ch. - Optimization with Equalit Constraints M Approach: Eample - Cramer s rule 7. M Approach: Restricted east Squares The aranean approach f.o.c: Then from the st equation: q r Min t t T t q rb q rb r b r b t t t t t t T t T t r b r b r b ' ' ' ' ' '

10 Ch. - Optimization with Equalit Constraints. M Approach: Restricted east Squares b =b r - remultipl both sides b r and then subtract q rb - q = rb r - - q = - r - + rb q Solvin for = [r - ] - rb - q Substitutin in b b= b - - r [r - ] - rb - q 9. M Approach: N-variable Case a b c d Where No one - variable variable test of soc variable test of soc n variable Z Z Z Z case soc p. 6 ; test because n there must be one more Z n Z Z n Z Z Z neative definite positive definite neative definite positive definite neative definite positive definite Z ; Z Z variable than constraint : : : : : : ma min ma min ma min

11 Ch. - Optimization with Equalit Constraints. Optimalit Conditions nconstrained Case et be the point that we think is the minimum for f. Necessar condition for optimalit: df = A point that satisfies the necessar condition is a stationar point It can be a minimum maimum or saddle point Q: ow do we know that we have a minimum? Answer: Sufficienc Condition: The sufficient conditions for to be a strict local minimum are: df = d f is positive definite. Constrained Case KKT Conditions To prove a claim of optimalit in constrained minimization or maimization we have to check the found point with respect to the Karesh Kuhn Tucker KKT conditions. Kuhn and Tucker etended the aranian theor to include the eneral classical sinle-objective nonlinear prorammin problem: minimize f Subject to j for j =... h k = for k =... K =... N

12 Ch. - Optimization with Equalit Constraints. Interior versus Eterior Solutions Interior: If no constraints are active and thus the solution lies at the interior of the feasible space then the necessar condition for optimalit is same as for unconstrained case: f = difference operator for matrices -- del Eample: Minimize f=- +- with constraints: - & -. Eterior: If solution lies at the eterior the condition f = does not appl because some constraints will block movement to this minimum.. Interior versus Eterior Solutions Eterior: If solution lies at the eterior the condition f = does not appl. Some constraints will block movement to this minimum. Some constraints are active. Eample: Minimize f=- +- with constraints: + ; -& -. We cannot et an more improvement if for there does not eist a vector d that is both a descent direction and a feasible direction. In other words: the possible feasible directions do not intersect the possible descent directions at all.

13 Ch. - Optimization with Equalit Constraints. Mathematical Form A vector d that is both descendin and feasible cannot eist if -f = i i with i for all active constraints ii. This can be rewritten as = f + i i This condition is correct IF feasibilit is defined as. If feasibilit is defined as then this becomes -f = i - i Aain this onl applies for the active constraints. suall the inactive constraints are included but the condition j j = with j is added for all inactive constraints j. This is referred to as the complimentar slackness condition. 5. Mathematical Form Note that the slackness condition is equivalent to statin that j = for inactive constraints -i.e. zero price for non-bindin constraints! That is each inequalit constraint is either active and in this case it turns into equalit constraint of the arane tpe or inactive and in this case it is void and does not constrains the solution. Note that I+ = m the total number of inequalit constraints. Analsis of the constraints can help to rule out some combinations. owever in eneral a brute force approach in a problem with inequalit constraints must be divided into cases. Each case must be solved independentl for a minima and the obtained solution if an must be checked to compl with the constrains. A lot of work!

14 Ch. - Optimization with Equalit Constraints. Necessar KKT Conditions For the problem: Min f s.t. n variables m constraints The necessar conditions are: f + i i = optimalit i for i =... m primar feasibilit i i = for i =... m complementar slackness i for i =... m non-neativit dual feasibilit Note that the first condition ives n equations. 7. Necessar KKT Conditions - Eample Eample: et s minimize f = - +- with constraints: + ; -& -. Form the aranian: f m k p k h k j j j There are inequalit constraints each can be chosen active/nonactive: possible combinations. But the constraints toether: += & =- & =- have no solution and a combination of an two of them ields a sinle intersection point.

15 Ch. - Optimization with Equalit Constraints. Necessar KKT Conditions - Eample The eneral case is: 5 6 and 7 We must consider all the combinations of active / non active constraints: and and and and nconstrained:. Necessar KKT Conditions - Eample 6 f f..6 5

16 Ch. - Optimization with Equalit Constraints. Necessar KKT Conditions - Eample 6 ; f 7 f ; f 5 ; f 5 7 ; f - beond the rane. Necessar KKT Conditions - Eample Finall we compare amon the cases we have studied: case 7 resulted was over-constrained and had no solutions case violated the constraint +. Amon the cased -6 it was case.. ; f. ieldin the lowest value of f. 6

17 Ch. - Optimization with Equalit Constraints. Necessar KKT Conditions: General Case For the eneral case n variables M Inequalities equalities: Min f s.t. i for i =... M h j = for =... In all this the assumption is that j for j belonin to active constraints and h k for k =...K are linearl independent. This is referred to as constraint qualification. The necessar conditions are: f + i i + j h j = optimalit i for i =... M primar feasibilit h j = for j =... primar feasibilit i i = for i =... M complementar slackness i for i =... M non-neativit dual feasibilit Note: j is unrestricted in sin. Necessar KKT Conditions if If the definition of feasibilit chanes the optimalit and feasibilit conditions chane. The necessar conditions become: f - i i + j h j = optimalit i for i =... M feasibilit h j = for j =... feasibilit i i = for i =... M complementar slackness i for i =... M non-neativit dual feasibilit 7

18 Ch. - Optimization with Equalit Constraints. Restatin the Optimization roblem Kuhn Tucker Optimization roblem: Find vectors N M and K that satisf: f + i i + j h j = optimalit i for i =... M feasibilit h j = for j =... feasibilit i i = for i =... M complementar slackness condition i for i =... M non-neativit If is an optimal solution to N then there eists a such that solves the KuhnTucker problem. The above equations not onl ive the necessar conditions for optimalit but also provide a wa of findin the optimal point. 5. KKT Conditions: imitations Necessit theorem helps identif points that are not optimal. A point is not optimal if it does not satisf the KuhnTucker conditions. On the other hand not all points that satisf the Kuhn-Tucker conditions are optimal points. The KuhnTucker sufficienc theorem ives conditions under which a point becomes an optimal solution to a sinle-objective N. 6

19 Ch. - Optimization with Equalit Constraints. KKT Conditions: Sufficienc Condition Sufficient conditions that a point is a strict local minimum of the classical sinle objective N problem where f j and h k are twice differentiable functions are that The necessar KKT conditions are met. The essian matri = f + i i + j h j is positive definite on a subspace of R n as defined b the condition: T is met for ever vector N satisfin: j = for j belonin to I = { j j = u j > } active constraints h k = for k =... K 7. KKT Sufficienc Theorem Special Case Consider the classical sinle objective N problem. minimize f Subject to j for j =... h k = for k =... K =... N et the objective function f be conve the inequalit constraints j be all conve functions for j =... and the equalit constraints h k for k =... K be linear. If this is true then the necessar KKT conditions are also sufficient. Therefore in this case if there eists a solution that satisfies the KKT necessar conditions then is an optimal solution to the N problem. In fact it is a lobal optimum. 9

20 Ch. - Optimization with Equalit Constraints. KKT Conditions: Closin Remarks Kuhn-Tucker Conditions are an etension of aranian function and method. The provide powerful means to verif solutions But there are limitations Sufficienc conditions are difficult to verif. ractical problems do not have required nice properties. For eample ou will have a problems if ou do not know the eplicit constraint equations. If ou have a multi-objective leicoraphic formulation then I would suest testin each priorit level separatel. Minimize Form aranian : F.o.c.:. KKT Conditions: Eample Case : et From and From Case : From and From et C s.t. 6; ; ; 6 - ; ; ; / 9 / 6 nd Constraint inactive : ; st Constraint inactive : 5 6 / / / 9 / / ; 6 / Violates KKT conditions ; Meets KKT conditions 6

21 Ch. - Optimization with Equalit Constraints minimum. do not meet thekkt conditionsfor a and to solvefor se eq.s & 6-. & therefore Assume. KKT Conditions: sin Cramer s rule minimum. meet the KKT conditions for a and to solve for & se eq.s. & therefore & Assume. KKT Conditions: Eample

22 Ch. - Optimization with Equalit Constraints.5 The Constraint Qualification Reason: on an inflectionpoint or cusp it shouldequalzero. when at a ma.owever - - & and - - subject to Maimize Eample points ties at boundar Irreulari & where and - - subject to Maimize Eample boundar points at the Irreularities.5 The Constraint Qualification

23 Ch. - Optimization with Equalit Constraints.5 The Constraint Qualification Eample The feasible reion of the problem contains no cusp Maimize π - subject to and where when it should equal zero 5