NONLINEAR OPTIMIZATION WITH CONVEX CONSTRAINTS. The Goldstein-Levitin-Polyak algorithm

Transcription

1 - (23) NLP - NONLINEAR OPTIMIZATION WITH CONVEX CONSTRAINTS The Goldstein-Levitin-Polya algorithm We consider an algorithm for solving the otimization roblem under convex constraints. Although the convexity of the constraints is treated in its generality, in ractice, convex constraints imly linear inequalities, esecially uer and lower bounds on the decision variables. Aart from convexity, the main assumtion on the constraints is that they are once differentiable. The method belongs to the owerful class of algorithms develoed by Goldstein- Levitin-Polya. It requires that the constraints are satisfied at every iteration. This is an inconvenience in general. Yet, for linear constraints and for secial constraint structures, it is simle to maintain feasibility. Two imortant advantage that seems to be offered in return for satisfying the constraints are the strong results concerning unit stesize achievement and suerlinear convergence. With regard to the latter, it is established that the necessary and sufficient condition for Q-suerlinear rate of convergence is the two sided rojected Hessian condition which, in other algorithms, can only ensure lesser Q-suerlinear rates..the PROBLEM Consider min š Y Ðx Ñ ¹ h (x) Ÿ, (.) n n i Y : Ä, the elements of h: Ä are differentiable functions. The feasible region n e = š x ¹ h (x) Ÿ (.2) is assumed to be convex and Y (x) is assumed to be bounded below on e. The convexity of the constraints allows the develoment of a secial class algorithm. In articular, we use the fact that if two oints belong to the convex set described by the constraints, then any oint on the line segment joining the two oints is also in the convex set. The oint chosen by the algorithm on the line segment is the oint that corresonds to an imroved value of the objective function. The method we describe for solving (.) under this convexity assumtion is a generalisation of the Goldstein-Levitin-Polya (GLP) algorithm (Goldstein, 964; Levitin and Polya, 966). The basic GLP algorithm consists of the iterative scheme x + = PeŠ x 7 d where d is a descent direction such as the steeest descent direction, fy (x ). 7 is the stesize and P e is the rojection of Š x 7 d onto the feasible region e.

2 - (23) NLP 2-2. THE ALGORITHM Let the quadratic aroximation be defined by q (x) = f Y (x ), x - x + x - x, d (x - x ) (2.) where d is a symmetric ositive definite aroximation to the Hessian d of Y(x) at x. The assumtion that d is ositive definite is not restrictive since it does not imair the convergence roerties of the algorithm even when the true Hessian is not strictly ositive definite. It will be shown in Section 4 that the convergence of the stesizes, discussed below, to unity and that the Q-suerlinear convergence rate deends on the accuracy of the rojection of d onto the tangent manifold of the constraints. The algorithm is based on the rojections of the unconstrained ste - x - = x! d f Y (x ), =,, 2,... (2.2) with,! [, -], -!! [, 2), onto the feasible region R using subroblem min š l x - x - 2 l ¹ x e (2.3, a) where ² v ² = v, d v. We note that for the choice =, reduces to the solution of 2 d d min š q (x) ¹ x e. (2.3, b) The lectures are mainly concerned with the choice! = and hence (2.3, b) but the results below are stated for the general case for! [, -!]. If x is the value of x that solves (2.3, a or b), then x + is comuted using + x = x + 7 (x - x ) (2.4) with 7 [, ] given by the smallest value of j =,, 2, 3,..., 7 = ( ), (, ), satisfying + Y(x ) Y (x ) Ÿ 7 3 q (x ), 3 (, ). (2.5) with 3 an arbitrary number in the above range. An alternative to (2.5) is the Armijo-tye stesize strategy Y(x ) Y (x ) Ÿ 7 3 fy (x ), x x, 3 Š,. (2.6) with 3 2 an arbitrary number in the above range. The matrix d is aroximated using Powell's (978, b) modification to a quasi- Newton formula due to Broyden (969; 97), Fletcher (97), Goldfarb (97), Shanno! j!-

3 - (23) NLP 3 - (97) for aroximating the Hessian of a Lagrangian. This modified BFGS (Broyden- Fletcher-Goldfarb-Shanno) formula is given by where d = d d $ x $ x T d T / / +. (2.8,a) + $ T d $ x x $ T / = fy(x ) f Y (x ); $ = (x x ) (2.8,b) + x + x / = ) + ( )) d $ (2.8,c) x T T if $.2 $ d $ x x x ) =. (2.8,d).8 $ x T d $ x $ T T x d $ x $ x if $ T.2 $ T d $ x x x For linearly constrained e and ositive definite d, (2.3) is a ositive definite quadratic rogramming roblem with a unique solution x. In the initial stages of the algorithm,! rovides the otion for choosing large stes. As discussed in Sections 3 and 4 below,! does not affect the convergence roerties of the algorithm, rovided it is chosen the range [, -!], with the sequence Ö! converging to unity. An examle is! = 2 7. It is shown below that 7, a. Also, if 7 = for!, then reducing! + to unity maes it easier to maintain unit. 7 EXERCISE: Write in seudocode the two versions of the GLP algorithm discussed in this section. Do a library search for suitable termination rules for the algorithm and incororate them in the seudocode. 3. CONVERGENCE EXERCISE: Assuming e is a system of linear inequalities, establish the descent roerty of the direction generated by the algorithm. (The descent roerty discussed below is for general convex feasible sets, which clearly include systems of inequalities. There is a direct way of establishing descent for linear inequalities.) In this section the stesize strategies (2.5) and (2.6) are justified for symmetric ositive definite and d and {! } satisfying the restrictions discussed in Section 2. It is shown that the algorithm is globally convergent. The main theorem of this section is (3.) below which justifies (2.5)-(2.6) and establishes the existence of 7 (, ] which satisfies the stesize rules. Most subsequent results are deendent on Theorem 3.. Lemma 3. Let! (, ), _ d be a bounded symmetric ositive definite matrix and x be given by (2.2). Then, x solving (2.3) satisfies

4 - (23) NLP 4-2 l x - x l Ÿ -! fy (x ), x x. (3.) d REMARK: If you have already done the exercise at the beginning of this section, you now this result holds for linear inequality constraints (for! =, thence (2.3, b)). Hence, you can ignore the roof. Since (2.3) is the rojection of x - on the convex region e, we have the inequality x -, d (x - x) - Ÿ (3.2) for any e (see Rustem, 998; Lemma 3..2). Thus, we have - 2 lx x l x x, Š x + Ÿ d! d fy (x ) x! fy (x ), x x. d The result follows as the first term on the right may be formulated as (3.2), for = x. Lemma 3.2 Let 7 [, ], d be symmetric ositive definite and x be given by (2.4). Then + q (x +) Ÿ 7 q (x ) (3.3) 2 q(x ) (x), x - x + x - x, + œ 7 fy 7 d (x - x) Ÿ 7 q(x). Lemma 3.3 Let, (, -], - (, 2),! -!!! d be bounded symmetric ositive definite and x be given by (2.2). Then, x solving (2.3) satisfies q (x ) Ÿ Š lx - x l Ÿ (3.4) 2! d! - d l x - x l 2 Ÿ q (x ). (3.5)! -!- Lemma 3., yields (3.4) immediately. Also, (3.5) holds for = Š. Theorem 3. Let n 2 (i) e be a convex set and Y (x), (ii) d in (2.)-(2.3) satisfy 2 m, 2 n ² ² Ÿ d Ÿ M ² ², a, Á, _ M m, (iii) {! } be any sequence converging to unity, with [, - ], -!!! [, 2).

5 - (23) NLP 5 - Then there exists a 7 (, ] that satisfies (2.5) or (2.6) and hence the sequence {x } comuted by (2.4) generates a monotonically decreasing sequence { Y(x )}. IF YOU UNDERSTAND THE DESCENT PROPERTY OF LEMMA 3. THE PROOF IS EASY TO ESTABLISH. Remar. Second order Taylor series exansion of a function f : n Ä, and f 2, for any x, d 8, ), is given by f (x ) d) œ f (x) + ) f f(x), d 2 2 ) Š ' ( t) d, f f (x + t ) d) (d) dt. Using the second order Taylor series exansion, Y(x ) Y(x ) Y(x ), x x x x, d + œ f + + Š x + x dt ' ( t) x x, (x (t) + š d Š d Š x + x (3.7) 2 2 Ÿ 7 q (x ) 7 lx x l (3.8) where x (t) œ x t (x x +), d(.) is the Hessian of Y at (.), = ' ( - t) ² d(x (t) d ² dt, and (3.3) is alied to obtain (3.8). From (3.5) and (3.8), we have Y (x ) Y (x ) Ÿ 7 q (x ) Š (3.9) + Since 3, there is a 7 (, ] such that 7 Ÿ. (3.) 3 m By Lemma (3.3), q (x ) Ÿ. Thus, (2.5) holds for this 7. Suose 7 is the largest 7 Ò!,! Ó satisfying inequality (2.5). All 7 Ÿ 7 also satisfy this condition and the selected 7!! Ò7, 7 Ó. It follows that š Y(x ) is monotonically decreasing. -! 7 m -!! yield To show that š Y (x ) is monotonically decreasing for (2.6), we use (3.) with (3.7) to 7! 7! + 2 m Y (x ) Y (x ) Ÿ 7 f Y(x ), x x Š

6 - (23) NLP 6 -!- 2 2 Since 3 Ÿ, there is a 7 (, ] such that (3.) 7! 7!! m 2 Ÿ Ÿ. (3.2) By Lemma (3.), fy(x ), x x Ÿ. Thus, (2.6) holds for this 7. Suose 7 is the largest 7 Ò!, Ó satisfying inequality (2.6). All 7 Ÿ 7! also satisfy this condition!! and the selected 7 Ò7, 7 Ó. It follows that š Y(x ) is monotonically decreasing. THEOREM 3.2 BELOW IS AN EXAMPLE FOR THE USE OF THE BOLZANO-WEIERSTRASS THEOREM. YOU ARE NOT RESPONSIBLE FOR ANY PROOFS FROM THIS POINT ON IN THIS SET OF NOTES. Theorem 3.2 Let (i) the assumtions of Theorem 3. be satisfied, (ii) the set = š x e ¹ Y (x) Ÿ Y (x ) be bounded.! and Then, we have Ä_ Ä_ Y q (x ) œ (3.3) f (x ), x x =. (3.4)! Given 3, by (3.9), the choice 7 = min š, - always satisfies m ( - 3 ) the stesize strategy (2.5). Clearly, 7, chosen as 7 œ j, as discussed above, is in the range!! 2 7 Ò7, 7 Ó and thereby also satisfies (2.5). As Y (x), and is comact, there is a scalar M - such that M. - -! _ Ÿ Similarly, given 32, (2.6) is satisfied by - 7 = min, ( 3 )!- š Š Š +. As m,!-, -, we have established that there 2 m is an % such that the stesizes determined by (2.5) or (2.6) satisfy 7 %, a. The boundedness of Y on and q (x ) Ÿ, imly, in the case of (2.5), that Ÿ 3! 7 ± q (x ) ± Ÿ! Y (x ) Y (x ) _. + Since 7 %, this yields (3.3). (3.4) is established similarly by using (2.6) and f Y(x ), x x Ÿ. Lemma 3.4 If (3.3) or if (3.4) are satisfied, then x x. (3.5) Ä_ The result follows from Lemmas (3.3) and (3.4) for (3.3) and (3.4) resectively. Theorem 3.3!

7 - (23) NLP 7 - Let (i) the assumtions of Theorem 3. be satisfied, (ii) the assumtions of Theorem 3.2 be satisfied, (iii) h (x) be once differentiable, (iv) the active constraint gradients at x are linearly indeendent Šalternatively, instead of the linear indeendence condition, it can be assumed that the multiliers associated with (2.3) at x are bounded (Fiacco and McCormic, 968) Then, (a) the algorithm in Section 2, with stesize strategy (2.5) or (2.6), generates a sequence {x } that converges to x, and (b) if, furthermore, strict comlementarity holds at the solution of subroblem (2.3), for large, redicts the active inequality constraints at x.. By Theorem 3., the algorithm ensures the decrease of Y (x) at each iteration, thereby ensuring x, for comact. Hence, { x } has a it oint, x. Without loss of generality, we can tae {x } Ä x. To show that x also satisfies the necessary conditions for otimality for roblem (.), we consider the otimality of subroblem (2.3). Let. + and f h denote resectively the multiliers of the subroblem and the matrix whose columns are the constraint gradients at x. Using (2.2) and (2.3), the conditions for (2.3) are given by f (x ) Y d (x x ) f h. œ (3.6, a) +! + + h (x ) Ÿ ; h (x ),. = ;.. (3.6, b) Ä_ Ä_ Using (3.5), we have x œ x œ x and f Y (x ) f h. œ. To show (b) we note that, in view of the last two conditions in (3.6, b), for sufficiently large and with strict comlementarity holding, none of the inactive constraints, i.e. j j h ;. œ (3.) + j j are redicted to be active at x, h (x ),. œ. 4. UNIT STEPSIZES AND SUPERLINEAR CONVERGENCE RATES We consider the convergence of 7 to unity. Couled with! Ä, this leads to the Q-suerlinear convergence rate of the algorithm. It is shown that both the convergence 7 to and suerlinear convergence are deendent on the Hessian aroximation d. The algorithm uses strictly ositive definite d even when the Hessian at the solution is not strictly ositive definite. First, we establish the subsace in which d needs to aroximate the Hessian of Y (x). This is used in Theorem 4. to establish the attainment of unit stesizes and in Theorem 4.2 to establish the convergence rate.

8 - (23) NLP 8 - A consequence of Theorem 3.3 is that, for sufficiently large, with strict comlementarity holding,. redicts the constraints active at x. Thus, for large, the constraints satisfied as strict inequalities at x do not affect the comutation of x. At that stage, it would mae no difference if the constraints satisfied as equalities at x were treated j j j as equality constraints. Hence, for large, h (x) = imlies h (x ) = and h (x) j imlies h (x ). Let h denote the vector constraints satisfied as equalities at x. Thus, we have h (x ) =, for large. In order to characterize the subsace for aroximating the Hessian of Y (x), we need to invoe a secial case of the mean-value theorem that holds for h (Ortega and Rheinboldt, 97). Remar: Mean-Value Theorems We note that mean value theorems usually aly to maings f: 8 Ä and do not in general hold for maings f: 8 Ä, m. We construct the result by treating each element of the vector h individually. Provided each element of the vector h is differentiable, on an oen convex set, then for any two oints x +, x, there exist t, t 2,..., t j,... (, ) such that [(x, x ) œ f h Š x + t (x x ), f h 2 Š x + t (x x ),... (4.) where f h j denotes the gradient of the j th element of h, and h (x ) h (x ) œ [ T (x, x ) Š x x. (4.2) Let the matrix [ have the same ran as [ (x +, x ) and let a linearly indeendent subset of the columns of [ (x +, x ) form the columns of [. Thus [ is of full ran and we can form T T the oerator P = I [ ([ [ ) [, which satisfies P P œ P and P Š x x œ x x. (4.3) + + As {x } Ä x, 2 š [ (x, x ) Ä f h (x ), fh (x ),... (4.4) + and P becomes the oerator rojecting vectors in n onto the active constraints at x. Theorem 4. Let the assumtions of Theorem 3.3 be satisfied. Then, there is a 5, such that (a) or, (b) for large, ½Š d d Š x + x ½ l x x l Ÿ 5, (4.5) + ½P Š d d P Š x + x ½ l x x l Ÿ 5, (4.6) +

9 - (23) NLP 9 - we have { 7 } Ä. Remar. Inequality (4.5) requires d to be close to d. This may be satisfied for d and d strictly ositive definite or for some d ositive definite and d not ositive definite. If d is not ositive definite, and there is no d satisfying (4.5), we need to consider (4.6). The latter demands only the rojections of d, d, on the constraints, to be close and the rojection of to be ositive definite. d For stesize (2.5), we can write (3.7) as Y (x ) Y (x ) Ÿ 7 q (x ) + ' ( t) x x, (x (t) + + šdš d d d Š x+ x dt (4.7) 2 l x x x d d x Ÿ 7 q(x) 7 lx x l l ½Š Š ½ (4.8) 7 ½Š d d Š x x ½ + Ÿ 7 q (x ) m x x (4.9) - l l! + where œ ' ( - t) ² d Š (x (t) d ² dt and (4.9) is obtained by invoing Lemma (3.3). The scalar 3 (, ) in (2.5) requires 7 to satisfy 7 ½Š d d Š x x ½ + 3 Ÿ m x x Ÿ. (4.)!- l l + As in (3.9)-(3.), there exists a 7 (, ], satisfying (4.9) and hence (2.5). If 5 in (4.5) is such that m! - 5 Ÿ 3 (4.) (in view of {x } Ä x, Ä, this defines the number 5) then (4.) holds with 7=, and therefore, because q (x ) Ÿ, (2.5) is satisfied with 7 =. If (4.5) cannot be achieved because d is not ositive definite, then the rojection of d d can be used, for large, by invoing (4.4) in (4.8) to yield 7 Y(x +) Y(x ) Ÿ 7 q (x ) m x x. - l l! ½P Š d d P Š x x ½ + +

10 - (23) NLP - (4.2) Using (4.6) and the same arguments as before, we establish that (2.5) is satisfied with =. 7 For stesize (2.6), (3.7) with Lemma (3.) yields + Y (x ) Y (x ) Ÿ 7 f Y(x ), x x!7 -!7 - ½Š d d Š x x ½ + + m lx x l!- 2 The scalar 3 Š, in (2.6) requires 7 to satisfy (4.3)!7 -!7 - ½Š d d Š x x ½ + 32 Ÿ m lx x l Ÿ. +!- (4.4) As in (3.)-(3.2), there exists a 7 (, ], satisfying (2.6). If 5 in (4.5) is such that! - -! m 5 Ÿ 3 (4.5) (since {x } Ä x, Ä, this defines 5) then (4.4) holds with 7=, and therefore, because fy(x ), x x Ÿ, (2.6) is satisfied with 7 =. If (4.5) cannot be achieved because d is not ositive definite, then the rojection of d d can be used, for large, by invoing (4.4) to yield + Y (x ) Y (x ) Ÿ 7 f Y(x ), x x!7 -!7 - ½P Š d d P Š x x ½ + m lx x l. + (4.6) Using (4.6) and the same arguments as before, we establish that (2.6) is satisfied with 7 =. Remar. The above roof illustrates that it is easier to attain 7 = for smaller values of -!. { 7 } will accelerate towards unity as {! } Ä. Furthermore, if 7 = while!, then reducing! to unity in subsequent iterations will increase - to its largest value -!! =. Consider, for examle, the effect of doing this in (4.) or (4.7). As! - gets larger, the bound m! - on these inequalities also gets larger. The same also alies to the bound on 5. Thus, conditions (4.5) and (4.6) become easier to satisfy and this maes it easier to maintain 7 =.

11 - (23) NLP - The left sides of bounds (4.5) and (4.6) are exected to aroach zero if suerlinear convergence is to be achieved. Theorem 4.2 below establishes these suerlinear convergence conditions. In chater 8, this discussion is revisited and further generalised for Quasi-Newton algorithms, following the earlier results of Dennis and More (977), Han (976) and Powell (978, a), among others. Lemma 4. Let {x } Ä x and lx x l Ÿ e lx x l, for some e [, _). Then where e = + e. + lx x l Ÿ e l x x l. (4.7) + The roof is immediate from the triangle inequality lx x x x l Ÿ l x x l l x x l Ÿ ( + e) lx x l. + + Remar. The hyothesis lx x + l Ÿ e l x x l is always satisfied whenever (4.7) is invoed below. Definition. Let the sequence {x } Ä x. If lx x+l Ä_ lx x l = (4.8) then {x } is convergent at a Q-suerlinear rate. Lemma 4.2 Let {x } Ä x. Then {x } is Q-suerlinearly convergent, i.e. () () lx x l Ÿ r l x x l ; r =, + iff lx + x l Ÿ r l x x - l with Ä_ r =. We have ² x x ²Ÿ! ² x x ² t- tä_ j+ j j= $ Ä_ Ÿ r ² x x - ² Ð = = =... Ñ r - - Ÿ = š ² x x ² ² x x ²

12 - (23) NLP 2 - for some = Ò, Ñ and sufficiently large. As Ö r Ä, = is chosen such that r + =, a K!. K! is an integer and is such that r, a K!. Rearranging the above exression, yields the required result. () () Suose that ² x x ² Ÿ r ² x x - ², Ä_ r =. The desired result () is obtained using Lemma (2.) and r () r - - r - ² x x ² Ÿ r () š ² x x ² + ² x x ² Ÿ Š () ² x x ². Theorem 4.2 Let (i) the assumtions of Theorem 3.3 be satisfied, (ii) be large, such that, by Theorem 4.,! = 7 =. Then, the sequence {x } generated by the GLP algorithm in Section 2 converges at a Q-suerlinear rate iff ½P Š d d P Š x + x ½ Ä_ l x x l + The first order exansion of fy (x) can be written using (4.4) as fy (x ) = (x ) fy - d Š x x- œ (4.9) ' šd Š (x (t) - d - P - Š x x - dt. (4.2) The gradient of the objective function in (2.3) is given by the first two terms on the right in (4.2). Thus, for x, the inequality + e fy (x ) - d Š x x -, x + x œ q -(x ), x + x (4.2) follows from the otimality of x in (2.3) for - min š ¾ x x - d- f Y (x -) ¾» x e. 2 d - Also, the inequality q (x ) fy (x ), x x (4.22) + + follows from the ositive definiteness of d. For large, using (2.4) with! = 7 =, (4.2)- (4.22) and Lemma 3.3, we have m + + l x x l Ÿ q (x )

13 - (23) NLP 3 - Ÿ fy (x ), x + x Ÿ q -(x ), x + x x x, P š d d P Š x x ' x x, šd Š (x (t) d Š x x dt Ÿ - ½ P š d d- P - Š x x - ½ lx x l - lx x l lx x l (4.23) where = ' ½d Š (x (t) d ½ dt and, as {x } Ä x, Ö } Ä. Hence, (4.23) yields where - lx x l Ÿ r lx x l + - r = P - 2 š d d- P - Š x x- ½ m lx x l - ½ - ½Š P P š d d P Š x x ½ lx x l Ÿ. - If (4.9) is satisfied, Ä_ r = and Lemma 4.2 yields the desired result. Suose, conversely, that Ö x converges Q-suerlinearly and thence, by Lemma 4.2, ² x x ² Ÿ r ² x x ² with r œ. By (4.23), we have (4.9). + - Ä_ REFERENCES Allwright, J.C. (98). A Feasible Direction Algorithm for Convex Otimization: Global Convergence Rates, J Otim Theory Al, -8. Aostol, T.M. (98). Mathematical Analysis, Second Edition, Addison Wesley, Reading, Massachusetts. Bertseas, D.P. (976). On the Goldstein-Levitin-Polya Gradient Projection Method, IEEE Trans on Automatic Control, AC-2,

14 - (23) NLP 4 - Bertseas, D.P. (982). Projected Newton Methods for Otimization Problems with Simle Constraints, SIAM J Control Otim, 2, Broyden C.G. (969). A New Method forsolving Nonlinear Simultaneous Equations, Comuter J., 2, 95-. Broyden C.G. (97). The Convergence of a Class of Double-Ran Minimisation Algorithms 2. The new Algorithm, J. Inst. Maths. Alics., 6, Demyanov, V.F. and A.M. Rubinov (97). Aroximate Methods in Otimization Problems, American Elsevier, New Yor. Dennis, J.E. and J.J. More (977). Quasi-Newton Methods, Motivation and Theory, SIAM Review, 9, Dunn, J.C. (979). Rates of Convergence for Conditional Gradient Algorithms Near Singular and Nonsingular Extremals, SIAM J Control Otim, 7, Dunn, J.C. (98). Newton's Method and the Goldstein Ste-length Rule for Constrained Minimization Problems, SIAM J Control Otim, 8, Dunn, J.C. (98). Global and Asymtotic Convergence Rate Estimates for a Class of Projected Gradient Processes, SIAM J Control Otim, 9, Fiacco, A.V. and G.P. McCormic (968). Nonlinear Programming: Sequential Unconstrained Minimization Techniques, Wiley, New Yor. Fletcher, R. (97). A New Aroach to Variable Metric Algorithms, Comuter J., 3, Gafni, E. and D. Bertseas (984). Two-Metric Projection Methods for Constrained Otimization, SIAM J. Control and Otimization, 22, Goldfarb D. (97). A Family of Variable Metric Algorithms Derived by Variational Means, Mathematics of Comutation, 24, Goldstein, A.A. (964). Convex Programming in Hilbert Sace, Bulletin AMS, 7, Han, S-P (976). Suerlinearly Convergent Variable Metric Algorithms for General Nonliner Programming Problems, Math Programming,, Levitin, E.S. and B.T Polya (966). Constrained Minimization Methods, USSR Com Math and Math Phys, 6, -5. McCormic, G.P. and R.A Taia (972). The Gradient Projection Method Under Mild Differentiability Conditions, SIAM J Control Otim,, 93-98

15 - (23) NLP 5 - Ortega, J.M. and W.C. Rheinboldt (97). Iterative Solution of Non-linear Equations in Several Variables, Academic Press, London and New Yor Pola, E. (97). Comutational Methods in Otimization, Academic Press, London and New Yor Powell, M.J.D. (978, a). The Convergence of Variable Metric Methods for Nonlinearly Constrained Otimization Calculations, In: J.B. Rosen, O.L. Mangasarian, K. Ritter (eds). Nonlinear Programming, Academic Press, New Yor:27-63 Powell, M.J.D. (978, b). A fast Algorithm for Nonlinearly Constrained Otimization Calculations, Numerical Analysis Proceedings, Dundee 977, Edited by G.A. Watson, Sringer-Verlag, Berlin. Psenichny, B.N. and Y.M. Danilin (978). Publishers, Moscow Numerical Methods in Extremal Problems. Mir Rustem, B. (984). A Class of Suerlinearly Convergent Projection Algorithms with Relaxed Stesizes, Al Math Otim, 2, Shanno, D.F. (97). Conditioning of Quasi-Newton Methods for Function Minimization, Mathematics of Comutation, 24,