2 Chapter 1 rely on approximating (x) by using progressively ner discretizations of [0; 1] (see, e.g. [5, 7, 8, 16, 18, 19, 20, 23]). Specically, such

Size: px
Start display at page:

Download "2 Chapter 1 rely on approximating (x) by using progressively ner discretizations of [0; 1] (see, e.g. [5, 7, 8, 16, 18, 19, 20, 23]). Specically, such"

Transcription

1 1 FEASIBLE SEQUENTIAL QUADRATIC PROGRAMMING FOR FINELY DISCRETIZED PROBLEMS FROM SIP Craig T. Lawrence and Andre L. Tits ABSTRACT Department of Electrical Engineering and Institute for Systems Research University of Maryland, College Par College Par, Maryland USA A Sequential Quadratic Programming algorithm designed to eciently solve nonlinear optimization problems with many inequality constraints, e.g. problems arising from nely discretized Semi-Innite Programming, is described and analyzed. The ey features of the algorithm are (i) that only a few of the constraints are used in the QP sub-problems at each iteration, and (ii) that every iterate satises all constraints. 1 INTRODUCTION Consider the Semi-Innite Programming (SIP) problem minimize f(x) subject to (x) 0; (SI) where f : IR n! IR is continuously dierentiable, and : IR n! IR is dened by (x) = sup (x; ); 2[0;1] with : IR n [0; 1]! IR continuously dierentiable in the rst argument. For an excellent survey of the theory behind the problem (SI), in addition to some algorithms and applications, see [9] as well as the other papers in the present volume. Many globally convergent algorithms designed to solve (SI) 1

2 2 Chapter 1 rely on approximating (x) by using progressively ner discretizations of [0; 1] (see, e.g. [5, 7, 8, 16, 18, 19, 20, 23]). Specically, such algorithms generate a sequence of problems of the form minimize f(x) subject to (x; ) 0; 8 2 ; (DSI) where [0; 1] is a (presumably large) nite set. For example, given q 2 IN, one could use the uniform discretization = 0; 1 q ; : : : ; q? 1 ; 1 : q Clearly these algorithms are crucially dependent upon being able to eciently solve problem (DSI). Of course, (DSI) involves only a nite number of smooth constraints, thus could be solved in principle via classical constrained optimization techniques. Note however that when jj is large compared to the number of variables n, it is liely that only a small subset of the constraints are active at a solution. A scheme which exploits this fact by cleverly using an appropriate small subset of the constraints at each step should, in most cases, enjoy substantial savings in computational eort without sacricing global and local convergence properties. Early eorts at employing such a scheme appear in [19, 16] in the context of rst order methods of feasible directions. In [19], at iteration, a search direction is computed based on the method of Zoutendij [28] using only the gradients of those constraints satisfying (x ; )?, where > 0 is small. Clearly, close to a solution, such \-active" constraints are sucient to ensure convergence. However, if the discretization is very ne, such an approach may still produce sub-problems with an unduly large number of constraints. It was shown in [16] that, by means of a scheme inspired by the bundle-type methods of nondierentiable optimization (see, e.g. [11, 13]), the number of constraints used in the sub-problems can be further reduced without jeopardizing global convergence. Specically, in [16], the constraints to be used in the computation of the search direction d +1 at iteration +1 are chosen as follows. Let be the set of constraints used to compute the search direction d, and let x +1 be the next iterate. Then +1 includes: All 2 such that (x +1 ; ) = 0 (i.e. the \active" constraints), All 2 which aected the computation of the search direction d, and

3 FSQP for Finely Discretized SIP 3 A 2, if it exists, which caused a step reduction in the line search at iteration. While the former is obviously needed to ensure that d is a feasible direction, it is argued in [16] that the latter two are necessary to avoid zig-zagging or other jamming phenomena. The number of constraints required to compute the search direction is thus typically small compared to jj, hence each iteration of such a method is computationally less costly. Unfortunately, for a xed level of discretization, the algorithms in [19, 16] converge at best at a linear rate. Sequential Quadratic Programming (SQP)-type algorithms exhibit fast local convergence and are well-suited for problems in which the number of variables is not too large but the evaluation of objective/constraint functions and their gradients is costly. In such algorithms, quadratic programs (QPs) are used as models to construct the search direction. For an excellent recent survey of SQP algorithms, see [2]. A number of attempts at applying the SQP scheme to problems with a large number of constraints, e.g. our discretized problem from SIP, have been documented in the literature. In [1], Biggs treats all active inequality constraints as equality constraints in the QP sub-problem, while ignoring all constraints which are not active. Pola and Tits [20], and Mine et al. [14], apply to the SQP framewor an -active scheme similar to that used in [19]. Similar to the -active idea, Powell proposes a \tolerant" algorithm for linearly constrained problems in [22]. Finally, in [26], Schittowsi proposes another modication of the SQP scheme for problems with many constraints, but does not prove convergence. In practice, the algorithm in [26] may or may not converge, dependent upon the heuristics applied to choose the constraints for the QP sub-problem. In this paper, the scheme introduced in [16] in the context of rst-order feasible direction methods is extended to the SQP framewor, specically, the Feasible SQP (FSQP) framewor introduced in [17] (the qualier \feasible" signies that all iterates x satisfy the constraints, i.e. (x ; ) 0; for all 2 ). Our presentation and analysis signicantly borrow from [27], where an important special case of (DSI) is considered, the unconstrained minimax problem. Let the feasible set be denoted X = fx 2 IR n j (x; ) 0; 8 2 g:

4 4 Chapter 1 For x 2 X, ^, and H 2 IR nn with H = H T > 0, let d 0 (x; H; ^) be the corresponding SQP direction, i.e. the unique solution of the QP minimize 1 2 hd 0 ; Hd 0 i + hrf(x); d 0 i subject to (x; ) + hr x (x; ); d 0 i 0; 8 2 ^: QP 0 (x; H; ^) At iteration, given an estimate x 2 X of the solution, a constraint index set, and a symmetric positive denite estimate H of the Hessian of the Lagrangian, rst compute d 0 = d 0 (x ; H ; ). Note that d 0 may not be a feasible search direction, as required in the FSQP context, but that at worst it is tangent to the feasible set. Since all iterates are to remain in the feasible set, following [17], an essentially arbitrary feasible descent direction d 1 is computed and the search direction is taen to be the convex combination d = (1? )d 0 + d 1 : The coecient = (d 0 ) 2 [0; 1] goes to zero fast enough, as x approaches a solution, to ensure the fast convergence rate of the standard SQP scheme is preserved. An Armijo-type line search is then performed along the direction d, yielding a step-size t 2 (0; 1]. The next iterate is taen to be x +1 = x + t d. Finally, H is updated yielding H +1, and a new constraint index set +1 is constructed following the ideas of [16]. As is pointed out in [27], the construction of [16] cannot be used meaningfully in the SQP framewor without modifying the update rule for the new metric H +1. The reason is as follows. As noted above, following [16], if t < 1, +1 is to include, among others, the index 2 of a constraint which was infeasible for the last trial point in the line search. 1 The rationale for including in +1 is that if had been in, then it is liely that the computed search direction would have allowed a longer step. Such reasoning is clearly justied in the context of rst-order search directions as is used in [16], but it is not clear that is the right constraint to include under the new metric H +1. To overcome this diculty, it is proposed in [27] that H not be updated whenever t <, a prescribed small positive number, and 62. We will show in Section 3 that, as is the case for the minimax algorithm of [27], for large enough, will always be in, thus normal updating will tae place eventually, preserving the local convergence rate properties of the SQP scheme. There is an important additional complication, with the update of, which was not present in the minimax case considered in [27]. As just pointed out, any 2 which aected the search direction is to be included in +1. In [27] (unconstrained minimax problem) this is accomplished by including those objectives whose multipliers are non-zero in the QP used to compute the search 1 Assuming that it was a constraint, and not the objective function, which caused a failure in the line search.

5 FSQP for Finely Discretized SIP 5 direction (analogous to QP 0 (x ; H ; ) above), i.e. the \binding" objectives. In our case, in addition to the binding constraints from QP 0 (x ; H ; ), we must also include those constraints which aect the computation of the feasible descent direction d 1. If this is not done, convergence is not ensured and a \zigzagging" phenomenon as discussed in [16] could result. As a nal matter on the update rule for, following [27], we allow for additional constraint indices to be added to the set. While not necessary for global convergence, cleverly choosing additional constraints can signicantly improve performance, especially in early iterations. In the context of discretized SIP, exploiting the possible regularity properties of the SIP constraints with respect to the independent parameter can give useful heuristics for choosing additional constraints. In order to guarantee fast (superlinear) local convergence, it is necessary that, for large enough, the line search always accept the step-size t = 1. It is well-nown in the SQP context that the line search could truncate the step size arbitrarily close to a solution (the so-called Maratos eect), thus preventing superlinear convergence. Various schemes have been devised to overcome such a situation. We will argue that a second-order correction, as used in [17], will overcome the Maratos eect without sacricing global convergence. The balance of the paper is organized as follows. In Section 2 we introduce the algorithm and present some preliminary material. Next, in Section 3, we give a complete convergence analysis of the algorithm proposed in Section 2. The local convergence analysis assumes the just mentioned second-order correction is used. To improve the continuity of the development, a few of the proofs are deferred to an appendix. In Section 4, the algorithm is extended to handle the constrained minimax case. Some implementation details, in addition to numerical results, are provided in Section 5. Finally, in Section 6, we oer some concluding remars. 2 ALGORITHM We begin by maing a few assumptions that will be in force throughout. The rst is a standard regularity assumption, while the second ensures that the set of active constraint gradients is always linearly independent.

6 6 Chapter 1 Assumption 1: The functions f : IR n! IR and (; ) : IR n! IR, 2 are continuously dierentiable. Dene the set of active constraints for a point x 2 X as act (x) = f 2 j (x; ) = 0g: Assumption 2: For all x 2 X with act (x) 6= ;, the set fr x (x; ) j 2 act (x)g is linearly independent. A point x 2 IR n is called a Karush-Kuhn-Tucer (KKT) point for the problem (DSI) if there exist KKT multipliers, 2, satisfying 8 >< >: rf(x ) + X 2 r x (x ; ) = 0; (x ; ) 0; 8 2 ; (x ; ) = 0 and 0; 8 2 : (1.1) Under our assumptions, any local minimizer x for (DSI) is a KKT point. Thus, (1.1) provides a set of rst-order necessary conditions of optimality. Throughout our analysis, we will often mae use of the KKT conditions for QP 0 (x; H; ^). Specically, given x 2 X, H = H T > 0, and ^, d 0 is a KKT point for QP 0 (x; H; ^) if there exist multipliers 0 ; 2 ^, satisfying 8 >< >: Hd 0 + rf(x) + X 2^ 0 r x (x; ) = 0; (x; ) + hr x (x; ); d 0 i 0; ^;? (x; ) + hrx (x; ); d 0 i = 0 and 0 0; 8 2 ^: (1.2) In fact, since the objective for QP 0 (x; H; ^) is strictly convex, such a d 0 is the unique KKT point, as well as the unique global minimizer (stated formally in Lemma 1 below). As noted above, d 0 need not be a feasible direction. The search direction d will be taen as a convex combination of d 0 and a rst-order feasible descent direction d 1. For x 2 X and ^, we compute d 1 = d 1 (x; ^), and = (x; ^),

7 FSQP for Finely Discretized SIP 7 as the solution of the QP minimize 1 2 d1 2 + subject to hrf(x); d 1 i ; (x; ) + hr x (x; ); d 1 i ; 8 2 ^: QP 1 (x; ^) The notation will be used throughout to denote the standard Euclidean norm. The pair (d 1 ; ) is a KKT point for QP 1 (x; ^) if there exist multipliers 1 and 1, 2 ^, satisfying 8 d 1 >< >: rf(x)?1 hrf(x); d 1 i ; (x; ) + hr x (x; ); d 1 i ; X + 1 rx (x; )?1 2^ 8 2 ^; = 0;? 1 hrf(x); d 1 i? = 0 and 1 0;? (x; ) + hrx (x; ); d 1 i? = 0 and 1 0; 8 2 ^: 1 (1.3) In Section 1 we stated that the feasible descent direction d 1 was essentially arbitrary. In the subsequent analysis we assume that d 1 is chosen specically as the solution of QP 1 (x; ^), though it can be shown that the results still hold if some minor variation is used. To be precise, following [17], we require that d 1 = d 1 (x; ^) satisfy: d 1 (x; ^) = 0 if x is a KKT point, hrf(x); d 1 (x; ^)i < 0 if x is not a KKT point, hr x (x; ); d 1 (x; ^)i < 0, for all 2 act if x is not a KKT point, and for ^ xed, d 1 (x; ^) is bounded over bounded subsets of X. It will be shown in Lemma 2 that the solution of QP 1 (x; ^) satises these requirements. In our context, d 1 must fulll one additional property, which is essentially captured by Lemma 7 in the appendix. Thus, at iteration, the search direction d is taen as a convex combination of d 0 and d1, i.e. d = (1? )d 0 + d 1, 2 [0; 1]. In order to guarantee a fast local rate of convergence while providing a suitably feasible search direction, we require the coecient of the convex combination = (d 0 ) to satisfy certain properties. Namely, () : IR n! [0; 1] must satisfy

8 8 Chapter 1 (d 0 ) is bounded away from zero outside every neighborhood of zero, and (d 0 ) = O(d 0 2 ): For example, we could tae (d 0 ) = minf1; d 0 g, where 2. It remains to explicitly specify the ey feature of the proposed algorithm: the update rule for. As discussed above, following [16], +1 will include (in addition to possible heuristics) three crucial components. The rst one is the set act (x +1 ) of indices of active constraints at the new iterate. The second component of +1 is the set b of indices of constraints that aected d. In particular, b will include all indices of constraints in QP 0 (x ; H ; ) and QP 1 (x ; ) which have positive multipliers, i.e. the binding constraints for these QPs. Specically, let 0 ;, and 1 ;, for 2, be the QP multipliers from QP 0 (x ; H ; ) and QP 1 (x ; ), respectively. Dening we let b;0 = f 2 j 0 ; > 0 g; b;1 = f 2 j 1 ; > 0 g; b = b;0 [ b;1 : Finally, the third component of +1 is the index of one constraint, if any exists, that forced a reduction of the step in the previous line search. While the exact type of line search we employ is not critical to our analysis, we assume from this point onward that it is an Armijo-type search. That is, given constants 2 (0; 1=2) and 2 (0; 1), the step-size t is taen as the rst number t in the set f1; ; 2 ; : : :g such that and f(x + td ) f(x ) + thrf(x ); d i; (1.4) (x + td ; ) 0; 8 2 : (1.5) Thus, t < 1 implies that either (1.4) or (1.5) is violated at x + t d. In the event that (1.5) is violated, there exists 2 such that x + t d ; > 0; (1.6) and in such a case we will include in +1.

9 FSQP for Finely Discretized SIP 9 Algorithm FSQP-MC Parameters. 2 (0; 1 ), 2 (0; 1), and 0 < 1. 2 Data. x 0 2 X, 0 < H 0 = H T 0 2 IR nn. Step 0 - Initialization. Set 0 and choose 0 act (x 0 ): Step 1 - Computation of search direction. (i). Compute d 0 = d 0 (x ; H ; ). If d 0 = 0, stop. (ii). Compute d 1 = d1 (x ; ). (iii). Compute = (d 0 ) and set d (1? )d 0 + d 1. Step 2 - Line search. Compute t, the rst number t in the sequence f1; ; 2 ; : : :g satisfying (1.4) and (1.5). Step 3 - Updates. (i). Set x +1 x + t d. (ii). If t < 1 and (1.5) was violated at x + t d, then let be such that (1.6) holds. (iii). Pic +1 act (x +1 ) [ b : If t < 1 and (1.6) holds for some 2, then set [ f g: (iv). If t and 62, set H +1 H. Otherwise, obtain a new symmetric positive denite estimate H +1 to the Hessian of the Lagrangian. (v). Set + 1 and go bac to Step 1. 3 CONVERGENCE ANALYSIS While there are some critical dierences, the analysis in this section closely parallels that of [27]. We begin by establishing that, under a few additional assumptions, algorithm FSQP-MC generates a sequence which converges to a KKT point for (DSI). Then, upon strengthening our assumptions slightly, we show that the rate of convergence is two-step superlinear.

10 10 Chapter Global Convergence The following will be assumed to hold throughout our analysis. Assumption 3: The level set f x 2 IR n j f(x) f(x 0 ) g \ X is compact. Assumption 4: There exist scalars 0 < 1 2 such that for all, 1 d 2 hd; H di 2 d 2 ; 8d 2 IR n : Given the scalars 0 < 1 2 from Assumption 4, dene H = fh = H T j 1 d 2 hd; Hdi 2 d 2 ; 8d 2 IR n g: First, we derive some properties of d 0 (x; H; ^) and d 1 (x; ^). Lemma 1 For all x 2 X, H 2 H, and ^ such that act (x) ^, the search direction d 0 = d 0 (x; H; ^) is well-dened, is continuous in x and H for ^ xed, and is the unique KKT point for QP 0 (x; H; ^). Furthermore, d 0 = 0 if, and only if, x is a KKT point for the problem (DSI). If x is not a KKT point for this problem, then d 0 satises hrf(x); d 0 i < 0; (1.7) hr x (x; ); d 0 i 0; 8 2 act (x): (1.8) Proof: H > 0 implies that QP 0 (x; H; ^) is strictly convex. Further d 0 = 0 is always feasible for the QP constraints. It follows that the solution d 0 is well-dened and the unique KKT point for the QP. As the set H is uniformly positive denite, continuity in x and H for xed ^ is a direct consequence of Theorem 4.4 in [4]. Now suppose d 0 is 0 and let f 0 j 2 ^ g be the QP multipliers. In view of the KKT conditions (1.2) for QP 0 (x; H; ^), since d 0 = 0 and x 2 X, we see that x satises the KKT conditions (1.1) for (DSI) with = ( 0 ; 2 ^; 0; 62 ^: The converse is proved similarly, appealing to the uniqueness of the KKT point for QP 0 (x; H; ^) and the fact that act (x) ^. Finally, since act (x) ^, (1.7) and (1.8) follow from Proposition 3.1 in [17]. 2

11 FSQP for Finely Discretized SIP 11 Lemma 2 For all x 2 X, and ^ such that act (x) ^, the direction d 1 = d 1 (x; ^) is well-dened and the pair (d 1 ; ), where = (x; ^), is the unique KKT point of QP 1 (x; ^). Furthermore, for given ^, d 1 (x; ^) is bounded over bounded subsets of X. In addition, d 1 = 0 if, and only if, x is a KKT point for the problem (DSI). If x is not a KKT point for this problem, then d 1 satises hrf(x); d 1 i < 0; (1.9) and satises < 0. hr x (x; ); d 1 i < 0; 8 2 act (x); (1.10) Proof: We begin by noting that (d 1 ; ) solves QP 1 (x; ^) if, and only if, d 1 solves minimize 1 2 d1 2 + max hrf(x); d 1 i; maxf(x; ) + hr x (x; ); d 1 ig : 2^ and = max hrf(x); d 1 i; maxf(x; ) + hr x (x; ); d 1 ig 2^ : (1.11) Since the objective function in (1.11) is strictly convex and radially unbounded, it follows that QP 1 (x; ^) has (d 1 ; ) = (d 1 (x; ^); (x; ^)) as its unique global minimizer. Since QP 1 (x; ^) is convex, (d 1 ; ) is also its unique KKT point. Boundedness of d 1 (x; ^) over bounded subsets of X follows from the rst equation of the optimality conditions (1.3), noting that the QP multipliers must all lie in [0; 1]. Now suppose d 1 = 0. Since x 2 X, it is clear that = 0. Substitute d 1 = 0 and = 0 into (1.3) and let 1 2 IR and 1 2 IR j^j be the corresponding multipliers. Note that, in view of Assumption 2, 1 > 0. Since x 2 X, it follows that x satises (1.1) with multipliers = ( 1 =1 ; 2 ^; 0; 62 ^: Therefore, x is a KKT point for (DSI). The converse is proved similarly, appealing to uniqueness of the KKT point for QP 1 (x; ^), and the fact that act (x) ^. To prove (1.9) and (1.10) note that if x is not a KKT point for (DSI), then as just shown d 1 6= 0, hence < 0 (since d 1 = 0 and = 0 form a feasible pair, the optimal value of QP 1 (x ; ) must be non-positive). The result then follows directly from the form of the QP constraints and the fact that act (x) ^. 2 Next, we establish that the line search is well-dened.

12 12 Chapter 1 Lemma 3 For each, the line search in Step 2 yields a step t = j for some nite j = j(). Proof: If x is a KKT point, then Step 2 is not reached, hence assume x is not KKT. In view of Lemma 1 and the properties of (), d 0 6= 0 and (d 0 ) > 0. Lemmas 1 and 2 imply, since act (x ), that hrf(x ); d i < 0; hr x (x ; ); d i < 0; 8 2 act (x ): Further, feasibility of x requires (x ; ) < 0 for all 2 n act (x ). The result then follows by considering rst order expansions of f(x + t d ) and (x +t d ; ), 2, about x and by appealing to our regularity assumptions. 2 The previous three lemmas imply that the algorithm is well-dened. In addition, Lemma 1 shows that if Algorithm FSQP-MC generates a nite sequence terminating at the point x N, then x N is a KKT point for the problem (DSI). We now concentrate on the case in which the algorithm never satises the termination condition in Step 1(i) and generates an innite sequence fx g. Given an innite index set K, we use the notation to mean 2K x?! x x! x as! 1; 2 K: Lemma 4 Let K be an innite index set such that = for all 2 K, 2K x?! x 2K, H?! H, d 0 2K?! d 0;, d 1 2K?! d 1; 2K, and?!. Then (i) d 0; is the unique solution of QP 0 (x ; H ; ), and (ii) (d 1; ; ) is the unique solution of QP 1 (x ; ). Proof: Part (i) follows from continuity of d 0 (x; H; ^) for xed ^ (Lemma 1). To prove part (ii), recall that in view of Lemma 2, (d 1 ; ) is the unique KKT point for QP 1 (x ; ), i.e. is the unique solution of (1.3) with corresponding multipliers 1 0 and 1 ; 0, 2. Note that the multipliers satisfy 1 + X 2 1 ; = 1; for all, hence are bounded. Let K 0 K be an innite index set such that 1 2K 0?! 1; and 1 2K 0 ;?! 1;, 2. Taing limits in the optimality conditions (1.3) shows that (d 1; ; ) is a KKT point for QP 1 (x ; ) with multipliers 1; and 1;, 2. Uniqueness of such points proves the result. 2

13 FSQP for Finely Discretized SIP 13 The following lemma establishes a few basic properties of some of the sequences generated by the algorithm. Lemma 5 (i) The sequences fx g, fd 0 g, and fd g are bounded; (ii) ff(x )g converges; (iii) t d?! 0. Proof: In view of Assumption 3 and the fact that the line search guarantees ff(x )g is a monotonically decreasing sequence, it follows that fx g is bounded, and since f() is continuous, that ff(x )g converges. Boundedness of fd 0 g follows from boundedness of fx g, Assumption 4, continuity of d 0 (x; H; ^) for xed ^, and the fact that there are only nitely many subsets ^ of. Boundedness of fd 1 g follows from Lemma 2 and boundedness of fx g. Since 2 [0; 1], fd g is bounded as well. Finally, suppose ft d g 6! 0. Then there exists an innite index set K IN such that t d is bounded away from zero on K. Since all sequences of interest are bounded and is nite, we may suppose 2K without loss of generality that x?! x 2K, H?! H, d 0 2K?! d 0;, d 1 2K?! d 1;, 2K?! 2K,?!, and = for all 2 K. Lemma 4 tells us that d 0; is the unique solution of QP 0 (x ; H ; ) and (d 1; ; ) is the unique solution pair for QP 1 (x ; ). Furthermore, since t d is bounded away from zero on K, there exists t > 0 such that t t for all 2 K, and since ft g is bounded (t 2 [0; 1]), it follows that either d 0; 6= 0 or d 1; 6= 0. Applying Lemmas 1 and 2 in both directions shows that x is not a KKT point for (DSI) and both d 0; 6= 0 and d 1; 6= 0. In addition, < 0 and < 0, for all 2 K (from Lemma 2) and is bounded away from zero on K (from the assumptions on () as given in Section 2 and the fact that d 0 is bounded away from zero on K). As a consequence, there exists > 0 such that = (d 0 ) for all 2 K and there exists < 0 such that for all 2 K. Now, since hrf(x ); d 1 i, for all, t hrf(x ); d i = t (1? )hrf(x ); d 0 i + t hrf(x ); d 1 i t t < 0; for all 2 K, where we have used (1.7). Thus, by the line search criterion of Step 2, f(x +1 ) f(x ) + t for all 2 K. Since f(x ) is monotone non-increasing, it follows that ff(x )g diverges, which contradicts (ii). 2 In order to establish convergence to a KKT point, it will be convenient to consider the value functions for the search direction QPs, QP 0 (x; H; ^) and

14 14 Chapter 1 QP 1 (x; ^). In particular, given the solutions d 0 = d 0 (x; H; ^) and (d 1 ; ) = (d 1 (x; ^); (x; ^)), dene v 0 (x; H; ^) = 1? 2 hd 0 ; Hd 0 i + hrf(x); d 0 i ; v 1 (x; ^) = 1? 2 d1 2 + ^ : Further, let v(x; H; ^) = v 0 (x; H; ^) + v 1 (x; ^); and, at iteration, dene v 0 = v0 (x ; H ; ), v 1 = v1 (x ; ), and v = v(x ; H ; ). Note that, since 0 is always feasible for both QPs, v 0 0 and v1 0, for all. Lemma 6 Let K be an innite index set. Then (i) d 0 2K?! 0 if and only if v 0 2K?! 0, (ii) (d 1 ; )?! 2K (0; 0) if and only if v 1 2K?! 0, and (iii) if d 0 2K?! 0, then all accumulation points of fx g 2K are KKT points for (DSI). Proof: First, if d 0 2K?! 0 then it is clear from the denition of v 0 that v 0 2K?! 0. Next, from (1.2) and the last statement in Lemma 1, it follows that hrf(x ); d 0 i?hd0 ; H d 0 i. Thus, using again the denition of v0, we get v 0 =?? 1 2 hd0 ; H d 0 i + hrf(x ); d 0 i? 1 2 hd0 ; H d 0 i + hd0 ; H d 0 i 1 2 d0 2 > 0; where we have used Assumption 4. Thus, if v 0 2K?! 0, then d 0 2K?! 0. If (d 1 ; )?! 2K (0; 0), then from the denition of v 1 we see that v1 2K?! 0. Now suppose v 1 2K?! 0: To show d 1 2K?! 0, note that from the optimality conditions (1.3), X d 1 2 =? 1 hrf(x ); d 1 i? 1 ; hr x(x ; ); d 1 i X 2 =? 1? 1 ;(? (x ; )) X 2 =? + 1 ;(x ; )? : 2 Thus, again using the denition of v 1, v 1 =? 1 2 d1 2? 1 2 d1 2 ;

15 FSQP for Finely Discretized SIP 15 and it immediately follows that d 1 2K 2K?! 0 and?! 0. To prove (iii), suppose K is such that d 0 2K?! 0. Let x be an accumulation point of fx g 2K and let K 0 2K K be an innite index set such that x 0?! x and, for some ^, = ^ for all 2 K 0. Let 0 2 IRj^j be the multiplier vector from QP 0 (x ; H ; ^) and dene 0 = f 2 ^ j (x ; ) + hr x (x ; ); d 0 i = 0 g: Suppose, without loss of generality, 0 = ^ 0 ; for all 2 K 0. As d 0 0, it is clear that ^ 0 act (x ) and, in view of Assumption 2, the set f r x (x ; ) j 2 ^ 0 g is linearly independent, for large enough. Thus, from (1.2), a unique expression for the QP multipliers (for large enough) is given by 0 =? ^R(x ) T ^R(x )?1 ^R(x ) T? H d 0 + rf(x ) ; 2K 0?! where ^R(x ) = [ r x (x ; ) j 2 ^ 0 ] 2 IR nj^ 0j. In view of Assumption 4, boundedness of fx g, the regularity assumptions, and the fact that d 0 2K?! 0 0, we see that 0 2K 0?! 0; =? ^R(x ) T ^R(x )?1 ^R(x ) T rf(x ): Taing limits in the optimality conditions (1.2) for QP 0 (x ; H ; ^) shows that x is a KKT point for (DSI) with multipliers = ( 0; 2 ^; 0 62 ^: We are now in a position to show that there exists an accumulation point of fx g which is a KKT point for (DSI). This result is, in fact, weaer than that obtained in [27] for the unconstrained minimax case, where under similar assumptions, but with a more involved argument, it is shown that all accumulation points are KKT. The price to be paid is the introduction of Assumption 5 below for proving Theorem 1. The proof of the following result is inspired by that of Theorem T in [16]. 2 Proposition 1 lim inf v = 0:

16 16 Chapter 1 Corollary 1 There exists an accumulation point x of fx g which is a KKT point for (DSI). Proof: Since v 0 0 and v1 0 for all, Proposition 1 implies lim inf v0 = 0; i.e. there exists an innite index set K such that v 0 2K?! 0. In view of Lemma 6, all accumulation points of fx g 2K are KKT points. Finally, boundedness of fx g implies at least one such point exists. 2 Dene the Lagrangian function for (DSI) as L(x; ) = f(x) + X 2 (x; ): In order to show that the entire sequence converges to a KKT point x, we strengthen our assumptions as follows. Assumption 1 0 : The functions f : IR n! IR and (; ) : IR n! IR, 2 are twice continuously dierentiable. Assumption 5: Some accumulation point x of fx g which is a KKT point for (DSI) also satises the second order suciency conditions with strict complementary slacness, i.e. there exists 2 IR jj satisfying (1.1) as well as r 2 xxl(x ; ) is positive denite on the subspace and > 0 for all 2 act(x ). fh j hr x (x ; ); hi = 0; 8 2 act (x )g; It is well-nown that such an assumption implies that x is an isolated KKT point for (DSI) as well as an isolated local minimizer. The following theorem is the main result of this section. Theorem 1 The sequence fx g generated by algorithm FSQP-MC converges to a strict local minimizer x of (DSI). Proof: First we show that there exists a neighborhood of x in which no other accumulation points of fx g can exist, KKT points or not. As x is a strict local minimizer, there exists > 0 such that f(x) > f(x ) for all x 6= x, x 2 S = B(x ; ) \ X, where B(x ; ) is the open ball of radius centered at

17 FSQP for Finely Discretized SIP 17 x. Proceeding by contradiction, suppose x 0 2 B(x ; ), x 0 6= x, is another accumulation point of fx g. Feasibility of the iterates implies that x 0 2 S. Thus f(x 0 ) > f(x ), which is in contradiction with Lemma 5(ii). Next, in view of Lemma 5(iii), (x +1? x )?! 0. Suppose K is an innite index set such 2K that x?! x. Then there exists 1 such that x? x < =4, for all 2 K, 1. Further, there exists 2 such that x +1? x < =4, for all > 2. Therefore, if there were another accumulation point outside of B(x ; ), then the sequence would have to pass through the compact set B(x ; ) n B(x ; =4) innitely many times. This contradicts the established fact that there are no accumulation points of fx g, other than x, in B(x ; ) Local Convergence We have thus shown that, with a liely dramatically reduced amount of wor per iteration, global convergence can be preserved. This would be of little interest, though, if the speed of convergence were to suer signicantly. In this section we establish that, under a few additional assumptions, the sequence fx g generated by a slightly modied version of algorithm FSQP-MC (to avoid the Maratos eect) exhibits 2-step superlinear convergence. To do this, the bul of our eort is focussed on showing that for large the set of constraints b which aect the search direction is precisely the set of active constraints at the solution, i.e. act (x ). In addition, we show that, eventually, no constraints outside of act (x ) aect the line search, and that H is updated normally at every iteration. Thus, for large enough, the algorithm behaves as if it were solving the problem minimize f(x) subject to (x; ) 0; 8 2 act (x ); (P ) using all constraints at every iteration. Establishing this allows us to apply nown results concerning local convergence rates. The following is proved in the appendix. Proposition 2 For large enough, (i) b;0 = b;1 = act (x ), and (ii) (x + td ; ) 0 for all t 2 [0; 1], 2 n act (x ).

18 18 Chapter 1 In order to achieve superlinear convergence, it is crucial that a unit step, i.e. t = 1, always be accepted for all suciently large. Several techniques have been introduced to avoid the so-called Maratos eect. We chose to include a second order correction such as that used in [17]. Specically, at iteration, let ~ d(x; d; H; ^) be the unique solution of the QP g QP (x; d; H; ^), dened for 2 (2; 3) as follows 1 minimize hd + ~ 2 d; H(d + d)i ~ + hrf(x); d + di ~ subject to (x + d; ) + hr x (x; ); d + di ~ d ; 8 2 ^; gqp (x; d; H; ^) if it exists and has norm less that minfd; Cg, where C is a large number. Otherwise, set d(x; ~ d; H; ^) = 0. The following step is added to algorithm FSQP-MC: Step 1(iv). Compute ~ d = ~ d(x ; d ; H ; ). In addition, the line search criterion (1.4) and (1.5) are replaced with and f(x + td + t 2 ~ d ) f(x ) + thrf(x ); d i; (1.12) Finally, the condition (1.6) is replaced with x + t d t + (x + td + t 2 ~ d ) 0; 8 2 : (1.13) 2 ~d ;! > 0: (1.14) With some eort, it can be shown that these modications do not aect any of the results obtained to this point. Further, for suciently large, the set of binding constraints in QP g (x ; d ; H ; ) is again act (x ). Hence, it is established that for large enough, the modied algorithm FSQP-MC behaves identically to that given in [17], applied to (P ). Assumption 1 is now further strengthened and a new assumption concerning the Hessian approximations H is given. These assumptions allow us to use the local convergence rate result from [17]. Assumption 1 00 : The functions f : IR n! IR, and (; ) : IR n! IR, 2, are three times continuously dierentiable. Assumption 6: As a result of the update rule chosen for Step 3(iv), H approaches the Hessian of the Lagrangian in the sense that P (H? r 2 xx lim L(x ; ))P d = 0;!1 d

19 FSQP for Finely Discretized SIP 19 where is the KKT multiplier vector associated with x and with R = [r x (x ; ) j 2 act (x )]. P = I? R (R T R )?1 R T Theorem 2 For all suciently large, the unit step t = 1 is accepted in Step 2. Further, the sequence fx g converges to x 2-step superlinearly, i.e. x +2? x lim!1 x? x = 0: 4 EXTENSION TO CONSTRAINED MINIMAX The algorithm we have discussed may be extended following the scheme of [27] to handle problems with many objective functions, i.e. large-scale constrained minimax. Specically, consider the problem minimize max f(x;!)!2 subject to (x; ) 0; 8 2 ; where and are nite (again, presumably large) sets, and f : IR n! IR and : IR n! IR are both three times continuously dierentiable with respect to their rst argument. Given ^, dene F ^(x) = max f(x;!):!2 ^ Given a direction d 2 IR n, dene a rst-order approximation of F ^(x + d)? F ^(x) by F 0^(x; d) = maxff(x + d;!) + hr x f(x;!); dig? F ^(x);!2 ^ and, nally, given a direction ~ d 2 IR n, let ~F 0^(x; d; d) ~ = maxff(x + d;!) + hr x f(x;!); dig ~? F ^(x + d):!2 ^

20 20 Chapter 1 Let be the set of objective functions used to compute the search direction at iteration. The modied QPs follow. To compute d 0 (x; H; ^; ^), we solve minimize 1 2 hd 0 ; Hd 0 i + F 0^(x; d 0 ) subject to (x; ) + hr x (x; ); d 0 i 0; 8 2 ^; QP 0 (x; H; ^; ^) and to compute d 1 (x; ^; ^), we solve minimize 1 2 d1 2 + subject to F 0^(x; d 1 ) ; (x; ) + hr x (x; ); d 1 i ; Finally, to compute ~ d(x; d; H; ^; ^), we solve 8 2 ^: QP 1 (x; ^; ^) minimize 1 2 hd + ~ d; H(d + ~ d)i + ~ F 0^(x; d; ~ d) subject to (x + d; ) + hr x (x; ); d + ~ di d ; 8 2 ^; gqp (x; d; H; ^; ^) where again, if the QP has no solution or if the solution has norm greater than minfd; Cg, we set ~ d(x; d; H; ^; ^) = 0. In order to describe the update rules for, following [27], we dene a few index sets for the objectives (in direct analogy with the index sets for the constraints as introduced in Section 2). The set of indices of \maximizing" objectives is dened in the obvious manner as max(x) = f! 2 j f(x;!) = F (x)g: At iteration, let 0 ;!,! 2, be the multipliers from QP 0 (x ; H ; ; ) associated with the objective functions. Liewise, let 1 ;!,! 2, be the multipliers from QP 1 (x ; ; ) associated with the objective functions. The set of indices of objective functions which aected the computation of the search direction d is given by 2 b = f! 2 j 0 ;! > 0 or 1 ;! > 0g: The line search criterion (1.12) is replaced with F (x + td + t 2 ~ d ) F (x ) + tf 0 (x ; d ): (1.15) 2 QP 1 (x; ^; ^) is not needed in the unconstrained case. Accordingly, in [27], b is dened based on a single set of multipliers.

21 FSQP for Finely Discretized SIP 21 If t < 1 and the truncation is due to an objective function, then dene! 2 such that f x + t d + 2 t ~d ;!! > F (x ) + t F (x 0 ; d ): (1.16) We are now in a position to state the extended algorithm. Algorithm FSQP-MOC Parameters. 2 (0; 1 ), 2 (0; 1), and 0 < 1. 2 Data. x 0 2 IR n, 0 < H 0 = H T 0 2 IR nn. Step 0 - Initialization. Set 0. Choose 0 max(x 0 ), 0 act (x 0 ): Step 1 - Computation of search direction. (i). Compute d 0 = d 0 (x ; H ; ; ). If d 0 = 0, stop. (ii). Compute d 1 = d1 (x ; ; ). (iii). Compute = (d 0) and set d (1? )d 0 + d 1. (iv). Compute ~ d = ~ d(x ; d ; H ; ; ). Step 2 - Line search. Compute t, the rst number t in the sequence f1; ; 2 ; : : :g satisfying (1.15) and (1.13). Step 3 - Updates. (i). Set x +1 x + t d + t 2 ~ d. (ii). If t < 1 and (1.15) was violated at x +1 = x + t d + t 2 ~d ; then let! be such that (1.16) holds. If (1.13) was violated at x +1, then let be such that (1.14) holds. (iii). Pic +1 max(x +1 ) [ b ; +1 act (x +1 ) [ b : and If t < 1 and (1.16) holds for some! 2, then set [ f!g: If t < 1 and (1.14) holds for some 2, then set [ fg: (iv). If t and! 62 or 62 set H +1 H. Otherwise, obtain a new symmetric positive denite estimate H +1 to the Hessian of the Lagrangian. (v). Set + 1 and go bac to Step 1.

22 22 Chapter 1 5 IMPLEMENTATION AND NUMERICAL RESULTS Algorithm FSQP-MOC has been implemented as part of the code cfsqp 3 [12]. The numerical test results reported in this section were obtained with a modied copy of cfsqp Version 2.4 (the relevant changes will be included in subsequent releases, beginning with Version 2.5). All test problems we consider here are instances of (DSI). Thus in this section we only discuss implementation details relevant to solving such problems, i.e. implementation details of algorithm FSQP-MC modied to include the second order correction ~ d. 4 The implementation allows for multiple discretized SIP constraints and contains special provisions for those which are ane in x. Specically, problem (DSI) is generalized to minimize f(x) subject to j (x; ) = hc j (); xi? d j () 0; 8 2 (j) ; j = 1; : : : ; m`; j (x; ) 0; 8 2 (j) ; j = m` + 1; : : : ; m; where c j : (j) `?! IR n, j = 1; : : : ; m`, d j : (j) `?! IR, j = 1; : : : ; m`, and (j) is nite, j = 1; : : : ; m. The assumptions and algorithm statement are generalized in the obvious manner. As far as the analysis of Section 3 is concerned, such a formulation could readily be adapted to the format of (DSI) by grouping all constraints together, i.e. letting = [ m j=1 (j), and ignoring the fact that some may be ane in x. In the case that the initial point x 0 provided is not feasible for the ane constraints, cfsqp rst computes v 2 IR n as the solution of the strictly convex QP minimize hv; vi subject to hc j (); x 0 + vi? d j () 0; 8 2 (j) ; j = 1; : : : ; m`; for ^v such that x 0 + ^v is feasible for linear constraints. If the new initial point is not feasible for all nonlinear constraints, then cfsqp iterates, using the algorithm FSQP-MOC, on the constrained minimax problem minimize max j=m`+1;:::;m max 2 (j) f j(x; )g subject to hc j (); xi? d j () 0; 8 2 (j) ` ; j = 1; : : : ; m`; 3 Available from the authors. See 4 That is, we will not discuss the implementation details specic to the minimax algorithm, even though in the case that the initial guess is infeasible for nonlinear constraints, the minimax algorithm is used to generate a feasible point.

23 FSQP for Finely Discretized SIP 23 until max max j=m`+1;:::;m 2 (j)f j(x; )g 0 is achieved. The nal iterate will be feasible for all constraints, allowing the algorithm to be applied to the original problem. Recall that it is only required that contain certain subsets of. The algorithm allows for additional elements of to be included in order to speed up initial convergence. Of course, there is a trade-o between speeding up initial convergence and increasing (i) the number of gradient evaluations and (ii) the size of the QPs. In the implementation, heuristics are applied to add potentially useful elements to (see, e.g. [26] for a discussion of such heuristics). In the case of discretized SIP, one may wish to exploit the nowledge that adjacent discretization points are liely to be closely related. Following [27, 16, 6], for some > 0, the cfsqp implementation includes in the set ``m (x ) of -active \left local maximizers" at x. At a point x 2 X, for j = 1; : : : ; m, dene the -active discretization points as (j) (x) = f 2 (j) j j (x; )?g: Such a discretization point (j) i 2 (j) = f (j) 1 ; : : : ; (j) j (j) jg is a left local maximizer if it satises one of the following three conditions: (i) i 2 f2; : : : ; j (j) j?1g and and j (x; (j) i ) > j (x; (j) i?1 ) (1.17) j (x; (j) i ) j (x; (j) i+1 ); (1.18) (ii) i = 1 and (1.18); (iii) i = j (j) j and (1.17). The set ``m (x) is the set of all left local maximizers in (x) = [ m j=1 (j) (x): The rst part of the update (i.e. before updates due to line search violations) in Step 3(iii) of the algorithm becomes +1 = act (x ) [ b [ ``m (x ): Finally, we have found that in practice, including the end-points (whether or not they are close to being active) during the rst iteration often leads to a better initial search direction. Thus we set 0 1 [ 0 = act (x 0 ) [ ``m (x 0 ) m (f (j) 1 g [ f(j) j (j) j g) A : A few other specics of the cfsqp implementation are as follows. First, as was discussed in Section 2, it is not required to use QP 1 (x; ^) to compute our j=1

24 24 Chapter 1 feasible descent direction d 1. In fact, at iteration, cfsqp uses the following QP, which is a function of the SQP direction d 0, minimize 2 d0? d1 2 + subject to hrf(x ); d 1 i ; (x ; ) + hr x (x ; ); d 1 i ; 8 2 ; where was set to 0:1 for our numerical tests. Using such an objective function encourages d 1 to be \close" to d 0, a condition we have found to be benecial in practice. It can be veried that the arguments given in Section 3 go through with little modication if we disable the inclusion of d 0 in the QP objective function when the step size t?1 from the previous iteration is less than a given small threshold. 5 The expression used for is given by = d 0 d 0 + ; where = maxf0:5; d 1 g, with = 2:1 and = 2:5 for our numerical experiments. The matrices H are updated using the BFGS formula with Powell's modication [21]. The multiplier estimates used for the updates are those obtained from QP 0 (x ; H ; ), with all multipliers corresponding to discretization points outside of set to zero. The QP sub-problems were solved using the routine QLD due to Powell and Schittowsi [25]. Finally, the following parameter values were used for all numerical testing: = 0:1, = 0:5, = 1, and was set to the square root of the machine precision. In order to judge the eciency of algorithm FSQP-MOC, we ran the same numerical tests with two other algorithms diering only in the manner in which is updated. The results are given in Tables 1 and 2. All test algorithms were implemented by maing the appropriate modications to cfsqp Version 2.4. In the tables, the implementation of FSQP-MOC just discussed is denoted NEW. A simple -active strategy was employed in the algorithm we call -ACT, i.e. we set = (x ) for all, where = 0:1. The standard FSQP scheme of [17] was applied in algorithm FULL by simply setting =, for all. All three algorithms were set to stop when d ?4. We report the numerical results for 13 discretized SIP test problems with discretization levels of 101 and 501. A uniform discretization was used in all cases. Problems cw 3, cw 5, and cw 6 are borrowed from [3]. Problems oet 1 through oet 7 are from [15]. Finally, hz 1 is from [10] and sch 3 is from [26]. In all cases except for oet 7, the initial guess x 0 is the same as that given 5 In the numerical experiments reported here, t remained bounded away from 0.

25 FSQP for Finely Discretized SIP 25 in the reference from which the problem was taen. For oet 7, -ACT and FULL had diculty generating a feasible point, thus we used the feasible initial guess x 0 = (0; 0; 0;?7;?3;?1; 3). The rst two columns of the tables are self-explanatory. A description of the remaining columns is as follows. The third column, n, indicates the number of variables, while m` and m n in the next two columns indicate the number of linear SIP constraints and nonlinear SIP constraints (m n = m? m`), respectively. Next, NF is the number of objective function evaluations, NG is the number of \scalar" constraint function evaluations (i.e. evaluation of some j (x; ) for a given x and ), and IT indicates the number of iterations required before the stopping criterion was satised. Finally, P j j is the sum over all iterations of the size of (it is equal to the number of gradient evaluations in the case of NEW and FULL), j j is the size of at the nal iterate, and TIME is the time of execution in seconds on a Sun Sparc 4 worstation. For all test problems and for all three algorithms, the value of the objective function at the nal iterate agreed (to within four signicant gures) with the optimal value as given in the original references. A few conclusions may be drawn from the results. In general, NEW requires the most iterations to \converge" to a solution, whereas FULL requires the least. Typically, though, the dierence is not large. Of course, such behavior is expected since NEW uses a simpler QP model at each iteration. It is clear from comparing the results for P j j that NEW provides signicant savings in the number of gradient evaluations and the size of the QP sub-problems. The savings for -ACT are not as dramatic. In almost all cases, comparing TIME of execution conrms that, indeed, NEW requires far less computational eort than either of the other two approaches. Further, note that j j remains, in general, unchanged for NEW when the discretization level is increased and is typically equal to, or less than, n. This is not the case for -ACT and FULL, as would be expected. Such behavior suggests that computational eort does not increase with respect to discretization level for NEW at the same rate as it does for -ACT and FULL. This conclusion is supported by the increase in execution TIME when discretization level increases.

26 26 Chapter 1 P PROB ALGO n m` m n NF NG IT j j j j TIME cw 3 NEW ACT FULL cw 5 NEW ACT FULL cw 6 NEW ACT FULL oet 1 NEW ACT FULL oet 2 NEW ACT FULL oet 3 NEW ACT FULL oet 4 NEW ACT FULL oet 5 NEW ACT FULL oet 6 NEW ACT FULL oet 7 NEW ACT FULL hz 1 NEW ACT FULL sch 3 NEW ACT FULL Table 1 Numerical results for problems with j (j) j = 101.

27 FSQP for Finely Discretized SIP 27 P PROB ALGO n m` m n NF NG IT j j j j TIME cw 3 NEW ACT FULL cw 5 NEW ACT FULL cw 6 NEW ACT FULL oet 1 NEW ACT FULL oet 2 NEW ACT FULL oet 3 NEW ACT FULL oet 4 NEW ACT FULL oet 5 NEW ACT FULL oet 6 NEW ACT FULL oet 7 NEW ACT FULL hz 1 NEW ACT FULL sch 3 NEW ACT FULL Table 2 Numerical results for problems with j (j) j = 501.

28 28 Chapter 1 6 CONCLUSIONS We have presented and analyzed a feasible SQP algorithm for tacling smooth nonlinear programming problems with a large number of constraints, e.g. those arising from discretization of SIP problems. At each iteration, only a small subset of the constraints are used in the QP sub-problems. Thus, fewer gradient evaluations are required and the computational eort to solve the QP sub-problems is decreased. We showed that the scheme for choosing which constraints are to be included in the QP sub-problems preserves global and fast local convergence. Numerical results obtained from the cfsqp implementation show that, indeed, the algorithm performs favorably. Acnowledgments. The authors wish to than Stephan Gorner for his many valuable comments on a preliminary draft of the paper. This research was supported in part by NSF grant No. DMI and by the NSF Engineering Research Centers Program NSFD-CDR REFERENCES [1] M. C. Biggs. Constrained minimization using recursive equality quadratic programming. In F. A. Lootsma, editor, Numerical Methods for Non- Linear Optimization, pages 411{428. Academic Press, New Yor, [2] P. T. Boggs and J. W. Tolle. Sequential quadratic programming. Acta Numerica, pages 1{51, [3] I. D. Coope and G. A. Watson. A projected lagrangian algorithm for semi-innite programming. Math. Programming, 32:337{356, [4] J. W. Daniel. Stability of the solution of denite quadratic programs. Math. Programming, 5:41{53, [5] C. Gonzaga and E. Pola. On constraint dropping schemes and optimality functions for a class of outer approximation algorithms. SIAM J. Control Optim., 17:477{493, [6] C. Gonzaga, E. Pola, and R. Trahan. An improved algorithm for optimization problems with functional inequality constraints. IEEE Transactions on Automatic Control, AC-25:49{54, [7] S. A. Gustafson. A three-phase algorithm for semi-innite programs. In A. V. Fiacco and K. O. Kortane, editors, Semi-Innite Programming

2 jian l. zhou and andre l. tits The diculties in solving (SI), and in particular (CMM), stem mostly from the facts that (i) the accurate evaluation o

2 jian l. zhou and andre l. tits The diculties in solving (SI), and in particular (CMM), stem mostly from the facts that (i) the accurate evaluation o SIAM J. Optimization Vol. x, No. x, pp. x{xx, xxx 19xx 000 AN SQP ALGORITHM FOR FINELY DISCRETIZED CONTINUOUS MINIMAX PROBLEMS AND OTHER MINIMAX PROBLEMS WITH MANY OBJECTIVE FUNCTIONS* JIAN L. ZHOUy AND

More information

A new ane scaling interior point algorithm for nonlinear optimization subject to linear equality and inequality constraints

A new ane scaling interior point algorithm for nonlinear optimization subject to linear equality and inequality constraints Journal of Computational and Applied Mathematics 161 (003) 1 5 www.elsevier.com/locate/cam A new ane scaling interior point algorithm for nonlinear optimization subject to linear equality and inequality

More information

A Primal-Dual Interior-Point Method for Nonlinear Programming with Strong Global and Local Convergence Properties

A Primal-Dual Interior-Point Method for Nonlinear Programming with Strong Global and Local Convergence Properties A Primal-Dual Interior-Point Method for Nonlinear Programming with Strong Global and Local Convergence Properties André L. Tits Andreas Wächter Sasan Bahtiari Thomas J. Urban Craig T. Lawrence ISR Technical

More information

2 JOSE HERSKOVITS The rst stage to get an optimal design is to dene the Optimization Model. That is, to select appropriate design variables, an object

2 JOSE HERSKOVITS The rst stage to get an optimal design is to dene the Optimization Model. That is, to select appropriate design variables, an object A VIEW ON NONLINEAR OPTIMIZATION JOSE HERSKOVITS Mechanical Engineering Program COPPE / Federal University of Rio de Janeiro 1 Caixa Postal 68503, 21945-970 Rio de Janeiro, BRAZIL 1. Introduction Once

More information

A Simple Primal-Dual Feasible Interior-Point Method for Nonlinear Programming with Monotone Descent

A Simple Primal-Dual Feasible Interior-Point Method for Nonlinear Programming with Monotone Descent A Simple Primal-Dual Feasible Interior-Point Method for Nonlinear Programming with Monotone Descent Sasan Bahtiari André L. Tits Department of Electrical and Computer Engineering and Institute for Systems

More information

1. Introduction The nonlinear complementarity problem (NCP) is to nd a point x 2 IR n such that hx; F (x)i = ; x 2 IR n + ; F (x) 2 IRn + ; where F is

1. Introduction The nonlinear complementarity problem (NCP) is to nd a point x 2 IR n such that hx; F (x)i = ; x 2 IR n + ; F (x) 2 IRn + ; where F is New NCP-Functions and Their Properties 3 by Christian Kanzow y, Nobuo Yamashita z and Masao Fukushima z y University of Hamburg, Institute of Applied Mathematics, Bundesstrasse 55, D-2146 Hamburg, Germany,

More information

National Institute of Standards and Technology USA. Jon W. Tolle. Departments of Mathematics and Operations Research. University of North Carolina USA

National Institute of Standards and Technology USA. Jon W. Tolle. Departments of Mathematics and Operations Research. University of North Carolina USA Acta Numerica (1996), pp. 1{000 Sequential Quadratic Programming Paul T. Boggs Applied and Computational Mathematics Division National Institute of Standards and Technology Gaithersburg, Maryland 20899

More information

Optimality Conditions for Constrained Optimization

Optimality Conditions for Constrained Optimization 72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)

More information

5 Handling Constraints

5 Handling Constraints 5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest

More information

ARE202A, Fall Contents

ARE202A, Fall Contents ARE202A, Fall 2005 LECTURE #2: WED, NOV 6, 2005 PRINT DATE: NOVEMBER 2, 2005 (NPP2) Contents 5. Nonlinear Programming Problems and the Kuhn Tucker conditions (cont) 5.2. Necessary and sucient conditions

More information

minimize x subject to (x 2)(x 4) u,

minimize x subject to (x 2)(x 4) u, Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for

More information

Nonmonotonic back-tracking trust region interior point algorithm for linear constrained optimization

Nonmonotonic back-tracking trust region interior point algorithm for linear constrained optimization Journal of Computational and Applied Mathematics 155 (2003) 285 305 www.elsevier.com/locate/cam Nonmonotonic bac-tracing trust region interior point algorithm for linear constrained optimization Detong

More information

Extra-Updates Criterion for the Limited Memory BFGS Algorithm for Large Scale Nonlinear Optimization M. Al-Baali y December 7, 2000 Abstract This pape

Extra-Updates Criterion for the Limited Memory BFGS Algorithm for Large Scale Nonlinear Optimization M. Al-Baali y December 7, 2000 Abstract This pape SULTAN QABOOS UNIVERSITY Department of Mathematics and Statistics Extra-Updates Criterion for the Limited Memory BFGS Algorithm for Large Scale Nonlinear Optimization by M. Al-Baali December 2000 Extra-Updates

More information

system of equations. In particular, we give a complete characterization of the Q-superlinear

system of equations. In particular, we give a complete characterization of the Q-superlinear INEXACT NEWTON METHODS FOR SEMISMOOTH EQUATIONS WITH APPLICATIONS TO VARIATIONAL INEQUALITY PROBLEMS Francisco Facchinei 1, Andreas Fischer 2 and Christian Kanzow 3 1 Dipartimento di Informatica e Sistemistica

More information

1 Introduction Sequential Quadratic Programming (SQP) methods have proved to be very ecient for solving medium-size nonlinear programming problems [12

1 Introduction Sequential Quadratic Programming (SQP) methods have proved to be very ecient for solving medium-size nonlinear programming problems [12 A Trust Region Method Based on Interior Point Techniques for Nonlinear Programming Richard H. Byrd Jean Charles Gilbert y Jorge Nocedal z August 10, 1998 Abstract An algorithm for minimizing a nonlinear

More information

A REDUCED HESSIAN METHOD FOR LARGE-SCALE CONSTRAINED OPTIMIZATION by Lorenz T. Biegler, Jorge Nocedal, and Claudia Schmid ABSTRACT We propose a quasi-

A REDUCED HESSIAN METHOD FOR LARGE-SCALE CONSTRAINED OPTIMIZATION by Lorenz T. Biegler, Jorge Nocedal, and Claudia Schmid ABSTRACT We propose a quasi- A REDUCED HESSIAN METHOD FOR LARGE-SCALE CONSTRAINED OPTIMIZATION by Lorenz T. Biegler, 1 Jorge Nocedal, 2 and Claudia Schmid 1 1 Chemical Engineering Department, Carnegie Mellon University, Pittsburgh,

More information

NOTES ON EXISTENCE AND UNIQUENESS THEOREMS FOR ODES

NOTES ON EXISTENCE AND UNIQUENESS THEOREMS FOR ODES NOTES ON EXISTENCE AND UNIQUENESS THEOREMS FOR ODES JONATHAN LUK These notes discuss theorems on the existence, uniqueness and extension of solutions for ODEs. None of these results are original. The proofs

More information

Spurious Chaotic Solutions of Dierential. Equations. Sigitas Keras. September Department of Applied Mathematics and Theoretical Physics

Spurious Chaotic Solutions of Dierential. Equations. Sigitas Keras. September Department of Applied Mathematics and Theoretical Physics UNIVERSITY OF CAMBRIDGE Numerical Analysis Reports Spurious Chaotic Solutions of Dierential Equations Sigitas Keras DAMTP 994/NA6 September 994 Department of Applied Mathematics and Theoretical Physics

More information

The Kuhn-Tucker Problem

The Kuhn-Tucker Problem Natalia Lazzati Mathematics for Economics (Part I) Note 8: Nonlinear Programming - The Kuhn-Tucker Problem Note 8 is based on de la Fuente (2000, Ch. 7) and Simon and Blume (1994, Ch. 18 and 19). The Kuhn-Tucker

More information

154 ADVANCES IN NONLINEAR PROGRAMMING Abstract: We propose an algorithm for nonlinear optimization that employs both trust region techniques and line

154 ADVANCES IN NONLINEAR PROGRAMMING Abstract: We propose an algorithm for nonlinear optimization that employs both trust region techniques and line 7 COMBINING TRUST REGION AND LINE SEARCH TECHNIQUES Jorge Nocedal Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL 60208-3118, USA. Ya-xiang Yuan State Key Laboratory

More information

arxiv: v1 [math.oc] 10 Apr 2017

arxiv: v1 [math.oc] 10 Apr 2017 A Method to Guarantee Local Convergence for Sequential Quadratic Programming with Poor Hessian Approximation Tuan T. Nguyen, Mircea Lazar and Hans Butler arxiv:1704.03064v1 math.oc] 10 Apr 2017 Abstract

More information

Numerical Optimization

Numerical Optimization Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,

More information

AM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods

AM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α

More information

Lecture 13: Constrained optimization

Lecture 13: Constrained optimization 2010-12-03 Basic ideas A nonlinearly constrained problem must somehow be converted relaxed into a problem which we can solve (a linear/quadratic or unconstrained problem) We solve a sequence of such problems

More information

An improved convergence theorem for the Newton method under relaxed continuity assumptions

An improved convergence theorem for the Newton method under relaxed continuity assumptions An improved convergence theorem for the Newton method under relaxed continuity assumptions Andrei Dubin ITEP, 117218, BCheremushinsaya 25, Moscow, Russia Abstract In the framewor of the majorization technique,

More information

Structural and Multidisciplinary Optimization. P. Duysinx and P. Tossings

Structural and Multidisciplinary Optimization. P. Duysinx and P. Tossings Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be

More information

QUADRATICALLY AND SUPERLINEARLY CONVERGENT ALGORITHMS FOR THE SOLUTION OF INEQUALITY CONSTRAINED MINIMIZATION PROBLEMS 1

QUADRATICALLY AND SUPERLINEARLY CONVERGENT ALGORITHMS FOR THE SOLUTION OF INEQUALITY CONSTRAINED MINIMIZATION PROBLEMS 1 QUADRATICALLY AND SUPERLINEARLY CONVERGENT ALGORITHMS FOR THE SOLUTION OF INEQUALITY CONSTRAINED MINIMIZATION PROBLEMS 1 F. FACCHINEI 2 AND S. LUCIDI 3 Communicated by L.C.W. Dixon 1 This research was

More information

Applied Mathematics &Optimization

Applied Mathematics &Optimization Appl Math Optim 29: 211-222 (1994) Applied Mathematics &Optimization c 1994 Springer-Verlag New Yor Inc. An Algorithm for Finding the Chebyshev Center of a Convex Polyhedron 1 N.D.Botin and V.L.Turova-Botina

More information

Constrained Optimization and Lagrangian Duality

Constrained Optimization and Lagrangian Duality CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

More information

Projected Gradient Methods for NCP 57. Complementarity Problems via Normal Maps

Projected Gradient Methods for NCP 57. Complementarity Problems via Normal Maps Projected Gradient Methods for NCP 57 Recent Advances in Nonsmooth Optimization, pp. 57-86 Eds..-Z. u, L. Qi and R.S. Womersley c1995 World Scientic Publishers Projected Gradient Methods for Nonlinear

More information

On the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method

On the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method Optimization Methods and Software Vol. 00, No. 00, Month 200x, 1 11 On the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method ROMAN A. POLYAK Department of SEOR and Mathematical

More information

Pacific Journal of Optimization (Vol. 2, No. 3, September 2006) ABSTRACT

Pacific Journal of Optimization (Vol. 2, No. 3, September 2006) ABSTRACT Pacific Journal of Optimization Vol., No. 3, September 006) PRIMAL ERROR BOUNDS BASED ON THE AUGMENTED LAGRANGIAN AND LAGRANGIAN RELAXATION ALGORITHMS A. F. Izmailov and M. V. Solodov ABSTRACT For a given

More information

Interior-Point Methods for Linear Optimization

Interior-Point Methods for Linear Optimization Interior-Point Methods for Linear Optimization Robert M. Freund and Jorge Vera March, 204 c 204 Robert M. Freund and Jorge Vera. All rights reserved. Linear Optimization with a Logarithmic Barrier Function

More information

CONSTRAINED NONLINEAR PROGRAMMING

CONSTRAINED NONLINEAR PROGRAMMING 149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach

More information

1. Introduction In this paper we study a new type of trust region method for solving the unconstrained optimization problem, min f(x): (1.1) x2< n Our

1. Introduction In this paper we study a new type of trust region method for solving the unconstrained optimization problem, min f(x): (1.1) x2< n Our Combining Trust Region and Line Search Techniques Jorge Nocedal y and Ya-xiang Yuan z March 14, 1998 Report OTC 98/04 Optimization Technology Center Abstract We propose an algorithm for nonlinear optimization

More information

S. Lucidi F. Rochetich M. Roma. Curvilinear Stabilization Techniques for. Truncated Newton Methods in. the complete results 1

S. Lucidi F. Rochetich M. Roma. Curvilinear Stabilization Techniques for. Truncated Newton Methods in. the complete results 1 Universita degli Studi di Roma \La Sapienza" Dipartimento di Informatica e Sistemistica S. Lucidi F. Rochetich M. Roma Curvilinear Stabilization Techniques for Truncated Newton Methods in Large Scale Unconstrained

More information

Numerical optimization

Numerical optimization Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal

More information

AN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING

AN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING AN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING XIAO WANG AND HONGCHAO ZHANG Abstract. In this paper, we propose an Augmented Lagrangian Affine Scaling (ALAS) algorithm for general

More information

On the Convergence of Newton Iterations to Non-Stationary Points Richard H. Byrd Marcelo Marazzi y Jorge Nocedal z April 23, 2001 Report OTC 2001/01 Optimization Technology Center Northwestern University,

More information

AM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods

AM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality

More information

An Active Set Strategy for Solving Optimization Problems with up to 200,000,000 Nonlinear Constraints

An Active Set Strategy for Solving Optimization Problems with up to 200,000,000 Nonlinear Constraints An Active Set Strategy for Solving Optimization Problems with up to 200,000,000 Nonlinear Constraints Klaus Schittkowski Department of Computer Science, University of Bayreuth 95440 Bayreuth, Germany e-mail:

More information

Garrett: `Bernstein's analytic continuation of complex powers' 2 Let f be a polynomial in x 1 ; : : : ; x n with real coecients. For complex s, let f

Garrett: `Bernstein's analytic continuation of complex powers' 2 Let f be a polynomial in x 1 ; : : : ; x n with real coecients. For complex s, let f 1 Bernstein's analytic continuation of complex powers c1995, Paul Garrett, garrettmath.umn.edu version January 27, 1998 Analytic continuation of distributions Statement of the theorems on analytic continuation

More information

Numerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems

Numerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems 1 Numerical optimization Alexander & Michael Bronstein, 2006-2009 Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Numerical optimization 048921 Advanced topics in vision Processing and Analysis of

More information

Algorithms for Constrained Optimization

Algorithms for Constrained Optimization 1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic

More information

Exercise Solutions to Functional Analysis

Exercise Solutions to Functional Analysis Exercise Solutions to Functional Analysis Note: References refer to M. Schechter, Principles of Functional Analysis Exersize that. Let φ,..., φ n be an orthonormal set in a Hilbert space H. Show n f n

More information

Lagrange Relaxation and Duality

Lagrange Relaxation and Duality Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler

More information

Vector Space Basics. 1 Abstract Vector Spaces. 1. (commutativity of vector addition) u + v = v + u. 2. (associativity of vector addition)

Vector Space Basics. 1 Abstract Vector Spaces. 1. (commutativity of vector addition) u + v = v + u. 2. (associativity of vector addition) Vector Space Basics (Remark: these notes are highly formal and may be a useful reference to some students however I am also posting Ray Heitmann's notes to Canvas for students interested in a direct computational

More information

Constrained optimization: direct methods (cont.)

Constrained optimization: direct methods (cont.) Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a

More information

Optimization Methods. Lecture 18: Optimality Conditions and. Gradient Methods. for Unconstrained Optimization

Optimization Methods. Lecture 18: Optimality Conditions and. Gradient Methods. for Unconstrained Optimization 5.93 Optimization Methods Lecture 8: Optimality Conditions and Gradient Methods for Unconstrained Optimization Outline. Necessary and sucient optimality conditions Slide. Gradient m e t h o d s 3. The

More information

Linear Regression and Its Applications

Linear Regression and Its Applications Linear Regression and Its Applications Predrag Radivojac October 13, 2014 Given a data set D = {(x i, y i )} n the objective is to learn the relationship between features and the target. We usually start

More information

SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS

SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 2005, DR RAPHAEL HAUSER 1. The Quasi-Newton Idea. In this lecture we will discuss

More information

Algorithms for constrained local optimization

Algorithms for constrained local optimization Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained

More information

4TE3/6TE3. Algorithms for. Continuous Optimization

4TE3/6TE3. Algorithms for. Continuous Optimization 4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca

More information

Generalization to inequality constrained problem. Maximize

Generalization to inequality constrained problem. Maximize Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum

More information

A globally and R-linearly convergent hybrid HS and PRP method and its inexact version with applications

A globally and R-linearly convergent hybrid HS and PRP method and its inexact version with applications A globally and R-linearly convergent hybrid HS and PRP method and its inexact version with applications Weijun Zhou 28 October 20 Abstract A hybrid HS and PRP type conjugate gradient method for smooth

More information

8 Barrier Methods for Constrained Optimization

8 Barrier Methods for Constrained Optimization IOE 519: NL, Winter 2012 c Marina A. Epelman 55 8 Barrier Methods for Constrained Optimization In this subsection, we will restrict our attention to instances of constrained problem () that have inequality

More information

Linear stochastic approximation driven by slowly varying Markov chains

Linear stochastic approximation driven by slowly varying Markov chains Available online at www.sciencedirect.com Systems & Control Letters 50 2003 95 102 www.elsevier.com/locate/sysconle Linear stochastic approximation driven by slowly varying Marov chains Viay R. Konda,

More information

PARAMETER IDENTIFICATION IN THE FREQUENCY DOMAIN. H.T. Banks and Yun Wang. Center for Research in Scientic Computation

PARAMETER IDENTIFICATION IN THE FREQUENCY DOMAIN. H.T. Banks and Yun Wang. Center for Research in Scientic Computation PARAMETER IDENTIFICATION IN THE FREQUENCY DOMAIN H.T. Banks and Yun Wang Center for Research in Scientic Computation North Carolina State University Raleigh, NC 7695-805 Revised: March 1993 Abstract In

More information

The iterative convex minorant algorithm for nonparametric estimation

The iterative convex minorant algorithm for nonparametric estimation The iterative convex minorant algorithm for nonparametric estimation Report 95-05 Geurt Jongbloed Technische Universiteit Delft Delft University of Technology Faculteit der Technische Wiskunde en Informatica

More information

Convex Optimization. Newton s method. ENSAE: Optimisation 1/44

Convex Optimization. Newton s method. ENSAE: Optimisation 1/44 Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)

More information

In view of (31), the second of these is equal to the identity I on E m, while this, in view of (30), implies that the first can be written

In view of (31), the second of these is equal to the identity I on E m, while this, in view of (30), implies that the first can be written 11.8 Inequality Constraints 341 Because by assumption x is a regular point and L x is positive definite on M, it follows that this matrix is nonsingular (see Exercise 11). Thus, by the Implicit Function

More information

Outline Introduction: Problem Description Diculties Algebraic Structure: Algebraic Varieties Rank Decient Toeplitz Matrices Constructing Lower Rank St

Outline Introduction: Problem Description Diculties Algebraic Structure: Algebraic Varieties Rank Decient Toeplitz Matrices Constructing Lower Rank St Structured Lower Rank Approximation by Moody T. Chu (NCSU) joint with Robert E. Funderlic (NCSU) and Robert J. Plemmons (Wake Forest) March 5, 1998 Outline Introduction: Problem Description Diculties Algebraic

More information

BFGS WITH UPDATE SKIPPING AND VARYING MEMORY. July 9, 1996

BFGS WITH UPDATE SKIPPING AND VARYING MEMORY. July 9, 1996 BFGS WITH UPDATE SKIPPING AND VARYING MEMORY TAMARA GIBSON y, DIANNE P. O'LEARY z, AND LARRY NAZARETH x July 9, 1996 Abstract. We give conditions under which limited-memory quasi-newton methods with exact

More information

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

More information

A Generalized Homogeneous and Self-Dual Algorithm. for Linear Programming. February 1994 (revised December 1994)

A Generalized Homogeneous and Self-Dual Algorithm. for Linear Programming. February 1994 (revised December 1994) A Generalized Homogeneous and Self-Dual Algorithm for Linear Programming Xiaojie Xu Yinyu Ye y February 994 (revised December 994) Abstract: A generalized homogeneous and self-dual (HSD) infeasible-interior-point

More information

MODIFYING SQP FOR DEGENERATE PROBLEMS

MODIFYING SQP FOR DEGENERATE PROBLEMS PREPRINT ANL/MCS-P699-1097, OCTOBER, 1997, (REVISED JUNE, 2000; MARCH, 2002), MATHEMATICS AND COMPUTER SCIENCE DIVISION, ARGONNE NATIONAL LABORATORY MODIFYING SQP FOR DEGENERATE PROBLEMS STEPHEN J. WRIGHT

More information

STATIONARITY RESULTS FOR GENERATING SET SEARCH FOR LINEARLY CONSTRAINED OPTIMIZATION

STATIONARITY RESULTS FOR GENERATING SET SEARCH FOR LINEARLY CONSTRAINED OPTIMIZATION STATIONARITY RESULTS FOR GENERATING SET SEARCH FOR LINEARLY CONSTRAINED OPTIMIZATION TAMARA G. KOLDA, ROBERT MICHAEL LEWIS, AND VIRGINIA TORCZON Abstract. We present a new generating set search (GSS) approach

More information

4.6 Montel's Theorem. Robert Oeckl CA NOTES 7 17/11/2009 1

4.6 Montel's Theorem. Robert Oeckl CA NOTES 7 17/11/2009 1 Robert Oeckl CA NOTES 7 17/11/2009 1 4.6 Montel's Theorem Let X be a topological space. We denote by C(X) the set of complex valued continuous functions on X. Denition 4.26. A topological space is called

More information

Optimization Tutorial 1. Basic Gradient Descent

Optimization Tutorial 1. Basic Gradient Descent E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.

More information

A CHARACTERIZATION OF STRICT LOCAL MINIMIZERS OF ORDER ONE FOR STATIC MINMAX PROBLEMS IN THE PARAMETRIC CONSTRAINT CASE

A CHARACTERIZATION OF STRICT LOCAL MINIMIZERS OF ORDER ONE FOR STATIC MINMAX PROBLEMS IN THE PARAMETRIC CONSTRAINT CASE Journal of Applied Analysis Vol. 6, No. 1 (2000), pp. 139 148 A CHARACTERIZATION OF STRICT LOCAL MINIMIZERS OF ORDER ONE FOR STATIC MINMAX PROBLEMS IN THE PARAMETRIC CONSTRAINT CASE A. W. A. TAHA Received

More information

Midterm 1. Every element of the set of functions is continuous

Midterm 1. Every element of the set of functions is continuous Econ 200 Mathematics for Economists Midterm Question.- Consider the set of functions F C(0, ) dened by { } F = f C(0, ) f(x) = ax b, a A R and b B R That is, F is a subset of the set of continuous functions

More information

CONSTRAINED OPTIMALITY CRITERIA

CONSTRAINED OPTIMALITY CRITERIA 5 CONSTRAINED OPTIMALITY CRITERIA In Chapters 2 and 3, we discussed the necessary and sufficient optimality criteria for unconstrained optimization problems. But most engineering problems involve optimization

More information

Search Directions for Unconstrained Optimization

Search Directions for Unconstrained Optimization 8 CHAPTER 8 Search Directions for Unconstrained Optimization In this chapter we study the choice of search directions used in our basic updating scheme x +1 = x + t d. for solving P min f(x). x R n All

More information

Constrained Optimization Theory

Constrained Optimization Theory Constrained Optimization Theory Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Constrained Optimization Theory IMA, August

More information

Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms: A Comparative Study

Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms: A Comparative Study International Journal of Mathematics And Its Applications Vol.2 No.4 (2014), pp.47-56. ISSN: 2347-1557(online) Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms:

More information

UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems

UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction

More information

Optimization. A first course on mathematics for economists

Optimization. A first course on mathematics for economists Optimization. A first course on mathematics for economists Xavier Martinez-Giralt Universitat Autònoma de Barcelona xavier.martinez.giralt@uab.eu II.3 Static optimization - Non-Linear programming OPT p.1/45

More information

The Steepest Descent Algorithm for Unconstrained Optimization

The Steepest Descent Algorithm for Unconstrained Optimization The Steepest Descent Algorithm for Unconstrained Optimization Robert M. Freund February, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 1 Steepest Descent Algorithm The problem

More information

IE 5531: Engineering Optimization I

IE 5531: Engineering Optimization I IE 5531: Engineering Optimization I Lecture 12: Nonlinear optimization, continued Prof. John Gunnar Carlsson October 20, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I October 20,

More information

MS&E 318 (CME 338) Large-Scale Numerical Optimization

MS&E 318 (CME 338) Large-Scale Numerical Optimization Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods

More information

Optimization Problems with Constraints - introduction to theory, numerical Methods and applications

Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)

More information

TMA 4180 Optimeringsteori KARUSH-KUHN-TUCKER THEOREM

TMA 4180 Optimeringsteori KARUSH-KUHN-TUCKER THEOREM TMA 4180 Optimeringsteori KARUSH-KUHN-TUCKER THEOREM H. E. Krogstad, IMF, Spring 2012 Karush-Kuhn-Tucker (KKT) Theorem is the most central theorem in constrained optimization, and since the proof is scattered

More information

Optimal Newton-type methods for nonconvex smooth optimization problems

Optimal Newton-type methods for nonconvex smooth optimization problems Optimal Newton-type methods for nonconvex smooth optimization problems Coralia Cartis, Nicholas I. M. Gould and Philippe L. Toint June 9, 20 Abstract We consider a general class of second-order iterations

More information

How to Pop a Deep PDA Matters

How to Pop a Deep PDA Matters How to Pop a Deep PDA Matters Peter Leupold Department of Mathematics, Faculty of Science Kyoto Sangyo University Kyoto 603-8555, Japan email:leupold@cc.kyoto-su.ac.jp Abstract Deep PDA are push-down automata

More information

Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted

More information

Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted

More information

Numerical Comparisons of. Path-Following Strategies for a. Basic Interior-Point Method for. Revised August Rice University

Numerical Comparisons of. Path-Following Strategies for a. Basic Interior-Point Method for. Revised August Rice University Numerical Comparisons of Path-Following Strategies for a Basic Interior-Point Method for Nonlinear Programming M. A rg a e z, R.A. T a p ia, a n d L. V e l a z q u e z CRPC-TR97777-S Revised August 1998

More information

Numerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09

Numerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09 Numerical Optimization 1 Working Horse in Computer Vision Variational Methods Shape Analysis Machine Learning Markov Random Fields Geometry Common denominator: optimization problems 2 Overview of Methods

More information

IE 5531: Engineering Optimization I

IE 5531: Engineering Optimization I IE 5531: Engineering Optimization I Lecture 15: Nonlinear optimization Prof. John Gunnar Carlsson November 1, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I November 1, 2010 1 / 24

More information

min f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;

min f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term; Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many

More information

1. Introduction Let the least value of an objective function F (x), x2r n, be required, where F (x) can be calculated for any vector of variables x2r

1. Introduction Let the least value of an objective function F (x), x2r n, be required, where F (x) can be calculated for any vector of variables x2r DAMTP 2002/NA08 Least Frobenius norm updating of quadratic models that satisfy interpolation conditions 1 M.J.D. Powell Abstract: Quadratic models of objective functions are highly useful in many optimization

More information

Lecture 3. Optimization Problems and Iterative Algorithms

Lecture 3. Optimization Problems and Iterative Algorithms Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex

More information

Solving generalized semi-infinite programs by reduction to simpler problems.

Solving generalized semi-infinite programs by reduction to simpler problems. Solving generalized semi-infinite programs by reduction to simpler problems. G. Still, University of Twente January 20, 2004 Abstract. The paper intends to give a unifying treatment of different approaches

More information

IBM Research Report. Line Search Filter Methods for Nonlinear Programming: Motivation and Global Convergence

IBM Research Report. Line Search Filter Methods for Nonlinear Programming: Motivation and Global Convergence RC23036 (W0304-181) April 21, 2003 Computer Science IBM Research Report Line Search Filter Methods for Nonlinear Programming: Motivation and Global Convergence Andreas Wächter, Lorenz T. Biegler IBM Research

More information

Average Reward Parameters

Average Reward Parameters Simulation-Based Optimization of Markov Reward Processes: Implementation Issues Peter Marbach 2 John N. Tsitsiklis 3 Abstract We consider discrete time, nite state space Markov reward processes which depend

More information

Lecture V. Numerical Optimization

Lecture V. Numerical Optimization Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize

More information

Chapter 8 Gradient Methods

Chapter 8 Gradient Methods Chapter 8 Gradient Methods An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Introduction Recall that a level set of a function is the set of points satisfying for some constant. Thus, a point

More information

Unconstrained optimization

Unconstrained optimization Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout

More information

INDEFINITE TRUST REGION SUBPROBLEMS AND NONSYMMETRIC EIGENVALUE PERTURBATIONS. Ronald J. Stern. Concordia University

INDEFINITE TRUST REGION SUBPROBLEMS AND NONSYMMETRIC EIGENVALUE PERTURBATIONS. Ronald J. Stern. Concordia University INDEFINITE TRUST REGION SUBPROBLEMS AND NONSYMMETRIC EIGENVALUE PERTURBATIONS Ronald J. Stern Concordia University Department of Mathematics and Statistics Montreal, Quebec H4B 1R6, Canada and Henry Wolkowicz

More information

118 Cores, D. and Tapia, R. A. A Robust Choice of the Lagrange Multiplier... For example: chemical equilibrium and process control oil extraction, ble

118 Cores, D. and Tapia, R. A. A Robust Choice of the Lagrange Multiplier... For example: chemical equilibrium and process control oil extraction, ble A Robust Choice of the Lagrange Multiplier in the SQP Newton Method Debora Cores 1 Richard A. Tapia 2 1 INTEVEP, S.A. (Research Center of the Venezuelan Oil Company) Apartado 76343, Caracas 1070-A, Venezuela

More information