2 Chapter 1 rely on approximating (x) by using progressively ner discretizations of [0; 1] (see, e.g. [5, 7, 8, 16, 18, 19, 20, 23]). Specically, such
|
|
- Ursula Garrison
- 5 years ago
- Views:
Transcription
1 1 FEASIBLE SEQUENTIAL QUADRATIC PROGRAMMING FOR FINELY DISCRETIZED PROBLEMS FROM SIP Craig T. Lawrence and Andre L. Tits ABSTRACT Department of Electrical Engineering and Institute for Systems Research University of Maryland, College Par College Par, Maryland USA A Sequential Quadratic Programming algorithm designed to eciently solve nonlinear optimization problems with many inequality constraints, e.g. problems arising from nely discretized Semi-Innite Programming, is described and analyzed. The ey features of the algorithm are (i) that only a few of the constraints are used in the QP sub-problems at each iteration, and (ii) that every iterate satises all constraints. 1 INTRODUCTION Consider the Semi-Innite Programming (SIP) problem minimize f(x) subject to (x) 0; (SI) where f : IR n! IR is continuously dierentiable, and : IR n! IR is dened by (x) = sup (x; ); 2[0;1] with : IR n [0; 1]! IR continuously dierentiable in the rst argument. For an excellent survey of the theory behind the problem (SI), in addition to some algorithms and applications, see [9] as well as the other papers in the present volume. Many globally convergent algorithms designed to solve (SI) 1
2 2 Chapter 1 rely on approximating (x) by using progressively ner discretizations of [0; 1] (see, e.g. [5, 7, 8, 16, 18, 19, 20, 23]). Specically, such algorithms generate a sequence of problems of the form minimize f(x) subject to (x; ) 0; 8 2 ; (DSI) where [0; 1] is a (presumably large) nite set. For example, given q 2 IN, one could use the uniform discretization = 0; 1 q ; : : : ; q? 1 ; 1 : q Clearly these algorithms are crucially dependent upon being able to eciently solve problem (DSI). Of course, (DSI) involves only a nite number of smooth constraints, thus could be solved in principle via classical constrained optimization techniques. Note however that when jj is large compared to the number of variables n, it is liely that only a small subset of the constraints are active at a solution. A scheme which exploits this fact by cleverly using an appropriate small subset of the constraints at each step should, in most cases, enjoy substantial savings in computational eort without sacricing global and local convergence properties. Early eorts at employing such a scheme appear in [19, 16] in the context of rst order methods of feasible directions. In [19], at iteration, a search direction is computed based on the method of Zoutendij [28] using only the gradients of those constraints satisfying (x ; )?, where > 0 is small. Clearly, close to a solution, such \-active" constraints are sucient to ensure convergence. However, if the discretization is very ne, such an approach may still produce sub-problems with an unduly large number of constraints. It was shown in [16] that, by means of a scheme inspired by the bundle-type methods of nondierentiable optimization (see, e.g. [11, 13]), the number of constraints used in the sub-problems can be further reduced without jeopardizing global convergence. Specically, in [16], the constraints to be used in the computation of the search direction d +1 at iteration +1 are chosen as follows. Let be the set of constraints used to compute the search direction d, and let x +1 be the next iterate. Then +1 includes: All 2 such that (x +1 ; ) = 0 (i.e. the \active" constraints), All 2 which aected the computation of the search direction d, and
3 FSQP for Finely Discretized SIP 3 A 2, if it exists, which caused a step reduction in the line search at iteration. While the former is obviously needed to ensure that d is a feasible direction, it is argued in [16] that the latter two are necessary to avoid zig-zagging or other jamming phenomena. The number of constraints required to compute the search direction is thus typically small compared to jj, hence each iteration of such a method is computationally less costly. Unfortunately, for a xed level of discretization, the algorithms in [19, 16] converge at best at a linear rate. Sequential Quadratic Programming (SQP)-type algorithms exhibit fast local convergence and are well-suited for problems in which the number of variables is not too large but the evaluation of objective/constraint functions and their gradients is costly. In such algorithms, quadratic programs (QPs) are used as models to construct the search direction. For an excellent recent survey of SQP algorithms, see [2]. A number of attempts at applying the SQP scheme to problems with a large number of constraints, e.g. our discretized problem from SIP, have been documented in the literature. In [1], Biggs treats all active inequality constraints as equality constraints in the QP sub-problem, while ignoring all constraints which are not active. Pola and Tits [20], and Mine et al. [14], apply to the SQP framewor an -active scheme similar to that used in [19]. Similar to the -active idea, Powell proposes a \tolerant" algorithm for linearly constrained problems in [22]. Finally, in [26], Schittowsi proposes another modication of the SQP scheme for problems with many constraints, but does not prove convergence. In practice, the algorithm in [26] may or may not converge, dependent upon the heuristics applied to choose the constraints for the QP sub-problem. In this paper, the scheme introduced in [16] in the context of rst-order feasible direction methods is extended to the SQP framewor, specically, the Feasible SQP (FSQP) framewor introduced in [17] (the qualier \feasible" signies that all iterates x satisfy the constraints, i.e. (x ; ) 0; for all 2 ). Our presentation and analysis signicantly borrow from [27], where an important special case of (DSI) is considered, the unconstrained minimax problem. Let the feasible set be denoted X = fx 2 IR n j (x; ) 0; 8 2 g:
4 4 Chapter 1 For x 2 X, ^, and H 2 IR nn with H = H T > 0, let d 0 (x; H; ^) be the corresponding SQP direction, i.e. the unique solution of the QP minimize 1 2 hd 0 ; Hd 0 i + hrf(x); d 0 i subject to (x; ) + hr x (x; ); d 0 i 0; 8 2 ^: QP 0 (x; H; ^) At iteration, given an estimate x 2 X of the solution, a constraint index set, and a symmetric positive denite estimate H of the Hessian of the Lagrangian, rst compute d 0 = d 0 (x ; H ; ). Note that d 0 may not be a feasible search direction, as required in the FSQP context, but that at worst it is tangent to the feasible set. Since all iterates are to remain in the feasible set, following [17], an essentially arbitrary feasible descent direction d 1 is computed and the search direction is taen to be the convex combination d = (1? )d 0 + d 1 : The coecient = (d 0 ) 2 [0; 1] goes to zero fast enough, as x approaches a solution, to ensure the fast convergence rate of the standard SQP scheme is preserved. An Armijo-type line search is then performed along the direction d, yielding a step-size t 2 (0; 1]. The next iterate is taen to be x +1 = x + t d. Finally, H is updated yielding H +1, and a new constraint index set +1 is constructed following the ideas of [16]. As is pointed out in [27], the construction of [16] cannot be used meaningfully in the SQP framewor without modifying the update rule for the new metric H +1. The reason is as follows. As noted above, following [16], if t < 1, +1 is to include, among others, the index 2 of a constraint which was infeasible for the last trial point in the line search. 1 The rationale for including in +1 is that if had been in, then it is liely that the computed search direction would have allowed a longer step. Such reasoning is clearly justied in the context of rst-order search directions as is used in [16], but it is not clear that is the right constraint to include under the new metric H +1. To overcome this diculty, it is proposed in [27] that H not be updated whenever t <, a prescribed small positive number, and 62. We will show in Section 3 that, as is the case for the minimax algorithm of [27], for large enough, will always be in, thus normal updating will tae place eventually, preserving the local convergence rate properties of the SQP scheme. There is an important additional complication, with the update of, which was not present in the minimax case considered in [27]. As just pointed out, any 2 which aected the search direction is to be included in +1. In [27] (unconstrained minimax problem) this is accomplished by including those objectives whose multipliers are non-zero in the QP used to compute the search 1 Assuming that it was a constraint, and not the objective function, which caused a failure in the line search.
5 FSQP for Finely Discretized SIP 5 direction (analogous to QP 0 (x ; H ; ) above), i.e. the \binding" objectives. In our case, in addition to the binding constraints from QP 0 (x ; H ; ), we must also include those constraints which aect the computation of the feasible descent direction d 1. If this is not done, convergence is not ensured and a \zigzagging" phenomenon as discussed in [16] could result. As a nal matter on the update rule for, following [27], we allow for additional constraint indices to be added to the set. While not necessary for global convergence, cleverly choosing additional constraints can signicantly improve performance, especially in early iterations. In the context of discretized SIP, exploiting the possible regularity properties of the SIP constraints with respect to the independent parameter can give useful heuristics for choosing additional constraints. In order to guarantee fast (superlinear) local convergence, it is necessary that, for large enough, the line search always accept the step-size t = 1. It is well-nown in the SQP context that the line search could truncate the step size arbitrarily close to a solution (the so-called Maratos eect), thus preventing superlinear convergence. Various schemes have been devised to overcome such a situation. We will argue that a second-order correction, as used in [17], will overcome the Maratos eect without sacricing global convergence. The balance of the paper is organized as follows. In Section 2 we introduce the algorithm and present some preliminary material. Next, in Section 3, we give a complete convergence analysis of the algorithm proposed in Section 2. The local convergence analysis assumes the just mentioned second-order correction is used. To improve the continuity of the development, a few of the proofs are deferred to an appendix. In Section 4, the algorithm is extended to handle the constrained minimax case. Some implementation details, in addition to numerical results, are provided in Section 5. Finally, in Section 6, we oer some concluding remars. 2 ALGORITHM We begin by maing a few assumptions that will be in force throughout. The rst is a standard regularity assumption, while the second ensures that the set of active constraint gradients is always linearly independent.
6 6 Chapter 1 Assumption 1: The functions f : IR n! IR and (; ) : IR n! IR, 2 are continuously dierentiable. Dene the set of active constraints for a point x 2 X as act (x) = f 2 j (x; ) = 0g: Assumption 2: For all x 2 X with act (x) 6= ;, the set fr x (x; ) j 2 act (x)g is linearly independent. A point x 2 IR n is called a Karush-Kuhn-Tucer (KKT) point for the problem (DSI) if there exist KKT multipliers, 2, satisfying 8 >< >: rf(x ) + X 2 r x (x ; ) = 0; (x ; ) 0; 8 2 ; (x ; ) = 0 and 0; 8 2 : (1.1) Under our assumptions, any local minimizer x for (DSI) is a KKT point. Thus, (1.1) provides a set of rst-order necessary conditions of optimality. Throughout our analysis, we will often mae use of the KKT conditions for QP 0 (x; H; ^). Specically, given x 2 X, H = H T > 0, and ^, d 0 is a KKT point for QP 0 (x; H; ^) if there exist multipliers 0 ; 2 ^, satisfying 8 >< >: Hd 0 + rf(x) + X 2^ 0 r x (x; ) = 0; (x; ) + hr x (x; ); d 0 i 0; ^;? (x; ) + hrx (x; ); d 0 i = 0 and 0 0; 8 2 ^: (1.2) In fact, since the objective for QP 0 (x; H; ^) is strictly convex, such a d 0 is the unique KKT point, as well as the unique global minimizer (stated formally in Lemma 1 below). As noted above, d 0 need not be a feasible direction. The search direction d will be taen as a convex combination of d 0 and a rst-order feasible descent direction d 1. For x 2 X and ^, we compute d 1 = d 1 (x; ^), and = (x; ^),
7 FSQP for Finely Discretized SIP 7 as the solution of the QP minimize 1 2 d1 2 + subject to hrf(x); d 1 i ; (x; ) + hr x (x; ); d 1 i ; 8 2 ^: QP 1 (x; ^) The notation will be used throughout to denote the standard Euclidean norm. The pair (d 1 ; ) is a KKT point for QP 1 (x; ^) if there exist multipliers 1 and 1, 2 ^, satisfying 8 d 1 >< >: rf(x)?1 hrf(x); d 1 i ; (x; ) + hr x (x; ); d 1 i ; X + 1 rx (x; )?1 2^ 8 2 ^; = 0;? 1 hrf(x); d 1 i? = 0 and 1 0;? (x; ) + hrx (x; ); d 1 i? = 0 and 1 0; 8 2 ^: 1 (1.3) In Section 1 we stated that the feasible descent direction d 1 was essentially arbitrary. In the subsequent analysis we assume that d 1 is chosen specically as the solution of QP 1 (x; ^), though it can be shown that the results still hold if some minor variation is used. To be precise, following [17], we require that d 1 = d 1 (x; ^) satisfy: d 1 (x; ^) = 0 if x is a KKT point, hrf(x); d 1 (x; ^)i < 0 if x is not a KKT point, hr x (x; ); d 1 (x; ^)i < 0, for all 2 act if x is not a KKT point, and for ^ xed, d 1 (x; ^) is bounded over bounded subsets of X. It will be shown in Lemma 2 that the solution of QP 1 (x; ^) satises these requirements. In our context, d 1 must fulll one additional property, which is essentially captured by Lemma 7 in the appendix. Thus, at iteration, the search direction d is taen as a convex combination of d 0 and d1, i.e. d = (1? )d 0 + d 1, 2 [0; 1]. In order to guarantee a fast local rate of convergence while providing a suitably feasible search direction, we require the coecient of the convex combination = (d 0 ) to satisfy certain properties. Namely, () : IR n! [0; 1] must satisfy
8 8 Chapter 1 (d 0 ) is bounded away from zero outside every neighborhood of zero, and (d 0 ) = O(d 0 2 ): For example, we could tae (d 0 ) = minf1; d 0 g, where 2. It remains to explicitly specify the ey feature of the proposed algorithm: the update rule for. As discussed above, following [16], +1 will include (in addition to possible heuristics) three crucial components. The rst one is the set act (x +1 ) of indices of active constraints at the new iterate. The second component of +1 is the set b of indices of constraints that aected d. In particular, b will include all indices of constraints in QP 0 (x ; H ; ) and QP 1 (x ; ) which have positive multipliers, i.e. the binding constraints for these QPs. Specically, let 0 ;, and 1 ;, for 2, be the QP multipliers from QP 0 (x ; H ; ) and QP 1 (x ; ), respectively. Dening we let b;0 = f 2 j 0 ; > 0 g; b;1 = f 2 j 1 ; > 0 g; b = b;0 [ b;1 : Finally, the third component of +1 is the index of one constraint, if any exists, that forced a reduction of the step in the previous line search. While the exact type of line search we employ is not critical to our analysis, we assume from this point onward that it is an Armijo-type search. That is, given constants 2 (0; 1=2) and 2 (0; 1), the step-size t is taen as the rst number t in the set f1; ; 2 ; : : :g such that and f(x + td ) f(x ) + thrf(x ); d i; (1.4) (x + td ; ) 0; 8 2 : (1.5) Thus, t < 1 implies that either (1.4) or (1.5) is violated at x + t d. In the event that (1.5) is violated, there exists 2 such that x + t d ; > 0; (1.6) and in such a case we will include in +1.
9 FSQP for Finely Discretized SIP 9 Algorithm FSQP-MC Parameters. 2 (0; 1 ), 2 (0; 1), and 0 < 1. 2 Data. x 0 2 X, 0 < H 0 = H T 0 2 IR nn. Step 0 - Initialization. Set 0 and choose 0 act (x 0 ): Step 1 - Computation of search direction. (i). Compute d 0 = d 0 (x ; H ; ). If d 0 = 0, stop. (ii). Compute d 1 = d1 (x ; ). (iii). Compute = (d 0 ) and set d (1? )d 0 + d 1. Step 2 - Line search. Compute t, the rst number t in the sequence f1; ; 2 ; : : :g satisfying (1.4) and (1.5). Step 3 - Updates. (i). Set x +1 x + t d. (ii). If t < 1 and (1.5) was violated at x + t d, then let be such that (1.6) holds. (iii). Pic +1 act (x +1 ) [ b : If t < 1 and (1.6) holds for some 2, then set [ f g: (iv). If t and 62, set H +1 H. Otherwise, obtain a new symmetric positive denite estimate H +1 to the Hessian of the Lagrangian. (v). Set + 1 and go bac to Step 1. 3 CONVERGENCE ANALYSIS While there are some critical dierences, the analysis in this section closely parallels that of [27]. We begin by establishing that, under a few additional assumptions, algorithm FSQP-MC generates a sequence which converges to a KKT point for (DSI). Then, upon strengthening our assumptions slightly, we show that the rate of convergence is two-step superlinear.
10 10 Chapter Global Convergence The following will be assumed to hold throughout our analysis. Assumption 3: The level set f x 2 IR n j f(x) f(x 0 ) g \ X is compact. Assumption 4: There exist scalars 0 < 1 2 such that for all, 1 d 2 hd; H di 2 d 2 ; 8d 2 IR n : Given the scalars 0 < 1 2 from Assumption 4, dene H = fh = H T j 1 d 2 hd; Hdi 2 d 2 ; 8d 2 IR n g: First, we derive some properties of d 0 (x; H; ^) and d 1 (x; ^). Lemma 1 For all x 2 X, H 2 H, and ^ such that act (x) ^, the search direction d 0 = d 0 (x; H; ^) is well-dened, is continuous in x and H for ^ xed, and is the unique KKT point for QP 0 (x; H; ^). Furthermore, d 0 = 0 if, and only if, x is a KKT point for the problem (DSI). If x is not a KKT point for this problem, then d 0 satises hrf(x); d 0 i < 0; (1.7) hr x (x; ); d 0 i 0; 8 2 act (x): (1.8) Proof: H > 0 implies that QP 0 (x; H; ^) is strictly convex. Further d 0 = 0 is always feasible for the QP constraints. It follows that the solution d 0 is well-dened and the unique KKT point for the QP. As the set H is uniformly positive denite, continuity in x and H for xed ^ is a direct consequence of Theorem 4.4 in [4]. Now suppose d 0 is 0 and let f 0 j 2 ^ g be the QP multipliers. In view of the KKT conditions (1.2) for QP 0 (x; H; ^), since d 0 = 0 and x 2 X, we see that x satises the KKT conditions (1.1) for (DSI) with = ( 0 ; 2 ^; 0; 62 ^: The converse is proved similarly, appealing to the uniqueness of the KKT point for QP 0 (x; H; ^) and the fact that act (x) ^. Finally, since act (x) ^, (1.7) and (1.8) follow from Proposition 3.1 in [17]. 2
11 FSQP for Finely Discretized SIP 11 Lemma 2 For all x 2 X, and ^ such that act (x) ^, the direction d 1 = d 1 (x; ^) is well-dened and the pair (d 1 ; ), where = (x; ^), is the unique KKT point of QP 1 (x; ^). Furthermore, for given ^, d 1 (x; ^) is bounded over bounded subsets of X. In addition, d 1 = 0 if, and only if, x is a KKT point for the problem (DSI). If x is not a KKT point for this problem, then d 1 satises hrf(x); d 1 i < 0; (1.9) and satises < 0. hr x (x; ); d 1 i < 0; 8 2 act (x); (1.10) Proof: We begin by noting that (d 1 ; ) solves QP 1 (x; ^) if, and only if, d 1 solves minimize 1 2 d1 2 + max hrf(x); d 1 i; maxf(x; ) + hr x (x; ); d 1 ig : 2^ and = max hrf(x); d 1 i; maxf(x; ) + hr x (x; ); d 1 ig 2^ : (1.11) Since the objective function in (1.11) is strictly convex and radially unbounded, it follows that QP 1 (x; ^) has (d 1 ; ) = (d 1 (x; ^); (x; ^)) as its unique global minimizer. Since QP 1 (x; ^) is convex, (d 1 ; ) is also its unique KKT point. Boundedness of d 1 (x; ^) over bounded subsets of X follows from the rst equation of the optimality conditions (1.3), noting that the QP multipliers must all lie in [0; 1]. Now suppose d 1 = 0. Since x 2 X, it is clear that = 0. Substitute d 1 = 0 and = 0 into (1.3) and let 1 2 IR and 1 2 IR j^j be the corresponding multipliers. Note that, in view of Assumption 2, 1 > 0. Since x 2 X, it follows that x satises (1.1) with multipliers = ( 1 =1 ; 2 ^; 0; 62 ^: Therefore, x is a KKT point for (DSI). The converse is proved similarly, appealing to uniqueness of the KKT point for QP 1 (x; ^), and the fact that act (x) ^. To prove (1.9) and (1.10) note that if x is not a KKT point for (DSI), then as just shown d 1 6= 0, hence < 0 (since d 1 = 0 and = 0 form a feasible pair, the optimal value of QP 1 (x ; ) must be non-positive). The result then follows directly from the form of the QP constraints and the fact that act (x) ^. 2 Next, we establish that the line search is well-dened.
12 12 Chapter 1 Lemma 3 For each, the line search in Step 2 yields a step t = j for some nite j = j(). Proof: If x is a KKT point, then Step 2 is not reached, hence assume x is not KKT. In view of Lemma 1 and the properties of (), d 0 6= 0 and (d 0 ) > 0. Lemmas 1 and 2 imply, since act (x ), that hrf(x ); d i < 0; hr x (x ; ); d i < 0; 8 2 act (x ): Further, feasibility of x requires (x ; ) < 0 for all 2 n act (x ). The result then follows by considering rst order expansions of f(x + t d ) and (x +t d ; ), 2, about x and by appealing to our regularity assumptions. 2 The previous three lemmas imply that the algorithm is well-dened. In addition, Lemma 1 shows that if Algorithm FSQP-MC generates a nite sequence terminating at the point x N, then x N is a KKT point for the problem (DSI). We now concentrate on the case in which the algorithm never satises the termination condition in Step 1(i) and generates an innite sequence fx g. Given an innite index set K, we use the notation to mean 2K x?! x x! x as! 1; 2 K: Lemma 4 Let K be an innite index set such that = for all 2 K, 2K x?! x 2K, H?! H, d 0 2K?! d 0;, d 1 2K?! d 1; 2K, and?!. Then (i) d 0; is the unique solution of QP 0 (x ; H ; ), and (ii) (d 1; ; ) is the unique solution of QP 1 (x ; ). Proof: Part (i) follows from continuity of d 0 (x; H; ^) for xed ^ (Lemma 1). To prove part (ii), recall that in view of Lemma 2, (d 1 ; ) is the unique KKT point for QP 1 (x ; ), i.e. is the unique solution of (1.3) with corresponding multipliers 1 0 and 1 ; 0, 2. Note that the multipliers satisfy 1 + X 2 1 ; = 1; for all, hence are bounded. Let K 0 K be an innite index set such that 1 2K 0?! 1; and 1 2K 0 ;?! 1;, 2. Taing limits in the optimality conditions (1.3) shows that (d 1; ; ) is a KKT point for QP 1 (x ; ) with multipliers 1; and 1;, 2. Uniqueness of such points proves the result. 2
13 FSQP for Finely Discretized SIP 13 The following lemma establishes a few basic properties of some of the sequences generated by the algorithm. Lemma 5 (i) The sequences fx g, fd 0 g, and fd g are bounded; (ii) ff(x )g converges; (iii) t d?! 0. Proof: In view of Assumption 3 and the fact that the line search guarantees ff(x )g is a monotonically decreasing sequence, it follows that fx g is bounded, and since f() is continuous, that ff(x )g converges. Boundedness of fd 0 g follows from boundedness of fx g, Assumption 4, continuity of d 0 (x; H; ^) for xed ^, and the fact that there are only nitely many subsets ^ of. Boundedness of fd 1 g follows from Lemma 2 and boundedness of fx g. Since 2 [0; 1], fd g is bounded as well. Finally, suppose ft d g 6! 0. Then there exists an innite index set K IN such that t d is bounded away from zero on K. Since all sequences of interest are bounded and is nite, we may suppose 2K without loss of generality that x?! x 2K, H?! H, d 0 2K?! d 0;, d 1 2K?! d 1;, 2K?! 2K,?!, and = for all 2 K. Lemma 4 tells us that d 0; is the unique solution of QP 0 (x ; H ; ) and (d 1; ; ) is the unique solution pair for QP 1 (x ; ). Furthermore, since t d is bounded away from zero on K, there exists t > 0 such that t t for all 2 K, and since ft g is bounded (t 2 [0; 1]), it follows that either d 0; 6= 0 or d 1; 6= 0. Applying Lemmas 1 and 2 in both directions shows that x is not a KKT point for (DSI) and both d 0; 6= 0 and d 1; 6= 0. In addition, < 0 and < 0, for all 2 K (from Lemma 2) and is bounded away from zero on K (from the assumptions on () as given in Section 2 and the fact that d 0 is bounded away from zero on K). As a consequence, there exists > 0 such that = (d 0 ) for all 2 K and there exists < 0 such that for all 2 K. Now, since hrf(x ); d 1 i, for all, t hrf(x ); d i = t (1? )hrf(x ); d 0 i + t hrf(x ); d 1 i t t < 0; for all 2 K, where we have used (1.7). Thus, by the line search criterion of Step 2, f(x +1 ) f(x ) + t for all 2 K. Since f(x ) is monotone non-increasing, it follows that ff(x )g diverges, which contradicts (ii). 2 In order to establish convergence to a KKT point, it will be convenient to consider the value functions for the search direction QPs, QP 0 (x; H; ^) and
14 14 Chapter 1 QP 1 (x; ^). In particular, given the solutions d 0 = d 0 (x; H; ^) and (d 1 ; ) = (d 1 (x; ^); (x; ^)), dene v 0 (x; H; ^) = 1? 2 hd 0 ; Hd 0 i + hrf(x); d 0 i ; v 1 (x; ^) = 1? 2 d1 2 + ^ : Further, let v(x; H; ^) = v 0 (x; H; ^) + v 1 (x; ^); and, at iteration, dene v 0 = v0 (x ; H ; ), v 1 = v1 (x ; ), and v = v(x ; H ; ). Note that, since 0 is always feasible for both QPs, v 0 0 and v1 0, for all. Lemma 6 Let K be an innite index set. Then (i) d 0 2K?! 0 if and only if v 0 2K?! 0, (ii) (d 1 ; )?! 2K (0; 0) if and only if v 1 2K?! 0, and (iii) if d 0 2K?! 0, then all accumulation points of fx g 2K are KKT points for (DSI). Proof: First, if d 0 2K?! 0 then it is clear from the denition of v 0 that v 0 2K?! 0. Next, from (1.2) and the last statement in Lemma 1, it follows that hrf(x ); d 0 i?hd0 ; H d 0 i. Thus, using again the denition of v0, we get v 0 =?? 1 2 hd0 ; H d 0 i + hrf(x ); d 0 i? 1 2 hd0 ; H d 0 i + hd0 ; H d 0 i 1 2 d0 2 > 0; where we have used Assumption 4. Thus, if v 0 2K?! 0, then d 0 2K?! 0. If (d 1 ; )?! 2K (0; 0), then from the denition of v 1 we see that v1 2K?! 0. Now suppose v 1 2K?! 0: To show d 1 2K?! 0, note that from the optimality conditions (1.3), X d 1 2 =? 1 hrf(x ); d 1 i? 1 ; hr x(x ; ); d 1 i X 2 =? 1? 1 ;(? (x ; )) X 2 =? + 1 ;(x ; )? : 2 Thus, again using the denition of v 1, v 1 =? 1 2 d1 2? 1 2 d1 2 ;
15 FSQP for Finely Discretized SIP 15 and it immediately follows that d 1 2K 2K?! 0 and?! 0. To prove (iii), suppose K is such that d 0 2K?! 0. Let x be an accumulation point of fx g 2K and let K 0 2K K be an innite index set such that x 0?! x and, for some ^, = ^ for all 2 K 0. Let 0 2 IRj^j be the multiplier vector from QP 0 (x ; H ; ^) and dene 0 = f 2 ^ j (x ; ) + hr x (x ; ); d 0 i = 0 g: Suppose, without loss of generality, 0 = ^ 0 ; for all 2 K 0. As d 0 0, it is clear that ^ 0 act (x ) and, in view of Assumption 2, the set f r x (x ; ) j 2 ^ 0 g is linearly independent, for large enough. Thus, from (1.2), a unique expression for the QP multipliers (for large enough) is given by 0 =? ^R(x ) T ^R(x )?1 ^R(x ) T? H d 0 + rf(x ) ; 2K 0?! where ^R(x ) = [ r x (x ; ) j 2 ^ 0 ] 2 IR nj^ 0j. In view of Assumption 4, boundedness of fx g, the regularity assumptions, and the fact that d 0 2K?! 0 0, we see that 0 2K 0?! 0; =? ^R(x ) T ^R(x )?1 ^R(x ) T rf(x ): Taing limits in the optimality conditions (1.2) for QP 0 (x ; H ; ^) shows that x is a KKT point for (DSI) with multipliers = ( 0; 2 ^; 0 62 ^: We are now in a position to show that there exists an accumulation point of fx g which is a KKT point for (DSI). This result is, in fact, weaer than that obtained in [27] for the unconstrained minimax case, where under similar assumptions, but with a more involved argument, it is shown that all accumulation points are KKT. The price to be paid is the introduction of Assumption 5 below for proving Theorem 1. The proof of the following result is inspired by that of Theorem T in [16]. 2 Proposition 1 lim inf v = 0:
16 16 Chapter 1 Corollary 1 There exists an accumulation point x of fx g which is a KKT point for (DSI). Proof: Since v 0 0 and v1 0 for all, Proposition 1 implies lim inf v0 = 0; i.e. there exists an innite index set K such that v 0 2K?! 0. In view of Lemma 6, all accumulation points of fx g 2K are KKT points. Finally, boundedness of fx g implies at least one such point exists. 2 Dene the Lagrangian function for (DSI) as L(x; ) = f(x) + X 2 (x; ): In order to show that the entire sequence converges to a KKT point x, we strengthen our assumptions as follows. Assumption 1 0 : The functions f : IR n! IR and (; ) : IR n! IR, 2 are twice continuously dierentiable. Assumption 5: Some accumulation point x of fx g which is a KKT point for (DSI) also satises the second order suciency conditions with strict complementary slacness, i.e. there exists 2 IR jj satisfying (1.1) as well as r 2 xxl(x ; ) is positive denite on the subspace and > 0 for all 2 act(x ). fh j hr x (x ; ); hi = 0; 8 2 act (x )g; It is well-nown that such an assumption implies that x is an isolated KKT point for (DSI) as well as an isolated local minimizer. The following theorem is the main result of this section. Theorem 1 The sequence fx g generated by algorithm FSQP-MC converges to a strict local minimizer x of (DSI). Proof: First we show that there exists a neighborhood of x in which no other accumulation points of fx g can exist, KKT points or not. As x is a strict local minimizer, there exists > 0 such that f(x) > f(x ) for all x 6= x, x 2 S = B(x ; ) \ X, where B(x ; ) is the open ball of radius centered at
17 FSQP for Finely Discretized SIP 17 x. Proceeding by contradiction, suppose x 0 2 B(x ; ), x 0 6= x, is another accumulation point of fx g. Feasibility of the iterates implies that x 0 2 S. Thus f(x 0 ) > f(x ), which is in contradiction with Lemma 5(ii). Next, in view of Lemma 5(iii), (x +1? x )?! 0. Suppose K is an innite index set such 2K that x?! x. Then there exists 1 such that x? x < =4, for all 2 K, 1. Further, there exists 2 such that x +1? x < =4, for all > 2. Therefore, if there were another accumulation point outside of B(x ; ), then the sequence would have to pass through the compact set B(x ; ) n B(x ; =4) innitely many times. This contradicts the established fact that there are no accumulation points of fx g, other than x, in B(x ; ) Local Convergence We have thus shown that, with a liely dramatically reduced amount of wor per iteration, global convergence can be preserved. This would be of little interest, though, if the speed of convergence were to suer signicantly. In this section we establish that, under a few additional assumptions, the sequence fx g generated by a slightly modied version of algorithm FSQP-MC (to avoid the Maratos eect) exhibits 2-step superlinear convergence. To do this, the bul of our eort is focussed on showing that for large the set of constraints b which aect the search direction is precisely the set of active constraints at the solution, i.e. act (x ). In addition, we show that, eventually, no constraints outside of act (x ) aect the line search, and that H is updated normally at every iteration. Thus, for large enough, the algorithm behaves as if it were solving the problem minimize f(x) subject to (x; ) 0; 8 2 act (x ); (P ) using all constraints at every iteration. Establishing this allows us to apply nown results concerning local convergence rates. The following is proved in the appendix. Proposition 2 For large enough, (i) b;0 = b;1 = act (x ), and (ii) (x + td ; ) 0 for all t 2 [0; 1], 2 n act (x ).
18 18 Chapter 1 In order to achieve superlinear convergence, it is crucial that a unit step, i.e. t = 1, always be accepted for all suciently large. Several techniques have been introduced to avoid the so-called Maratos eect. We chose to include a second order correction such as that used in [17]. Specically, at iteration, let ~ d(x; d; H; ^) be the unique solution of the QP g QP (x; d; H; ^), dened for 2 (2; 3) as follows 1 minimize hd + ~ 2 d; H(d + d)i ~ + hrf(x); d + di ~ subject to (x + d; ) + hr x (x; ); d + di ~ d ; 8 2 ^; gqp (x; d; H; ^) if it exists and has norm less that minfd; Cg, where C is a large number. Otherwise, set d(x; ~ d; H; ^) = 0. The following step is added to algorithm FSQP-MC: Step 1(iv). Compute ~ d = ~ d(x ; d ; H ; ). In addition, the line search criterion (1.4) and (1.5) are replaced with and f(x + td + t 2 ~ d ) f(x ) + thrf(x ); d i; (1.12) Finally, the condition (1.6) is replaced with x + t d t + (x + td + t 2 ~ d ) 0; 8 2 : (1.13) 2 ~d ;! > 0: (1.14) With some eort, it can be shown that these modications do not aect any of the results obtained to this point. Further, for suciently large, the set of binding constraints in QP g (x ; d ; H ; ) is again act (x ). Hence, it is established that for large enough, the modied algorithm FSQP-MC behaves identically to that given in [17], applied to (P ). Assumption 1 is now further strengthened and a new assumption concerning the Hessian approximations H is given. These assumptions allow us to use the local convergence rate result from [17]. Assumption 1 00 : The functions f : IR n! IR, and (; ) : IR n! IR, 2, are three times continuously dierentiable. Assumption 6: As a result of the update rule chosen for Step 3(iv), H approaches the Hessian of the Lagrangian in the sense that P (H? r 2 xx lim L(x ; ))P d = 0;!1 d
19 FSQP for Finely Discretized SIP 19 where is the KKT multiplier vector associated with x and with R = [r x (x ; ) j 2 act (x )]. P = I? R (R T R )?1 R T Theorem 2 For all suciently large, the unit step t = 1 is accepted in Step 2. Further, the sequence fx g converges to x 2-step superlinearly, i.e. x +2? x lim!1 x? x = 0: 4 EXTENSION TO CONSTRAINED MINIMAX The algorithm we have discussed may be extended following the scheme of [27] to handle problems with many objective functions, i.e. large-scale constrained minimax. Specically, consider the problem minimize max f(x;!)!2 subject to (x; ) 0; 8 2 ; where and are nite (again, presumably large) sets, and f : IR n! IR and : IR n! IR are both three times continuously dierentiable with respect to their rst argument. Given ^, dene F ^(x) = max f(x;!):!2 ^ Given a direction d 2 IR n, dene a rst-order approximation of F ^(x + d)? F ^(x) by F 0^(x; d) = maxff(x + d;!) + hr x f(x;!); dig? F ^(x);!2 ^ and, nally, given a direction ~ d 2 IR n, let ~F 0^(x; d; d) ~ = maxff(x + d;!) + hr x f(x;!); dig ~? F ^(x + d):!2 ^
20 20 Chapter 1 Let be the set of objective functions used to compute the search direction at iteration. The modied QPs follow. To compute d 0 (x; H; ^; ^), we solve minimize 1 2 hd 0 ; Hd 0 i + F 0^(x; d 0 ) subject to (x; ) + hr x (x; ); d 0 i 0; 8 2 ^; QP 0 (x; H; ^; ^) and to compute d 1 (x; ^; ^), we solve minimize 1 2 d1 2 + subject to F 0^(x; d 1 ) ; (x; ) + hr x (x; ); d 1 i ; Finally, to compute ~ d(x; d; H; ^; ^), we solve 8 2 ^: QP 1 (x; ^; ^) minimize 1 2 hd + ~ d; H(d + ~ d)i + ~ F 0^(x; d; ~ d) subject to (x + d; ) + hr x (x; ); d + ~ di d ; 8 2 ^; gqp (x; d; H; ^; ^) where again, if the QP has no solution or if the solution has norm greater than minfd; Cg, we set ~ d(x; d; H; ^; ^) = 0. In order to describe the update rules for, following [27], we dene a few index sets for the objectives (in direct analogy with the index sets for the constraints as introduced in Section 2). The set of indices of \maximizing" objectives is dened in the obvious manner as max(x) = f! 2 j f(x;!) = F (x)g: At iteration, let 0 ;!,! 2, be the multipliers from QP 0 (x ; H ; ; ) associated with the objective functions. Liewise, let 1 ;!,! 2, be the multipliers from QP 1 (x ; ; ) associated with the objective functions. The set of indices of objective functions which aected the computation of the search direction d is given by 2 b = f! 2 j 0 ;! > 0 or 1 ;! > 0g: The line search criterion (1.12) is replaced with F (x + td + t 2 ~ d ) F (x ) + tf 0 (x ; d ): (1.15) 2 QP 1 (x; ^; ^) is not needed in the unconstrained case. Accordingly, in [27], b is dened based on a single set of multipliers.
21 FSQP for Finely Discretized SIP 21 If t < 1 and the truncation is due to an objective function, then dene! 2 such that f x + t d + 2 t ~d ;!! > F (x ) + t F (x 0 ; d ): (1.16) We are now in a position to state the extended algorithm. Algorithm FSQP-MOC Parameters. 2 (0; 1 ), 2 (0; 1), and 0 < 1. 2 Data. x 0 2 IR n, 0 < H 0 = H T 0 2 IR nn. Step 0 - Initialization. Set 0. Choose 0 max(x 0 ), 0 act (x 0 ): Step 1 - Computation of search direction. (i). Compute d 0 = d 0 (x ; H ; ; ). If d 0 = 0, stop. (ii). Compute d 1 = d1 (x ; ; ). (iii). Compute = (d 0) and set d (1? )d 0 + d 1. (iv). Compute ~ d = ~ d(x ; d ; H ; ; ). Step 2 - Line search. Compute t, the rst number t in the sequence f1; ; 2 ; : : :g satisfying (1.15) and (1.13). Step 3 - Updates. (i). Set x +1 x + t d + t 2 ~ d. (ii). If t < 1 and (1.15) was violated at x +1 = x + t d + t 2 ~d ; then let! be such that (1.16) holds. If (1.13) was violated at x +1, then let be such that (1.14) holds. (iii). Pic +1 max(x +1 ) [ b ; +1 act (x +1 ) [ b : and If t < 1 and (1.16) holds for some! 2, then set [ f!g: If t < 1 and (1.14) holds for some 2, then set [ fg: (iv). If t and! 62 or 62 set H +1 H. Otherwise, obtain a new symmetric positive denite estimate H +1 to the Hessian of the Lagrangian. (v). Set + 1 and go bac to Step 1.
22 22 Chapter 1 5 IMPLEMENTATION AND NUMERICAL RESULTS Algorithm FSQP-MOC has been implemented as part of the code cfsqp 3 [12]. The numerical test results reported in this section were obtained with a modied copy of cfsqp Version 2.4 (the relevant changes will be included in subsequent releases, beginning with Version 2.5). All test problems we consider here are instances of (DSI). Thus in this section we only discuss implementation details relevant to solving such problems, i.e. implementation details of algorithm FSQP-MC modied to include the second order correction ~ d. 4 The implementation allows for multiple discretized SIP constraints and contains special provisions for those which are ane in x. Specically, problem (DSI) is generalized to minimize f(x) subject to j (x; ) = hc j (); xi? d j () 0; 8 2 (j) ; j = 1; : : : ; m`; j (x; ) 0; 8 2 (j) ; j = m` + 1; : : : ; m; where c j : (j) `?! IR n, j = 1; : : : ; m`, d j : (j) `?! IR, j = 1; : : : ; m`, and (j) is nite, j = 1; : : : ; m. The assumptions and algorithm statement are generalized in the obvious manner. As far as the analysis of Section 3 is concerned, such a formulation could readily be adapted to the format of (DSI) by grouping all constraints together, i.e. letting = [ m j=1 (j), and ignoring the fact that some may be ane in x. In the case that the initial point x 0 provided is not feasible for the ane constraints, cfsqp rst computes v 2 IR n as the solution of the strictly convex QP minimize hv; vi subject to hc j (); x 0 + vi? d j () 0; 8 2 (j) ; j = 1; : : : ; m`; for ^v such that x 0 + ^v is feasible for linear constraints. If the new initial point is not feasible for all nonlinear constraints, then cfsqp iterates, using the algorithm FSQP-MOC, on the constrained minimax problem minimize max j=m`+1;:::;m max 2 (j) f j(x; )g subject to hc j (); xi? d j () 0; 8 2 (j) ` ; j = 1; : : : ; m`; 3 Available from the authors. See 4 That is, we will not discuss the implementation details specic to the minimax algorithm, even though in the case that the initial guess is infeasible for nonlinear constraints, the minimax algorithm is used to generate a feasible point.
23 FSQP for Finely Discretized SIP 23 until max max j=m`+1;:::;m 2 (j)f j(x; )g 0 is achieved. The nal iterate will be feasible for all constraints, allowing the algorithm to be applied to the original problem. Recall that it is only required that contain certain subsets of. The algorithm allows for additional elements of to be included in order to speed up initial convergence. Of course, there is a trade-o between speeding up initial convergence and increasing (i) the number of gradient evaluations and (ii) the size of the QPs. In the implementation, heuristics are applied to add potentially useful elements to (see, e.g. [26] for a discussion of such heuristics). In the case of discretized SIP, one may wish to exploit the nowledge that adjacent discretization points are liely to be closely related. Following [27, 16, 6], for some > 0, the cfsqp implementation includes in the set ``m (x ) of -active \left local maximizers" at x. At a point x 2 X, for j = 1; : : : ; m, dene the -active discretization points as (j) (x) = f 2 (j) j j (x; )?g: Such a discretization point (j) i 2 (j) = f (j) 1 ; : : : ; (j) j (j) jg is a left local maximizer if it satises one of the following three conditions: (i) i 2 f2; : : : ; j (j) j?1g and and j (x; (j) i ) > j (x; (j) i?1 ) (1.17) j (x; (j) i ) j (x; (j) i+1 ); (1.18) (ii) i = 1 and (1.18); (iii) i = j (j) j and (1.17). The set ``m (x) is the set of all left local maximizers in (x) = [ m j=1 (j) (x): The rst part of the update (i.e. before updates due to line search violations) in Step 3(iii) of the algorithm becomes +1 = act (x ) [ b [ ``m (x ): Finally, we have found that in practice, including the end-points (whether or not they are close to being active) during the rst iteration often leads to a better initial search direction. Thus we set 0 1 [ 0 = act (x 0 ) [ ``m (x 0 ) m (f (j) 1 g [ f(j) j (j) j g) A : A few other specics of the cfsqp implementation are as follows. First, as was discussed in Section 2, it is not required to use QP 1 (x; ^) to compute our j=1
24 24 Chapter 1 feasible descent direction d 1. In fact, at iteration, cfsqp uses the following QP, which is a function of the SQP direction d 0, minimize 2 d0? d1 2 + subject to hrf(x ); d 1 i ; (x ; ) + hr x (x ; ); d 1 i ; 8 2 ; where was set to 0:1 for our numerical tests. Using such an objective function encourages d 1 to be \close" to d 0, a condition we have found to be benecial in practice. It can be veried that the arguments given in Section 3 go through with little modication if we disable the inclusion of d 0 in the QP objective function when the step size t?1 from the previous iteration is less than a given small threshold. 5 The expression used for is given by = d 0 d 0 + ; where = maxf0:5; d 1 g, with = 2:1 and = 2:5 for our numerical experiments. The matrices H are updated using the BFGS formula with Powell's modication [21]. The multiplier estimates used for the updates are those obtained from QP 0 (x ; H ; ), with all multipliers corresponding to discretization points outside of set to zero. The QP sub-problems were solved using the routine QLD due to Powell and Schittowsi [25]. Finally, the following parameter values were used for all numerical testing: = 0:1, = 0:5, = 1, and was set to the square root of the machine precision. In order to judge the eciency of algorithm FSQP-MOC, we ran the same numerical tests with two other algorithms diering only in the manner in which is updated. The results are given in Tables 1 and 2. All test algorithms were implemented by maing the appropriate modications to cfsqp Version 2.4. In the tables, the implementation of FSQP-MOC just discussed is denoted NEW. A simple -active strategy was employed in the algorithm we call -ACT, i.e. we set = (x ) for all, where = 0:1. The standard FSQP scheme of [17] was applied in algorithm FULL by simply setting =, for all. All three algorithms were set to stop when d ?4. We report the numerical results for 13 discretized SIP test problems with discretization levels of 101 and 501. A uniform discretization was used in all cases. Problems cw 3, cw 5, and cw 6 are borrowed from [3]. Problems oet 1 through oet 7 are from [15]. Finally, hz 1 is from [10] and sch 3 is from [26]. In all cases except for oet 7, the initial guess x 0 is the same as that given 5 In the numerical experiments reported here, t remained bounded away from 0.
25 FSQP for Finely Discretized SIP 25 in the reference from which the problem was taen. For oet 7, -ACT and FULL had diculty generating a feasible point, thus we used the feasible initial guess x 0 = (0; 0; 0;?7;?3;?1; 3). The rst two columns of the tables are self-explanatory. A description of the remaining columns is as follows. The third column, n, indicates the number of variables, while m` and m n in the next two columns indicate the number of linear SIP constraints and nonlinear SIP constraints (m n = m? m`), respectively. Next, NF is the number of objective function evaluations, NG is the number of \scalar" constraint function evaluations (i.e. evaluation of some j (x; ) for a given x and ), and IT indicates the number of iterations required before the stopping criterion was satised. Finally, P j j is the sum over all iterations of the size of (it is equal to the number of gradient evaluations in the case of NEW and FULL), j j is the size of at the nal iterate, and TIME is the time of execution in seconds on a Sun Sparc 4 worstation. For all test problems and for all three algorithms, the value of the objective function at the nal iterate agreed (to within four signicant gures) with the optimal value as given in the original references. A few conclusions may be drawn from the results. In general, NEW requires the most iterations to \converge" to a solution, whereas FULL requires the least. Typically, though, the dierence is not large. Of course, such behavior is expected since NEW uses a simpler QP model at each iteration. It is clear from comparing the results for P j j that NEW provides signicant savings in the number of gradient evaluations and the size of the QP sub-problems. The savings for -ACT are not as dramatic. In almost all cases, comparing TIME of execution conrms that, indeed, NEW requires far less computational eort than either of the other two approaches. Further, note that j j remains, in general, unchanged for NEW when the discretization level is increased and is typically equal to, or less than, n. This is not the case for -ACT and FULL, as would be expected. Such behavior suggests that computational eort does not increase with respect to discretization level for NEW at the same rate as it does for -ACT and FULL. This conclusion is supported by the increase in execution TIME when discretization level increases.
26 26 Chapter 1 P PROB ALGO n m` m n NF NG IT j j j j TIME cw 3 NEW ACT FULL cw 5 NEW ACT FULL cw 6 NEW ACT FULL oet 1 NEW ACT FULL oet 2 NEW ACT FULL oet 3 NEW ACT FULL oet 4 NEW ACT FULL oet 5 NEW ACT FULL oet 6 NEW ACT FULL oet 7 NEW ACT FULL hz 1 NEW ACT FULL sch 3 NEW ACT FULL Table 1 Numerical results for problems with j (j) j = 101.
27 FSQP for Finely Discretized SIP 27 P PROB ALGO n m` m n NF NG IT j j j j TIME cw 3 NEW ACT FULL cw 5 NEW ACT FULL cw 6 NEW ACT FULL oet 1 NEW ACT FULL oet 2 NEW ACT FULL oet 3 NEW ACT FULL oet 4 NEW ACT FULL oet 5 NEW ACT FULL oet 6 NEW ACT FULL oet 7 NEW ACT FULL hz 1 NEW ACT FULL sch 3 NEW ACT FULL Table 2 Numerical results for problems with j (j) j = 501.
28 28 Chapter 1 6 CONCLUSIONS We have presented and analyzed a feasible SQP algorithm for tacling smooth nonlinear programming problems with a large number of constraints, e.g. those arising from discretization of SIP problems. At each iteration, only a small subset of the constraints are used in the QP sub-problems. Thus, fewer gradient evaluations are required and the computational eort to solve the QP sub-problems is decreased. We showed that the scheme for choosing which constraints are to be included in the QP sub-problems preserves global and fast local convergence. Numerical results obtained from the cfsqp implementation show that, indeed, the algorithm performs favorably. Acnowledgments. The authors wish to than Stephan Gorner for his many valuable comments on a preliminary draft of the paper. This research was supported in part by NSF grant No. DMI and by the NSF Engineering Research Centers Program NSFD-CDR REFERENCES [1] M. C. Biggs. Constrained minimization using recursive equality quadratic programming. In F. A. Lootsma, editor, Numerical Methods for Non- Linear Optimization, pages 411{428. Academic Press, New Yor, [2] P. T. Boggs and J. W. Tolle. Sequential quadratic programming. Acta Numerica, pages 1{51, [3] I. D. Coope and G. A. Watson. A projected lagrangian algorithm for semi-innite programming. Math. Programming, 32:337{356, [4] J. W. Daniel. Stability of the solution of denite quadratic programs. Math. Programming, 5:41{53, [5] C. Gonzaga and E. Pola. On constraint dropping schemes and optimality functions for a class of outer approximation algorithms. SIAM J. Control Optim., 17:477{493, [6] C. Gonzaga, E. Pola, and R. Trahan. An improved algorithm for optimization problems with functional inequality constraints. IEEE Transactions on Automatic Control, AC-25:49{54, [7] S. A. Gustafson. A three-phase algorithm for semi-innite programs. In A. V. Fiacco and K. O. Kortane, editors, Semi-Innite Programming
2 jian l. zhou and andre l. tits The diculties in solving (SI), and in particular (CMM), stem mostly from the facts that (i) the accurate evaluation o
SIAM J. Optimization Vol. x, No. x, pp. x{xx, xxx 19xx 000 AN SQP ALGORITHM FOR FINELY DISCRETIZED CONTINUOUS MINIMAX PROBLEMS AND OTHER MINIMAX PROBLEMS WITH MANY OBJECTIVE FUNCTIONS* JIAN L. ZHOUy AND
More informationA new ane scaling interior point algorithm for nonlinear optimization subject to linear equality and inequality constraints
Journal of Computational and Applied Mathematics 161 (003) 1 5 www.elsevier.com/locate/cam A new ane scaling interior point algorithm for nonlinear optimization subject to linear equality and inequality
More informationA Primal-Dual Interior-Point Method for Nonlinear Programming with Strong Global and Local Convergence Properties
A Primal-Dual Interior-Point Method for Nonlinear Programming with Strong Global and Local Convergence Properties André L. Tits Andreas Wächter Sasan Bahtiari Thomas J. Urban Craig T. Lawrence ISR Technical
More information2 JOSE HERSKOVITS The rst stage to get an optimal design is to dene the Optimization Model. That is, to select appropriate design variables, an object
A VIEW ON NONLINEAR OPTIMIZATION JOSE HERSKOVITS Mechanical Engineering Program COPPE / Federal University of Rio de Janeiro 1 Caixa Postal 68503, 21945-970 Rio de Janeiro, BRAZIL 1. Introduction Once
More informationA Simple Primal-Dual Feasible Interior-Point Method for Nonlinear Programming with Monotone Descent
A Simple Primal-Dual Feasible Interior-Point Method for Nonlinear Programming with Monotone Descent Sasan Bahtiari André L. Tits Department of Electrical and Computer Engineering and Institute for Systems
More information1. Introduction The nonlinear complementarity problem (NCP) is to nd a point x 2 IR n such that hx; F (x)i = ; x 2 IR n + ; F (x) 2 IRn + ; where F is
New NCP-Functions and Their Properties 3 by Christian Kanzow y, Nobuo Yamashita z and Masao Fukushima z y University of Hamburg, Institute of Applied Mathematics, Bundesstrasse 55, D-2146 Hamburg, Germany,
More informationNational Institute of Standards and Technology USA. Jon W. Tolle. Departments of Mathematics and Operations Research. University of North Carolina USA
Acta Numerica (1996), pp. 1{000 Sequential Quadratic Programming Paul T. Boggs Applied and Computational Mathematics Division National Institute of Standards and Technology Gaithersburg, Maryland 20899
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More information5 Handling Constraints
5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest
More informationARE202A, Fall Contents
ARE202A, Fall 2005 LECTURE #2: WED, NOV 6, 2005 PRINT DATE: NOVEMBER 2, 2005 (NPP2) Contents 5. Nonlinear Programming Problems and the Kuhn Tucker conditions (cont) 5.2. Necessary and sucient conditions
More informationminimize x subject to (x 2)(x 4) u,
Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for
More informationNonmonotonic back-tracking trust region interior point algorithm for linear constrained optimization
Journal of Computational and Applied Mathematics 155 (2003) 285 305 www.elsevier.com/locate/cam Nonmonotonic bac-tracing trust region interior point algorithm for linear constrained optimization Detong
More informationExtra-Updates Criterion for the Limited Memory BFGS Algorithm for Large Scale Nonlinear Optimization M. Al-Baali y December 7, 2000 Abstract This pape
SULTAN QABOOS UNIVERSITY Department of Mathematics and Statistics Extra-Updates Criterion for the Limited Memory BFGS Algorithm for Large Scale Nonlinear Optimization by M. Al-Baali December 2000 Extra-Updates
More informationsystem of equations. In particular, we give a complete characterization of the Q-superlinear
INEXACT NEWTON METHODS FOR SEMISMOOTH EQUATIONS WITH APPLICATIONS TO VARIATIONAL INEQUALITY PROBLEMS Francisco Facchinei 1, Andreas Fischer 2 and Christian Kanzow 3 1 Dipartimento di Informatica e Sistemistica
More information1 Introduction Sequential Quadratic Programming (SQP) methods have proved to be very ecient for solving medium-size nonlinear programming problems [12
A Trust Region Method Based on Interior Point Techniques for Nonlinear Programming Richard H. Byrd Jean Charles Gilbert y Jorge Nocedal z August 10, 1998 Abstract An algorithm for minimizing a nonlinear
More informationA REDUCED HESSIAN METHOD FOR LARGE-SCALE CONSTRAINED OPTIMIZATION by Lorenz T. Biegler, Jorge Nocedal, and Claudia Schmid ABSTRACT We propose a quasi-
A REDUCED HESSIAN METHOD FOR LARGE-SCALE CONSTRAINED OPTIMIZATION by Lorenz T. Biegler, 1 Jorge Nocedal, 2 and Claudia Schmid 1 1 Chemical Engineering Department, Carnegie Mellon University, Pittsburgh,
More informationNOTES ON EXISTENCE AND UNIQUENESS THEOREMS FOR ODES
NOTES ON EXISTENCE AND UNIQUENESS THEOREMS FOR ODES JONATHAN LUK These notes discuss theorems on the existence, uniqueness and extension of solutions for ODEs. None of these results are original. The proofs
More informationSpurious Chaotic Solutions of Dierential. Equations. Sigitas Keras. September Department of Applied Mathematics and Theoretical Physics
UNIVERSITY OF CAMBRIDGE Numerical Analysis Reports Spurious Chaotic Solutions of Dierential Equations Sigitas Keras DAMTP 994/NA6 September 994 Department of Applied Mathematics and Theoretical Physics
More informationThe Kuhn-Tucker Problem
Natalia Lazzati Mathematics for Economics (Part I) Note 8: Nonlinear Programming - The Kuhn-Tucker Problem Note 8 is based on de la Fuente (2000, Ch. 7) and Simon and Blume (1994, Ch. 18 and 19). The Kuhn-Tucker
More information154 ADVANCES IN NONLINEAR PROGRAMMING Abstract: We propose an algorithm for nonlinear optimization that employs both trust region techniques and line
7 COMBINING TRUST REGION AND LINE SEARCH TECHNIQUES Jorge Nocedal Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL 60208-3118, USA. Ya-xiang Yuan State Key Laboratory
More informationarxiv: v1 [math.oc] 10 Apr 2017
A Method to Guarantee Local Convergence for Sequential Quadratic Programming with Poor Hessian Approximation Tuan T. Nguyen, Mircea Lazar and Hans Butler arxiv:1704.03064v1 math.oc] 10 Apr 2017 Abstract
More informationNumerical Optimization
Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,
More informationAM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α
More informationLecture 13: Constrained optimization
2010-12-03 Basic ideas A nonlinearly constrained problem must somehow be converted relaxed into a problem which we can solve (a linear/quadratic or unconstrained problem) We solve a sequence of such problems
More informationAn improved convergence theorem for the Newton method under relaxed continuity assumptions
An improved convergence theorem for the Newton method under relaxed continuity assumptions Andrei Dubin ITEP, 117218, BCheremushinsaya 25, Moscow, Russia Abstract In the framewor of the majorization technique,
More informationStructural and Multidisciplinary Optimization. P. Duysinx and P. Tossings
Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be
More informationQUADRATICALLY AND SUPERLINEARLY CONVERGENT ALGORITHMS FOR THE SOLUTION OF INEQUALITY CONSTRAINED MINIMIZATION PROBLEMS 1
QUADRATICALLY AND SUPERLINEARLY CONVERGENT ALGORITHMS FOR THE SOLUTION OF INEQUALITY CONSTRAINED MINIMIZATION PROBLEMS 1 F. FACCHINEI 2 AND S. LUCIDI 3 Communicated by L.C.W. Dixon 1 This research was
More informationApplied Mathematics &Optimization
Appl Math Optim 29: 211-222 (1994) Applied Mathematics &Optimization c 1994 Springer-Verlag New Yor Inc. An Algorithm for Finding the Chebyshev Center of a Convex Polyhedron 1 N.D.Botin and V.L.Turova-Botina
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationProjected Gradient Methods for NCP 57. Complementarity Problems via Normal Maps
Projected Gradient Methods for NCP 57 Recent Advances in Nonsmooth Optimization, pp. 57-86 Eds..-Z. u, L. Qi and R.S. Womersley c1995 World Scientic Publishers Projected Gradient Methods for Nonlinear
More informationOn the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method
Optimization Methods and Software Vol. 00, No. 00, Month 200x, 1 11 On the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method ROMAN A. POLYAK Department of SEOR and Mathematical
More informationPacific Journal of Optimization (Vol. 2, No. 3, September 2006) ABSTRACT
Pacific Journal of Optimization Vol., No. 3, September 006) PRIMAL ERROR BOUNDS BASED ON THE AUGMENTED LAGRANGIAN AND LAGRANGIAN RELAXATION ALGORITHMS A. F. Izmailov and M. V. Solodov ABSTRACT For a given
More informationInterior-Point Methods for Linear Optimization
Interior-Point Methods for Linear Optimization Robert M. Freund and Jorge Vera March, 204 c 204 Robert M. Freund and Jorge Vera. All rights reserved. Linear Optimization with a Logarithmic Barrier Function
More informationCONSTRAINED NONLINEAR PROGRAMMING
149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach
More information1. Introduction In this paper we study a new type of trust region method for solving the unconstrained optimization problem, min f(x): (1.1) x2< n Our
Combining Trust Region and Line Search Techniques Jorge Nocedal y and Ya-xiang Yuan z March 14, 1998 Report OTC 98/04 Optimization Technology Center Abstract We propose an algorithm for nonlinear optimization
More informationS. Lucidi F. Rochetich M. Roma. Curvilinear Stabilization Techniques for. Truncated Newton Methods in. the complete results 1
Universita degli Studi di Roma \La Sapienza" Dipartimento di Informatica e Sistemistica S. Lucidi F. Rochetich M. Roma Curvilinear Stabilization Techniques for Truncated Newton Methods in Large Scale Unconstrained
More informationNumerical optimization
Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal
More informationAN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING
AN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING XIAO WANG AND HONGCHAO ZHANG Abstract. In this paper, we propose an Augmented Lagrangian Affine Scaling (ALAS) algorithm for general
More informationOn the Convergence of Newton Iterations to Non-Stationary Points Richard H. Byrd Marcelo Marazzi y Jorge Nocedal z April 23, 2001 Report OTC 2001/01 Optimization Technology Center Northwestern University,
More informationAM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality
More informationAn Active Set Strategy for Solving Optimization Problems with up to 200,000,000 Nonlinear Constraints
An Active Set Strategy for Solving Optimization Problems with up to 200,000,000 Nonlinear Constraints Klaus Schittkowski Department of Computer Science, University of Bayreuth 95440 Bayreuth, Germany e-mail:
More informationGarrett: `Bernstein's analytic continuation of complex powers' 2 Let f be a polynomial in x 1 ; : : : ; x n with real coecients. For complex s, let f
1 Bernstein's analytic continuation of complex powers c1995, Paul Garrett, garrettmath.umn.edu version January 27, 1998 Analytic continuation of distributions Statement of the theorems on analytic continuation
More informationNumerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems
1 Numerical optimization Alexander & Michael Bronstein, 2006-2009 Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Numerical optimization 048921 Advanced topics in vision Processing and Analysis of
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationExercise Solutions to Functional Analysis
Exercise Solutions to Functional Analysis Note: References refer to M. Schechter, Principles of Functional Analysis Exersize that. Let φ,..., φ n be an orthonormal set in a Hilbert space H. Show n f n
More informationLagrange Relaxation and Duality
Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler
More informationVector Space Basics. 1 Abstract Vector Spaces. 1. (commutativity of vector addition) u + v = v + u. 2. (associativity of vector addition)
Vector Space Basics (Remark: these notes are highly formal and may be a useful reference to some students however I am also posting Ray Heitmann's notes to Canvas for students interested in a direct computational
More informationConstrained optimization: direct methods (cont.)
Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a
More informationOptimization Methods. Lecture 18: Optimality Conditions and. Gradient Methods. for Unconstrained Optimization
5.93 Optimization Methods Lecture 8: Optimality Conditions and Gradient Methods for Unconstrained Optimization Outline. Necessary and sucient optimality conditions Slide. Gradient m e t h o d s 3. The
More informationLinear Regression and Its Applications
Linear Regression and Its Applications Predrag Radivojac October 13, 2014 Given a data set D = {(x i, y i )} n the objective is to learn the relationship between features and the target. We usually start
More informationSECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS
SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 2005, DR RAPHAEL HAUSER 1. The Quasi-Newton Idea. In this lecture we will discuss
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More informationA globally and R-linearly convergent hybrid HS and PRP method and its inexact version with applications
A globally and R-linearly convergent hybrid HS and PRP method and its inexact version with applications Weijun Zhou 28 October 20 Abstract A hybrid HS and PRP type conjugate gradient method for smooth
More information8 Barrier Methods for Constrained Optimization
IOE 519: NL, Winter 2012 c Marina A. Epelman 55 8 Barrier Methods for Constrained Optimization In this subsection, we will restrict our attention to instances of constrained problem () that have inequality
More informationLinear stochastic approximation driven by slowly varying Markov chains
Available online at www.sciencedirect.com Systems & Control Letters 50 2003 95 102 www.elsevier.com/locate/sysconle Linear stochastic approximation driven by slowly varying Marov chains Viay R. Konda,
More informationPARAMETER IDENTIFICATION IN THE FREQUENCY DOMAIN. H.T. Banks and Yun Wang. Center for Research in Scientic Computation
PARAMETER IDENTIFICATION IN THE FREQUENCY DOMAIN H.T. Banks and Yun Wang Center for Research in Scientic Computation North Carolina State University Raleigh, NC 7695-805 Revised: March 1993 Abstract In
More informationThe iterative convex minorant algorithm for nonparametric estimation
The iterative convex minorant algorithm for nonparametric estimation Report 95-05 Geurt Jongbloed Technische Universiteit Delft Delft University of Technology Faculteit der Technische Wiskunde en Informatica
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationIn view of (31), the second of these is equal to the identity I on E m, while this, in view of (30), implies that the first can be written
11.8 Inequality Constraints 341 Because by assumption x is a regular point and L x is positive definite on M, it follows that this matrix is nonsingular (see Exercise 11). Thus, by the Implicit Function
More informationOutline Introduction: Problem Description Diculties Algebraic Structure: Algebraic Varieties Rank Decient Toeplitz Matrices Constructing Lower Rank St
Structured Lower Rank Approximation by Moody T. Chu (NCSU) joint with Robert E. Funderlic (NCSU) and Robert J. Plemmons (Wake Forest) March 5, 1998 Outline Introduction: Problem Description Diculties Algebraic
More informationBFGS WITH UPDATE SKIPPING AND VARYING MEMORY. July 9, 1996
BFGS WITH UPDATE SKIPPING AND VARYING MEMORY TAMARA GIBSON y, DIANNE P. O'LEARY z, AND LARRY NAZARETH x July 9, 1996 Abstract. We give conditions under which limited-memory quasi-newton methods with exact
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationA Generalized Homogeneous and Self-Dual Algorithm. for Linear Programming. February 1994 (revised December 1994)
A Generalized Homogeneous and Self-Dual Algorithm for Linear Programming Xiaojie Xu Yinyu Ye y February 994 (revised December 994) Abstract: A generalized homogeneous and self-dual (HSD) infeasible-interior-point
More informationMODIFYING SQP FOR DEGENERATE PROBLEMS
PREPRINT ANL/MCS-P699-1097, OCTOBER, 1997, (REVISED JUNE, 2000; MARCH, 2002), MATHEMATICS AND COMPUTER SCIENCE DIVISION, ARGONNE NATIONAL LABORATORY MODIFYING SQP FOR DEGENERATE PROBLEMS STEPHEN J. WRIGHT
More informationSTATIONARITY RESULTS FOR GENERATING SET SEARCH FOR LINEARLY CONSTRAINED OPTIMIZATION
STATIONARITY RESULTS FOR GENERATING SET SEARCH FOR LINEARLY CONSTRAINED OPTIMIZATION TAMARA G. KOLDA, ROBERT MICHAEL LEWIS, AND VIRGINIA TORCZON Abstract. We present a new generating set search (GSS) approach
More information4.6 Montel's Theorem. Robert Oeckl CA NOTES 7 17/11/2009 1
Robert Oeckl CA NOTES 7 17/11/2009 1 4.6 Montel's Theorem Let X be a topological space. We denote by C(X) the set of complex valued continuous functions on X. Denition 4.26. A topological space is called
More informationOptimization Tutorial 1. Basic Gradient Descent
E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.
More informationA CHARACTERIZATION OF STRICT LOCAL MINIMIZERS OF ORDER ONE FOR STATIC MINMAX PROBLEMS IN THE PARAMETRIC CONSTRAINT CASE
Journal of Applied Analysis Vol. 6, No. 1 (2000), pp. 139 148 A CHARACTERIZATION OF STRICT LOCAL MINIMIZERS OF ORDER ONE FOR STATIC MINMAX PROBLEMS IN THE PARAMETRIC CONSTRAINT CASE A. W. A. TAHA Received
More informationMidterm 1. Every element of the set of functions is continuous
Econ 200 Mathematics for Economists Midterm Question.- Consider the set of functions F C(0, ) dened by { } F = f C(0, ) f(x) = ax b, a A R and b B R That is, F is a subset of the set of continuous functions
More informationCONSTRAINED OPTIMALITY CRITERIA
5 CONSTRAINED OPTIMALITY CRITERIA In Chapters 2 and 3, we discussed the necessary and sufficient optimality criteria for unconstrained optimization problems. But most engineering problems involve optimization
More informationSearch Directions for Unconstrained Optimization
8 CHAPTER 8 Search Directions for Unconstrained Optimization In this chapter we study the choice of search directions used in our basic updating scheme x +1 = x + t d. for solving P min f(x). x R n All
More informationConstrained Optimization Theory
Constrained Optimization Theory Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Constrained Optimization Theory IMA, August
More informationDetermination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms: A Comparative Study
International Journal of Mathematics And Its Applications Vol.2 No.4 (2014), pp.47-56. ISSN: 2347-1557(online) Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms:
More informationUNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems
UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction
More informationOptimization. A first course on mathematics for economists
Optimization. A first course on mathematics for economists Xavier Martinez-Giralt Universitat Autònoma de Barcelona xavier.martinez.giralt@uab.eu II.3 Static optimization - Non-Linear programming OPT p.1/45
More informationThe Steepest Descent Algorithm for Unconstrained Optimization
The Steepest Descent Algorithm for Unconstrained Optimization Robert M. Freund February, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 1 Steepest Descent Algorithm The problem
More informationIE 5531: Engineering Optimization I
IE 5531: Engineering Optimization I Lecture 12: Nonlinear optimization, continued Prof. John Gunnar Carlsson October 20, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I October 20,
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods
More informationOptimization Problems with Constraints - introduction to theory, numerical Methods and applications
Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)
More informationTMA 4180 Optimeringsteori KARUSH-KUHN-TUCKER THEOREM
TMA 4180 Optimeringsteori KARUSH-KUHN-TUCKER THEOREM H. E. Krogstad, IMF, Spring 2012 Karush-Kuhn-Tucker (KKT) Theorem is the most central theorem in constrained optimization, and since the proof is scattered
More informationOptimal Newton-type methods for nonconvex smooth optimization problems
Optimal Newton-type methods for nonconvex smooth optimization problems Coralia Cartis, Nicholas I. M. Gould and Philippe L. Toint June 9, 20 Abstract We consider a general class of second-order iterations
More informationHow to Pop a Deep PDA Matters
How to Pop a Deep PDA Matters Peter Leupold Department of Mathematics, Faculty of Science Kyoto Sangyo University Kyoto 603-8555, Japan email:leupold@cc.kyoto-su.ac.jp Abstract Deep PDA are push-down automata
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationNumerical Comparisons of. Path-Following Strategies for a. Basic Interior-Point Method for. Revised August Rice University
Numerical Comparisons of Path-Following Strategies for a Basic Interior-Point Method for Nonlinear Programming M. A rg a e z, R.A. T a p ia, a n d L. V e l a z q u e z CRPC-TR97777-S Revised August 1998
More informationNumerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09
Numerical Optimization 1 Working Horse in Computer Vision Variational Methods Shape Analysis Machine Learning Markov Random Fields Geometry Common denominator: optimization problems 2 Overview of Methods
More informationIE 5531: Engineering Optimization I
IE 5531: Engineering Optimization I Lecture 15: Nonlinear optimization Prof. John Gunnar Carlsson November 1, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I November 1, 2010 1 / 24
More informationmin f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;
Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many
More information1. Introduction Let the least value of an objective function F (x), x2r n, be required, where F (x) can be calculated for any vector of variables x2r
DAMTP 2002/NA08 Least Frobenius norm updating of quadratic models that satisfy interpolation conditions 1 M.J.D. Powell Abstract: Quadratic models of objective functions are highly useful in many optimization
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationSolving generalized semi-infinite programs by reduction to simpler problems.
Solving generalized semi-infinite programs by reduction to simpler problems. G. Still, University of Twente January 20, 2004 Abstract. The paper intends to give a unifying treatment of different approaches
More informationIBM Research Report. Line Search Filter Methods for Nonlinear Programming: Motivation and Global Convergence
RC23036 (W0304-181) April 21, 2003 Computer Science IBM Research Report Line Search Filter Methods for Nonlinear Programming: Motivation and Global Convergence Andreas Wächter, Lorenz T. Biegler IBM Research
More informationAverage Reward Parameters
Simulation-Based Optimization of Markov Reward Processes: Implementation Issues Peter Marbach 2 John N. Tsitsiklis 3 Abstract We consider discrete time, nite state space Markov reward processes which depend
More informationLecture V. Numerical Optimization
Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize
More informationChapter 8 Gradient Methods
Chapter 8 Gradient Methods An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Introduction Recall that a level set of a function is the set of points satisfying for some constant. Thus, a point
More informationUnconstrained optimization
Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout
More informationINDEFINITE TRUST REGION SUBPROBLEMS AND NONSYMMETRIC EIGENVALUE PERTURBATIONS. Ronald J. Stern. Concordia University
INDEFINITE TRUST REGION SUBPROBLEMS AND NONSYMMETRIC EIGENVALUE PERTURBATIONS Ronald J. Stern Concordia University Department of Mathematics and Statistics Montreal, Quebec H4B 1R6, Canada and Henry Wolkowicz
More information118 Cores, D. and Tapia, R. A. A Robust Choice of the Lagrange Multiplier... For example: chemical equilibrium and process control oil extraction, ble
A Robust Choice of the Lagrange Multiplier in the SQP Newton Method Debora Cores 1 Richard A. Tapia 2 1 INTEVEP, S.A. (Research Center of the Venezuelan Oil Company) Apartado 76343, Caracas 1070-A, Venezuela
More information