# Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms: A Comparative Study

Save this PDF as:
Size: px
Start display at page:

## Transcription

1 International Journal of Mathematics And Its Applications Vol.2 No.4 (2014), pp ISSN: (online) Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms: A Comparative Study Tripti Sharma 1 and Semeneh Hunachew Addis Ababa Science and Technology University, Ethiopia. Arba Minch University, Ethiopia. Abstract : This study is focused on comparison between Method of Zoutendijk and Successive Quadratic Programming (SQP) Method, which are the methods to find feasible directions while solving a non-linear programming problem by moving from a feasible point to an improved feasible point. Keywords : Zoutendijk Algorithm, Successive Quadratic Programming Method, optimization, Feasible points, Feasible directions. 1 Introduction ZOUTENDIJK ALGORITHM In the Zoutendijk method of finding feasible directions, at each iteration, the method generates an improving feasible direction and then optimizes along that direction. Definition 1.1. Consider the problem to minimize f(x) subject to x S, where f : R n R and S is a non empty set in R n. A non zero vector d is called a feasible direction at x S if there exists a δ > 0 such that x + λd S for all λ (0, δ). Furthermore, d is called an improving feasible direction at x S if there exists a δ > 0 such that f(x + λd) < f(x) and x + λd S for all (0, δ). In Case of linear constraints, first consider the case where the feasible region S is defined by a system of linear constraints, so that the problem under consideration is of then form: Minimize f(x) Subject to: Ax b Qx = q Where, A = m n matrix, Q = l n matrix, b = m matrix, q = l matrix. Lemma 1.2. Consider the problem to minimize f(x) subject to Ax b and Qx = q. Let x be a feasible solution and suppose that A 1x = b 1 and A 2x < b 2, where A T is decomposed in to (A T 1, A T 2 ) and b T is decomposed in to 1 Corresponding author (Tripti Sharma)

2 48 Int. J. Math. And Its App. Vol.2 No.4 (2014)/ Tripti Sharma and Semeneh Hunachew (b T 1, b T 2 ). Then a non zero vector d is a feasible direction at x if and only if A 1d 0 and Qd = 0. If f(x) T d < 0, then d is an improving direction. Generating Improving Feasible Directions Given a feasible point x as shown in lemma 1.2, a non zero vector d is an improving feasible direction if f(x) T d < 0, A 1d 0 and Qd = 0. A natural method for generating such a direction is to minimize f(x) T d subject to the constraints A 1d 0 and Qd = 0. Note, however, that if a vector d such that f(x) T d < 0, A1 d 0, Q d = 0 exists, then the optimal objective value of the forgoing problem is by considering λ d, where λ. Thus a constraint that bounds the vector d or the objective function must be introduced. Such a restriction is usually referred to as a normalization constraint. There are the following three problems for generating an improving feasible direction. Each of the problems uses a different normalization constraint. Problem (P1): Minimize f(x) T d Subject to: A 1x b Qx = q; 1 d j 1 for j = 1, 2,..., n Problem (P2): Minimize f(x) T d Subject to: A 1x b Qx = q, d T d 1 Problem (P3): Minimize f(x) T d Subject to: A 1x b Qx = q; f(x) T d 1 Problem P 1 and P 2 are linear in the variables d 1,..., d n and can be solved by the simplex method. Problem P 2 contains a quadratic constraint but could be considerably simplified. Since d = 0 is a feasible solution to each of the above problems and since its objective value is equal to zero, the optimal objective value of problems P 1, P 2 and P 3 cannot be positive. If the minimal objective function value of P 1, P 2 and P 3 is negative then by lemma 1.2; an improving g feasible direction is generated. On the other hand, if the minimal objective function value is equal to zero, then x is a KKT point as shown below. Lemma 1.3. Consider the problem to minimize f(x) subject to Ax b and Qx = q. Let x be a feasible solution such that A 1x = b 1 and A 2x < b 2, where AT = (A T 1, A T 2 ) and b T = (b T 1, b T 2 ). Then for each i = 1, 2, 3, x is a KKT point if and only if the optimal objective value of problem P i is equal to zero. Line Search: Let x k be the current vector, and let d k be an improving feasible direction. The next x k+1 is given by x k + λ k d k, where the step size λ k is obtained by solving the following one dimensional problem: Minimize f(x k + λ k d k ) Subject to: A(x k + λ k d k ) b Q(x k + λ k d k ) = q; λ 0 Now, suppose that A T is decomposed in to (A T 1, A T 2 ) and b T is decomposed into (b T 1, b T 2 ) such that A 1x k = b 1 and A 2x k b2. Then the above problem could be simplified as follows:

3 Determination of Feasible Directions by Successive First note that Qx k = q and Qx k = 0, So that the constraint Q(x k + λ k d k ) = q is redundant. Since A 1x k = b 1 and A 1d k 0, then A 1(x k +λ k d k ) b 1 for all λ 0. Hence, we only need to restrict λ so that λa 2d k b2 A 2x k and the above problem reduce to the following line search problem i.e; Minimize f(x k + λ k d k ) Subject to: 0 λ λ max, { } min b1 : d1 > 0, if d > 0 d where λ max = 1, if d 0 (1.1) b = b2 A 2x k and d = A 2x k In case of Linear Constraints Consider the problem (P): Minimize f(x) Subject to: Ax b; Qx = q Initial Step: Find a starting feasible solution x 1 with Ax 1 b and Qx 1 = q. Let k = 1 and go to the main step: Main Step: 1. Given x k, suppose that A T and b T are decomposed in to (A T 1, A T 2 )(b T 1, b T 2 ) So that A 1x k = b 1 and A 2x k b2. Let d k be an optimal solution to the following problem(note that problem p 2 or p 3 could be used instead): Minimize f(x) T d Subject to: A 1d 0 Qd = 0 1 d j 1 for j = 1,..., n If f(x) T d k = 0, Stop; x k is KKT point, with the dual variables to the forgoing problem giving the corresponding Lagrange multipliers. Let λ k be an optimal solution to the following line search problem: Otherwise, go to step 2 Let λ k be an optimal solution to the following line search problem: Minimize f(x k + λd k ) subject to: 0 λ λ max Where λ max is determined according to (2.1a). Let x k+1 = x k + λ k d k. Identify the new set of binding constraints at x k+1, and update A 1 and A 2 accordingly. Replace k by k + 1 and go to step 1. Problems with non-linear inequality constraints The following theorem gives the sufficient condition for a vector d to be an improving feasible direction. Theorem 1.4. Consider the following problem Minimize f(x) subject to: g i(x) 0 for i = 1,..., m. Let x be a feasible solution, and let I be the set of binding or active constraints, that is I = {i : g i(x) = 0}. Furthermore, suppose that f and g i for i I are differentiable at x and that each g i for i / I is continuous at x. If f(x) T d < 0 and g i(x) T d < 0 for i I, then d is an improving direction. Theorem 1.5. Consider the problem to minimize f(x) subject to g i(x) 0 for i = 1, 2, 3,..., m. Let x be a

4 50 Int. J. Math. And Its App. Vol.2 No.4 (2014)/ Tripti Sharma and Semeneh Hunachew feasible solution and let I = {i : g i(x) = 0}. Consider the following direction finding problem: Minimize z Subject to: f(x) T d z 0 g i(x) T d z 0 for i I 1 d j 1 for j = 1,..., n Then x is a Fritz John point if and only if the optimal objective value to the above problem is zero. In Case of Non-linear Inequality Constraints Initial step: Choose a starting point x 1 such that g i(x 1) 0 for i = 1, 2, 3,..., m. Let k = 1 and go to the main step. Main step: 1. I = {i : g i(x k ) = 0} and solve the following problem: Minimize z Subject to: f(x k ) T d z 0 g i(x k ) T d z 0 for i I 1 d j 1 for j = 1,..., n Let (z k, d k ) be an optimal solution. If z k = 0, stop; x k is a fritz John point. If z k < 0, then go to step Let λ k be an optimal solution to the following line search problem: Minimize f(x k + λd k ) Subject to: 0 λ λ max Where, λ max = sup{λ : g i(x k + λd k ) 0 for i = 1, 2, 3,..., m}. Let x k+1 = x k + λ k d k, replace k by k + 1 and go to step 1. 2 Topkis-Veintt s modification of the feasible direction algorithm A modification of Zoutendijks method of feasible directions was proposed by Topkis and Veinott [1967] and guarantees convergence to a Fritz John point. The problem under consideration is given by Minimize f(x) Subject to: g i(x) 0 for i = 1,..., m Generating a Feasible Direction Given a feasible point x, a direction is found by solving the following direction finding linear programming problem DF(x): Problem DF(x): Minimize z Subject to: f(x) T d z 0 g i(x) T d z g i(x) for i = 1,..., m 1 d j 1 for j = 1,..., n Here both binding and non binding constraints play a role in determining the direction of movement. As opposed to the method of feasible direction of approaching the boundary of a currently nonbinding constraint.

5 Determination of Feasible Directions by Successive Topkis-Veinotts Algorithm Initial Step: Choose a point x 1 such that g i(x 1) 0 for i = 1,..., m. Let k = 1 and go to the main step Main step: 1. Let (z k, d k ) be an optimal solution to the following linear programming problem : Minimize z Subject to: f(x k ) T d z 0 g i(x k ) T d z g i(x k ) for i = 1,..., m 1 d j 1 for j = 1,..., n If z k = 0, stop: x k is a Fritz John point. Otherwise, z k 0 and go to step Let λ k be an optimal solution to the following line search problem: Minimize f(x k + λd k ) Subject to: 0 λ λ max, Here λ max = sup{λ : g i(x k + λd k ) 0 for i = 1,..., m}. Let x k+1 = x k + λ k d k, replace k by k + 1 and go to step 1. Theorem 2.1. Let x be a feasible solution to the problem to minimize f(x) subject to g i(x) 0 for i = 1,..., m. Let ( z, d) be an optimal solution to the problem DF (x). If z 0, then d is an improving feasible direction. Also, z = 0 if and only if x is a Fritz John point. Lemma 2.2. Let s be a non empty set in R n and let f : R n R be continuously differentiable. Consider the problem to minimize f(x) subject to x S. Further more, consider any feasible direction algorithm whose map A = MD is defined as follows. Given x, (x, d) D(x) means that d is an improving feasible direction of f at x. Furthermore, y M(x, d) means that y = x + λd, where λ solves the line search problem to minimize f(x + λd) subject to λ 0 and x + λd S. Let {x k } be any sequence generated by such an algorithm, and let {d k } be the corresponding sequence of directions. Then there cannot exist a subsequence {(x k, d k )} K satisfying the following properties: i) x k x for k K ii) d k d for k K iii) x k + λd k S for all λ [0, δ] and for each k K for some δ > 0 iv) f(x) T d < 0 Theorem 2.3. Let f, g i : R n R for i = 1,..., m be continuously differentiable, and consider problem to minimize f(x) subject to g i(x) 0 for i = 1,..., m. Suppose that the sequence {x k } is generated by the algorithm of Topkis and Veinott. Then any accumulation point of {x k } is a Fritz John point. 3 Successive Quadratic Programming (SQP) Algorithm SQP methods, also known as sequential, or recursive, quadratic programming approaches, employ Newtons method (or quasi-newton methods) to directly solve the KKT conditions for the original problem. As a result, the accompanying sub-problem turns out to be the minimization of a quadratic approximation to the Lagrange function optimized over a linear approximation to the constraints. Hence, this type of process is also known as

6 52 Int. J. Math. And Its App. Vol.2 No.4 (2014)/ Tripti Sharma and Semeneh Hunachew a projected Lagrangian, or Lagrange-Newton approach. By its nature this method produces both primal and dual(lagrange multiplier) solutions. SQP for equality constrained problem Consider the following equality constrained problem (P): P: Minimize f(x) (ECP) subject to: h(x) = 0, x R n Where f : R n R and h : R n R m are assumed to be continuously twice differentiable (smooth) functions. An understanding of this problem is essential in the design of SQP methods for general non-linear programming problems. The KKT optimality conditions for problem P require a primal solution x R n and a Lagrange multiplier vector λ R m such that If we use the Lagrangian f(x) + m λ i h i(x) = 0 h i(x) = 0, L(x, λ) = f(x) + i = 1,..., m (3.1) m λ ih i(x) (3.2) xl(x, λ) we can write the KKT conditions (4.1a) more compactly as = 0 (EQKKT) The main idea λ L(x, λ) behind SQP is to model problem (ECP) at the given point x (k) by a quadratic programming subproblem and then use the solution to this problem to construct a more accurate approximation x (k+1).if we perform a Taylor series expansion of the system (EQKKT) about (x (k), λ (k) ) we obtain xl(x(k), λ (k) ) + 2 xl(x (k), λ (k) ) h(x (k) ) λ L(x (k), λ (k) ) h(x (k) ) T 0 δx δ λ = 0 Where δ x = x (k+1) x (k), δ λ = λ (k+1) λ (k) and 2 xl(x, λ) = 2 f(x) + m λ i 2 h i(x) is the Hessian matrix of the Lagrangian function. Taylor series expansion can be written equivalently as 2 xl(x (k), λ (k) ) h(x (k) ) δx = f(x(k) ) h(x (k) )λ (k) h(x (k) ) T 0 h(x (k) ) δ λ or, setting d = δ x and bearing in mind that λ (k+1) = δ λ + λ (k) 2 xl(x (k), λ (k) ) h(x (k) ) h(x (k) ) T 0 d λ (k+1) = f(x(k) ) h(x (k) ) (3.3) Algorithm of SQP method 1. Determine (x (0), λ (0) ) 2. Set K = 0 3. Repeat until convergence test is satisfied 4. Solve the system (3.2) to determine (d (k), λ (k+1) ) 5. Set x (k+1) = x (k) + d (k) 6. Set k = k End ( got to step 2) In SQP methods, problem (ECP) is modelled by a quadratic programming sub-problem (QPS for short), whose optimality conditions are the same as in the system (4.1c). The algorithm is also the same as that of SQP method,

7 Determination of Feasible Directions by Successive but instead of solving the system (4.1c) in step 4, we solve the following quadratic programming sub-problem (QPS): Minimize f(x (k) ) T d dt 2 xl(x (k), λ (k) )d Subject to: h(x (k) ) + h(x (k) ) T d = 0 (3.4) Since the first order conditions for the previous problem at (x (k), λ (k) ) are given by the system (3.3) and therefore d (k) is a stationary point of (3.4). If d (k) satisfies second order sufficient conditions, then d (k) minimizes problem (3.4). We also observe that the constraints in (3.4) are derived by a first order Taylor series approximation of the constraints of the original problem (ECP). The objective function of the QPS is a truncated second order Taylor series expansion of the Lagrangian function. Convergence Rate Analysis Under appropriate conditions, we can argue a quadratic convergence behavior for the or going algorithm. Specifically, suppose that x is a regular KKT solution for problem p which together with a set of Lagrange multipliers, x satisfies the second order sufficient conditions. Then W ( x, λ) = 2 xl( x, λ) h( x) is non-singular. h( x) T 0 To see this, let us show that the system W ( x, λ) d1 d 2 = 0 has the unique solution given by (d T 1, d T 2 ) = 0. Consider any solution (d T 1, d T 2 ). Since x is a regular solution, h( x) T has full rank; so if d 1 = 0, then d 2 = 0 as well. If d 1 0, Since h( x) T d 1 = 0, we have by the second order sufficient conditions that d T 1 2 L( x)d 1 > 0. However since 2 L( x)d 1 + h( x) T d 2 = 0, we have that 2 L( x)d 1 = d T 2 h( x)d 1 = 0, a contradiction. Hence W ( x, λ) is non-singular and thus for (x k, λ k ) sufficiently close to ( x, λ), W (x k, λ k ) is non-singular. Extension to Include Inequality Constraints The sequential quadratic programming framework can be extended to general non-linear constrained problem P: Minimize f(x) Subject to: h i(x) = 0, g i(x) 0, i = 1,..., m i = 1,..., l (3.5) Where, f, h i, g i are continuously twice differentiable for each i. For instance, given an iterative (x k, u k, v k ) where u k 0 and v k are respectively, the Lagrange multiplier estimates for the inequality and the equality constrains, we consider the following quadratic sub-problem. Minimize f(x k ) + f(x k ) T d dt 2 xl(x k, λ k )d Subject to: h i(x k ) + h i(x k ) T d = 0, i = 1,..., m (3.6) g i(x k ) + g i(x k ) T d 0, i = 1,..., l Where 2 xl(x k, λ k ) = 2 f(x k ) + l u ki 2 g i(x k ) + m v ki 2 h i(x k ). Note that the KKT conditions for this problem require that in addition to primal feasibility, we find Lagrange multipliers u and v such that l m f(x k ) + 2 xl(x k, λ k )d + u i g i(x k ) + v i h i(x k ) = 0

8 54 Int. J. Math. And Its App. Vol.2 No.4 (2014)/ Tripti Sharma and Semeneh Hunachew u i[g i(x k ) + g i(x k )T d] = 0, i = 1,.., l u 0, v unrestricted. (3.7) Hence, if d k solves (3.5) with Lagrange multipliers u k+1 and v k+1 and if d k = 0, then x k along with (u k+1, v k+1 ) yields a KKT solution for the original problem (P). Otherwise, we set x k+1 = x k + d k as before, increment k by 1, and repeat the process.in similar manner, It can be shown that if x is a regular KKT solution which, together with (ū, v) satisfies the second order sufficiency conditions, and if (x k, u k, v k ) is initialized sufficiently close to ( x, ū, v), the forgoing iterative process will converge quadratically to close ( x, ū, v). Lemma 3.1. Given an iterate x k, consider the quadratic subproblem QP given by (3.6) where 2 xl(x k, λ k ) is replaced by any positive definite approximation B k. Let d solve this problem with Lagrange multipliers u and v associated with the inequality and the equality constraints, respectively. If d 0, and if µ max{u 1,..., u l, v 1,..., v m }, then d is a descent direction at x = x k for the merit function F E given above. 4 Summary Zoutendijks Feasible Directions Basic idea: move along steepest descent direction until constraints are encountered at constraint surface, solve sub-problem to find descending feasible direction repeat until KKT point is found Method Sub-problem linear: efficiently solved Determine active set before solving sub-problem! When a = 0 : KKT point found Method needs feasible starting point. Convergence The direction-finding problem only uses the binding constraints. Nearly binding constraints can cause very short steps to be taken and also drastic changes in direction. This causes the algorithmic map not to be closed. This can cause jamming and slow convergence. Idea: Use constraints that are nearly binding in the direction-finding problem. Even this is not enough to guarantee convergence Method of Topkis and Veinott Try to eliminate drastic changes in direction by accounting for all constraints. Use the following direction-finding problem: Minimize z Subject to: f(x ) T d z 0 g i(x ) T d z g i(x ), 1 d j 1 This is enough to guarantee convergence to an FJ point.

9 Determination of Feasible Directions by Successive Convergence of Topkis and Veinott Note that the solution to the direction-finding problem is feasible and improving. Also, the optimal solution is 0 if and only if the current point is an FJ point. Taking all the constraints into account eliminates drastic changes in direction and ensures that the algorithmic map is closed. Under the assumption that all the functions involved are continuously differentiable, a sequence {x k } is generated by this algorithm, then allaccumulation points are FJ points. Method of Topkis and Veinott Basic idea The basic idea is analogous to Newtons method for unconstrained optimization. In unconstrained optimization, only the objective function must be approximated, in the NLP, both the objective and the constraint must be modeled. An sequential quadratic programming method uses a quadratic for the objective and a linear model of the constraint ( i.e., a quadratic program at each iteration) Solve the KKT conditions directly using a Newton method. This leads to a method which amounts to minimizing a second-order approximation of the Lagrangian. From this, we can get a quadratic convergence rate Minimize f(x) Subject to: h i(x) = 0, i = 1,..., m g i(x) 0, i = 1,..., l = Minimize f(x k ) + f(x k ) T d dt 2 xl(x k, λ k )d Subject to: h i(x k ) + h i(x k ) T d = 0, i = 1,..., m = x (k+1) = x (k) + d (k) g i(x k ) + g i(x k ) T d 0, i = 1,..., l Basic SQP Algorithm 1. Choose initial point x 0 and initial multiplier estimates λ 0 2. Set up matrices for QP sub-problem 3. Solve QP sub-problem d k, λ k+1 4. Set x k+1 = x k + d k 5. Check convergence criteria Finished. Otherwise go to 2. 5 Conclusion Comparison of Zounendijk and SQP method :

10 56 Int. J. Math. And Its App. Vol.2 No.4 (2014)/ Tripti Sharma and Semeneh Hunachew S.No Criteria Zoutendijk SQP 1 Feasible starting point? Yes No 2 Nonlinear constraints? Yes Yes 3 Equality constraints? Hard Yes 4 Uses active set? Yes Yes 5 Iterates feasible? Yes No 6 Derivatives needed? Yes Yes Table: Comparison of Zounendijk and SQP method Generally, SQP seen as best general-purpose method for constrained problems. It relies on a profound theoretical foundation and provides powerful algorithmic tools for the solution of large-scale technologically relevant problems. Since other feasible direction methods are losing popularity, SQP is still a good option. References [1] M.S.Bazara, H.D.Sherall and C.M.Shetty, Nonlinear Programming Theory andalgorithms 2 nd edition New Jersey, (1993). [2] W.Hock and K.Schittkowski, A comparative performance evaluation of 27 nonlinear programming codes, Computing, 30(1983), (1985), 1-6. [3] Jorge Nocedal and Stephen J.Wright, Numerical Optimization, Springer operation research. [4] K.Schittkowski, Nonlinear Programming Codes, Lecture Notes in economics and Mathematical Systems, 183(1980). [5] K.Schittkowski, The non-linear programming method of Wilson, Han and Powell. Part 1: Convergence analysis, Numerische Mathematik, 38 (1981), [6] R. Fletcher, An ideal penalty function for constrained optimization, Journal of the Institute of Mathematics and its Applications, 15(1975), [7] R. Fletcher, Practical Methods of Optimization, John Wiley, Chichester, (1987). [8] F. Harary, On the group of the decomposition of two graphs, Duke Math. J., 26(1959), [9] R. Fletcher and CM. Reeves, Function minimization by conjugate gradients, The Computer Journal, 7(2)(1964), [10] S.M.Sinha, Mathematical Programming Theory and Algorithms (First edition), Elsevier, (2006). [11] Fred van Keulen and Matthijs Langelaar, Engineering Optimization:Concepts and Applications, CLA H21.1; TU Delft - PME - SOCM, (2008)

### Constrained optimization: direct methods (cont.)

Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a

### ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints

ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained

### E5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization

E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained

### 5 Handling Constraints

5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest

### Optimization Problems with Constraints - introduction to theory, numerical Methods and applications

Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)

### minimize x subject to (x 2)(x 4) u,

Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for

### Algorithms for constrained local optimization

Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained

### Numerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen

Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen

### 10 Numerical methods for constrained problems

10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside

### SF2822 Applied Nonlinear Optimization. Preparatory question. Lecture 9: Sequential quadratic programming. Anders Forsgren

SF2822 Applied Nonlinear Optimization Lecture 9: Sequential quadratic programming Anders Forsgren SF2822 Applied Nonlinear Optimization, KTH / 24 Lecture 9, 207/208 Preparatory question. Try to solve theory

### 1 Computing with constraints

Notes for 2017-04-26 1 Computing with constraints Recall that our basic problem is minimize φ(x) s.t. x Ω where the feasible set Ω is defined by equality and inequality conditions Ω = {x R n : c i (x)

### AM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods

AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality

### 2.3 Linear Programming

2.3 Linear Programming Linear Programming (LP) is the term used to define a wide range of optimization problems in which the objective function is linear in the unknown variables and the constraints are

### Numerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09

Numerical Optimization 1 Working Horse in Computer Vision Variational Methods Shape Analysis Machine Learning Markov Random Fields Geometry Common denominator: optimization problems 2 Overview of Methods

### Computational Optimization. Augmented Lagrangian NW 17.3

Computational Optimization Augmented Lagrangian NW 17.3 Upcoming Schedule No class April 18 Friday, April 25, in class presentations. Projects due unless you present April 25 (free extension until Monday

### Algorithms for Constrained Optimization

1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic

### CE 191: Civil and Environmental Engineering Systems Analysis. LEC 05 : Optimality Conditions

CE 191: Civil and Environmental Engineering Systems Analysis LEC : Optimality Conditions Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 214 Prof. Moura

### Optimization Methods

Optimization Methods Decision making Examples: determining which ingredients and in what quantities to add to a mixture being made so that it will meet specifications on its composition allocating available

### Multidisciplinary System Design Optimization (MSDO)

Multidisciplinary System Design Optimization (MSDO) Numerical Optimization II Lecture 8 Karen Willcox 1 Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Today s Topics Sequential

### Numerical optimization

Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal

### Lecture 13: Constrained optimization

2010-12-03 Basic ideas A nonlinearly constrained problem must somehow be converted relaxed into a problem which we can solve (a linear/quadratic or unconstrained problem) We solve a sequence of such problems

### Nonlinear Optimization: What s important?

Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global

### AM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods

AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α

### Second Order Optimality Conditions for Constrained Nonlinear Programming

Second Order Optimality Conditions for Constrained Nonlinear Programming Lecture 10, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk)

### Lecture 15: SQP methods for equality constrained optimization

Lecture 15: SQP methods for equality constrained optimization Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lecture 15: SQP methods for equality constrained

### MS&E 318 (CME 338) Large-Scale Numerical Optimization

Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods

### Constrained Optimization

1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange

### Numerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems

1 Numerical optimization Alexander & Michael Bronstein, 2006-2009 Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Numerical optimization 048921 Advanced topics in vision Processing and Analysis of

### 4TE3/6TE3. Algorithms for. Continuous Optimization

4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca

### Optimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23

Optimization: Nonlinear Optimization without Constraints Nonlinear Optimization without Constraints 1 / 23 Nonlinear optimization without constraints Unconstrained minimization min x f(x) where f(x) is

### 5.6 Penalty method and augmented Lagrangian method

5.6 Penalty method and augmented Lagrangian method Consider a generic NLP problem min f (x) s.t. c i (x) 0 i I c i (x) = 0 i E (1) x R n where f and the c i s are of class C 1 or C 2, and I and E are the

### Continuous Optimisation, Chpt 6: Solution methods for Constrained Optimisation

Continuous Optimisation, Chpt 6: Solution methods for Constrained Optimisation Peter J.C. Dickinson DMMP, University of Twente p.j.c.dickinson@utwente.nl http://dickinson.website/teaching/2017co.html version:

### Computational Finance

Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples

### Optimality Conditions for Constrained Optimization

72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)

### Optimization. Yuh-Jye Lee. March 21, Data Science and Machine Intelligence Lab National Chiao Tung University 1 / 29

Optimization Yuh-Jye Lee Data Science and Machine Intelligence Lab National Chiao Tung University March 21, 2017 1 / 29 You Have Learned (Unconstrained) Optimization in Your High School Let f (x) = ax

### LINEAR AND NONLINEAR PROGRAMMING

LINEAR AND NONLINEAR PROGRAMMING Stephen G. Nash and Ariela Sofer George Mason University The McGraw-Hill Companies, Inc. New York St. Louis San Francisco Auckland Bogota Caracas Lisbon London Madrid Mexico

### Nonlinear Programming

Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week

### Higher-Order Methods

Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth

### Solution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark

Solution Methods Richard Lusby Department of Management Engineering Technical University of Denmark Lecture Overview (jg Unconstrained Several Variables Quadratic Programming Separable Programming SUMT

### Optimization Tutorial 1. Basic Gradient Descent

E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.

### Newton s Method. Ryan Tibshirani Convex Optimization /36-725

Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x

### An Inexact Newton Method for Nonlinear Constrained Optimization

An Inexact Newton Method for Nonlinear Constrained Optimization Frank E. Curtis Numerical Analysis Seminar, January 23, 2009 Outline Motivation and background Algorithm development and theoretical results

### March 8, 2010 MATH 408 FINAL EXAM SAMPLE

March 8, 200 MATH 408 FINAL EXAM SAMPLE EXAM OUTLINE The final exam for this course takes place in the regular course classroom (MEB 238) on Monday, March 2, 8:30-0:20 am. You may bring two-sided 8 page

### Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

### Numerical Optimization

Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,

### Algorithms for nonlinear programming problems II

Algorithms for nonlinear programming problems II Martin Branda Charles University Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Computational Aspects of Optimization

### On the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method

Optimization Methods and Software Vol. 00, No. 00, Month 200x, 1 11 On the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method ROMAN A. POLYAK Department of SEOR and Mathematical

### CONSTRAINED NONLINEAR PROGRAMMING

149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach

### Lecture 18: Optimization Programming

Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming

### A Trust-region-based Sequential Quadratic Programming Algorithm

Downloaded from orbit.dtu.dk on: Oct 19, 2018 A Trust-region-based Sequential Quadratic Programming Algorithm Henriksen, Lars Christian; Poulsen, Niels Kjølstad Publication date: 2010 Document Version

### An Inexact Newton Method for Optimization

New York University Brown Applied Mathematics Seminar, February 10, 2009 Brief biography New York State College of William and Mary (B.S.) Northwestern University (M.S. & Ph.D.) Courant Institute (Postdoc)

### Primal/Dual Decomposition Methods

Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients

### 8 Numerical methods for unconstrained problems

8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields

### An Inexact Sequential Quadratic Optimization Method for Nonlinear Optimization

An Inexact Sequential Quadratic Optimization Method for Nonlinear Optimization Frank E. Curtis, Lehigh University involving joint work with Travis Johnson, Northwestern University Daniel P. Robinson, Johns

### Lecture 11 and 12: Penalty methods and augmented Lagrangian methods for nonlinear programming

Lecture 11 and 12: Penalty methods and augmented Lagrangian methods for nonlinear programming Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lecture 11 and

### Line Search Methods. Shefali Kulkarni-Thaker

1 BISECTION METHOD Line Search Methods Shefali Kulkarni-Thaker Consider the following unconstrained optimization problem min f(x) x R Any optimization algorithm starts by an initial point x 0 and performs

### Nonlinear Optimization for Optimal Control

Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]

### On Lagrange multipliers of trust-region subproblems

On Lagrange multipliers of trust-region subproblems Ladislav Lukšan, Ctirad Matonoha, Jan Vlček Institute of Computer Science AS CR, Prague Programy a algoritmy numerické matematiky 14 1.- 6. června 2008

### Written Examination

Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes

### Part 4: Active-set methods for linearly constrained optimization. Nick Gould (RAL)

Part 4: Active-set methods for linearly constrained optimization Nick Gould RAL fx subject to Ax b Part C course on continuoue optimization LINEARLY CONSTRAINED MINIMIZATION fx subject to Ax { } b where

### Algorithms for nonlinear programming problems II

Algorithms for nonlinear programming problems II Martin Branda Charles University in Prague Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Computational Aspects

### Applications of Linear Programming

Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal

### Lecture 3. Optimization Problems and Iterative Algorithms

Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex

### Constrained Nonlinear Optimization Algorithms

Department of Industrial Engineering and Management Sciences Northwestern University waechter@iems.northwestern.edu Institute for Mathematics and its Applications University of Minnesota August 4, 2016

### MATH 4211/6211 Optimization Quasi-Newton Method

MATH 4211/6211 Optimization Quasi-Newton Method Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 Quasi-Newton Method Motivation:

### On the acceleration of augmented Lagrangian method for linearly constrained optimization

On the acceleration of augmented Lagrangian method for linearly constrained optimization Bingsheng He and Xiaoming Yuan October, 2 Abstract. The classical augmented Lagrangian method (ALM plays a fundamental

### Lecture V. Numerical Optimization

Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize

### Convex Optimization. Newton s method. ENSAE: Optimisation 1/44

Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)

### Lectures 9 and 10: Constrained optimization problems and their optimality conditions

Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained

### The Squared Slacks Transformation in Nonlinear Programming

Technical Report No. n + P. Armand D. Orban The Squared Slacks Transformation in Nonlinear Programming August 29, 2007 Abstract. We recall the use of squared slacks used to transform inequality constraints

### SF2822 Applied nonlinear optimization, final exam Saturday December

SF2822 Applied nonlinear optimization, final exam Saturday December 5 27 8. 3. Examiner: Anders Forsgren, tel. 79 7 27. Allowed tools: Pen/pencil, ruler and rubber; plus a calculator provided by the department.

### Nonlinear Programming, Elastic Mode, SQP, MPEC, MPCC, complementarity

Preprint ANL/MCS-P864-1200 ON USING THE ELASTIC MODE IN NONLINEAR PROGRAMMING APPROACHES TO MATHEMATICAL PROGRAMS WITH COMPLEMENTARITY CONSTRAINTS MIHAI ANITESCU Abstract. We investigate the possibility

### Optimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30

Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained

### Lecture 14: October 17

1-725/36-725: Convex Optimization Fall 218 Lecture 14: October 17 Lecturer: Lecturer: Ryan Tibshirani Scribes: Pengsheng Guo, Xian Zhou Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:

### Numerical Optimization. Review: Unconstrained Optimization

Numerical Optimization Finding the best feasible solution Edward P. Gatzke Department of Chemical Engineering University of South Carolina Ed Gatzke (USC CHE ) Numerical Optimization ECHE 589, Spring 2011

### Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

### Unconstrained optimization

Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout

### 1 Newton s Method. Suppose we want to solve: x R. At x = x, f (x) can be approximated by:

Newton s Method Suppose we want to solve: (P:) min f (x) At x = x, f (x) can be approximated by: n x R. f (x) h(x) := f ( x)+ f ( x) T (x x)+ (x x) t H ( x)(x x), 2 which is the quadratic Taylor expansion

### Optimization and Root Finding. Kurt Hornik

Optimization and Root Finding Kurt Hornik Basics Root finding and unconstrained smooth optimization are closely related: Solving ƒ () = 0 can be accomplished via minimizing ƒ () 2 Slide 2 Basics Root finding

### A New Penalty-SQP Method

Background and Motivation Illustration of Numerical Results Final Remarks Frank E. Curtis Informs Annual Meeting, October 2008 Background and Motivation Illustration of Numerical Results Final Remarks

CHAPTER 2: QUADRATIC PROGRAMMING Overview Quadratic programming (QP) problems are characterized by objective functions that are quadratic in the design variables, and linear constraints. In this sense,

### Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:

CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

### 6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE. Three Alternatives/Remedies for Gradient Projection

6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE Three Alternatives/Remedies for Gradient Projection Two-Metric Projection Methods Manifold Suboptimization Methods

### Infeasibility Detection and an Inexact Active-Set Method for Large-Scale Nonlinear Optimization

Infeasibility Detection and an Inexact Active-Set Method for Large-Scale Nonlinear Optimization Frank E. Curtis, Lehigh University involving joint work with James V. Burke, University of Washington Daniel

### Constrained Optimization Theory

Constrained Optimization Theory Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Constrained Optimization Theory IMA, August

### CS711008Z Algorithm Design and Analysis

CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief

### Inexact Newton Methods and Nonlinear Constrained Optimization

Inexact Newton Methods and Nonlinear Constrained Optimization Frank E. Curtis EPSRC Symposium Capstone Conference Warwick Mathematics Institute July 2, 2009 Outline PDE-Constrained Optimization Newton

### CONVERGENCE ANALYSIS OF AN INTERIOR-POINT METHOD FOR NONCONVEX NONLINEAR PROGRAMMING

CONVERGENCE ANALYSIS OF AN INTERIOR-POINT METHOD FOR NONCONVEX NONLINEAR PROGRAMMING HANDE Y. BENSON, ARUN SEN, AND DAVID F. SHANNO Abstract. In this paper, we present global and local convergence results

### Numerical Optimization: Basic Concepts and Algorithms

May 27th 2015 Numerical Optimization: Basic Concepts and Algorithms R. Duvigneau R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 1 Outline Some basic concepts in optimization Some

### On Lagrange multipliers of trust region subproblems

On Lagrange multipliers of trust region subproblems Ladislav Lukšan, Ctirad Matonoha, Jan Vlček Institute of Computer Science AS CR, Prague Applied Linear Algebra April 28-30, 2008 Novi Sad, Serbia Outline

### Quasi-Newton Methods

Newton s Method Pros and Cons Quasi-Newton Methods MA 348 Kurt Bryan Newton s method has some very nice properties: It s extremely fast, at least once it gets near the minimum, and with the simple modifications

### Uses of duality. Geoff Gordon & Ryan Tibshirani Optimization /

Uses of duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Remember conjugate functions Given f : R n R, the function is called its conjugate f (y) = max x R n yt x f(x) Conjugates appear

5.5 Quadratic programming Minimize a quadratic function subject to linear constraints: 1 min x t Qx + c t x 2 s.t. a t i x b i i I (P a t i x = b i i E x R n, where Q is an n n matrix, I and E are the

### What s New in Active-Set Methods for Nonlinear Optimization?

What s New in Active-Set Methods for Nonlinear Optimization? Philip E. Gill Advances in Numerical Computation, Manchester University, July 5, 2011 A Workshop in Honor of Sven Hammarling UCSD Center for

### Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization. Nick Gould (RAL)

Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization Nick Gould (RAL) x IR n f(x) subject to c(x) = Part C course on continuoue optimization CONSTRAINED MINIMIZATION x

### Generalization to inequality constrained problem. Maximize

Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum

### Constrained Optimization and Lagrangian Duality

CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

### IE 5531: Engineering Optimization I

IE 5531: Engineering Optimization I Lecture 12: Nonlinear optimization, continued Prof. John Gunnar Carlsson October 20, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I October 20,