CONSTRAINED NONLINEAR PROGRAMMING
|
|
- Lynne Mae Wilcox
- 5 years ago
- Views:
Transcription
1 149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach the constrained nonlinear program is transformed into an unconstrained problem (or more commonly, a series of unconstrained problems) and solved using one (or some variant) of the algorithms we have already seen. Transformation methods may further be classified as (1) Exterior Penalty Function Methods (2) Interior Penalty (or Barrier) Function Methods (3) Augmented Lagrangian (or Multiplier) Methods 2. PRIMAL METHODS: These are methods that work directly on the original problem. We shall look at each of these in turn; we first start with transformation methods.
2 150 Exterior Penalty Function Methods These methods generate a sequence of infeasible points whose limit is an optimal solution to the original problem. The original constrained problem is transformed into an unconstrained one (or a series of unconstrained problems).the constraints are incorporated into the objective by means of a "penalty parameter" which penalizes any constraint violation. Consider the problem Min f(x) st g j (x) 0, j=1,2,,m, h j (x) = 0, j=1,2,,p, x R n Define a penalty function p as follows: ( ) = Max{0, ( )} + h ( ) where α is some positive integer. It is easy to see that If g j (x) 0 then Max{0, g j (x)} = 0 If g j (x) > 0 then Max{0, g j (x)} = g j (x) (violation) Now define the auxiliary function as P(x)= f(x) + µ (x),where µ >> 0. Thus if a constraint is violated, (x)>0 and a "penalty" of µ (x) is incurred.
3 151 The (UNCONSTRAINED) problem now is: Minimize P(x)= f(x) + µ (x) st x R n EXAMPLE: Minimize f(x)=x, st g(x)=2-x 0, x R (Note that minimum is at x * =2) Let (x)= Max[{0,g(x)}] 2, i. e., ( ) = (2 ) if < 2 0 if 2 If (x)=0, then the optimal solution to Minimize ( ) + ( ) is at x * = - and this is infeasible; so ( ) + ( ) = + (2 ) Minimize P( ) = ( ) + ( ) has optimum solution = 2. : As, (= ) µ 2 p µ 1 =0.5 µ 2 =1.5 f+µ 2 p µ 1 =0.5 µ 2 =1.5 p f+µ 1 p µ 1 p Penalty Function Auxiliary Function
4 152 In general: 1. We convert the constrained problem to an unconstrained one by using the penalty function. 2. The solution to the unconstrained problem can be made arbitrarily close to that of the original one by choosing sufficiently large µ. In practice, if µ is very large, too much emphasis is placed on feasibility. Often, algorithms for unconstrained optimization would stop prematurely because the step sizes required for improvement become very small. Usually, we solve a sequence of problems with successively increasing µ values; the optimal point from one iteration becomes the starting point for the next problem. PROCEDURE: Choose the following initially: 1. a tolerance ε, 2. an increase factor β, 3. a starting point x 1, and 4. an initial µ 1. At Iteration i 1. Solve the problem Minimize f(x)+µ i p(x); st x X (usually R n ). Use x i as the starting point and let the optimal solution be x i+1 2. If µ i p(x i+1 ) ε then STOP; else let µ i+1 =(µ i )β and start iteration (i+1)
5 153 EXAMPLE: Minimize f(x) = x x 2 2 Define the penalty function st g(x) = 1 x 1 x 2 0; x R 2 ( ) = (1 ) if ( ) > 0 0 if ( ) 0 The unconstrained problem is Minimize x x µ (x) If (x)=0, then the optimal solution is x * =(0,0) INFEASIBLE! (x) = (1 x 1 x 2 ) 2, P(x) = x x µ [(1 x 1 x 2 ) 2 ], and the necessary conditions for the optimal solution ( P(x)=0) yield the following: P = (1 )( 1) = 0, P = (1 ) ( 1) = 0 Thus = 2 (2 + 3 ) and = (2 + 3 ) Starting with µ =0.1, β=10 and x 1 =(0,0) and using a tolerance of (say), we have the following: Iter. (i) µ i x i+1 g(x i+1 ) µ i p(x i+1 ) (0.087, 0.043) (0.4, 0.2) (0.625, ) (0.6622, ) (0.666, 0.333) Thus the optimal solution is x * = (2/3, 1/3).
6 154 INTERIOR PENALTY (BARRIER) FUNCTION METHODS These methods also transform the original problem into an unconstrained one; however the barrier functions prevent the current solution from ever leaving the feasible region. These require that the interior of the feasible sets be nonempty, which is impossible if equality constraints are present. Therefore, they are used with problems having only inequality constraints. A barrier function B is one that is continuous and nonnegative over the interior of {x g(x) 0}, i.e., over the set {x g(x)<0}, and approaches as the boundary is approached from the interior. Let ( ) 0 if y<0 and lim ( ) = Then ( ) = [ ( )] Usually, ( ) = ( ) or ( ) = ( ( )) In both cases, note that lim ( ) = ( ) The auxiliary function is now f(x) + µb(x) where µ is a SMALL positive number.
7 155 Q. Why should µ be SMALL? A. Ideally we would like B(x)=0 if g j (x)<0 and B(x)= if g j (x)=0, so that we never leave the region{x g(x) 0}. However, B(x) is now discontinuous. This causes serious computational problems during the unconstrained optimization. Consider the same problem as the one we looked at with (exterior) penalty functions: Minimize f(x) = x, st x+2 0 Define the barrier function B(x)= -1/(-x+2) So the auxiliary function is f(x) + µb(x) = x - µ/(-x+2), and has its optimum solution at x * =2 +, and as µ 0, x * µ 2 B B µ 1 B µ 1 =0.5 µ 2 = µ f+µ 2 B 1 =0.5 µ 2 =1.5 f+µ 1 B Barrier Function Auxiliary Function Similar to exterior penalty functions we don't just choose one small value for rather we start with some µ 1 and generate a sequence of points.
8 156 PROCEDURE: Initially choose a tolerance ε, a decrease factor β, an interior starting point x 1, and an initial µ 1. At Iteration i 1. Solve the problem Minimize f(x)+µ i B(x),st x X (usually R n ). Use x i as the starting point and let the optimal solution be x i+1 2. If µ i B(x i+1 ) < ε then STOP; else let µ i+1 =(µ i )β and start iteration (i+1) Consider the previous example once again EXAMPLE: Minimize f(x) = x x 2 st g(x) = 1 x 1 x 2 0; x R 2 Define the barrier function ( ) = ( ) = ( + 1) The unconstrained problem is Minimize x x µ (x) = x x µ ( + 1) The necessary conditions for the optimal solution ( f(x)=0) yield the following: = 2 /( + 1) = 0, = 4 /( + 1) = 0
9 157 Solving, we get = 1 ± and = 1 ± Since the negative signs lead to infeasibility, we = and = Starting with µ =1, β=0.1 and x 1 =(0,0) and using a tolerance of (say), we have the following: Iter. (i) µ i x i+1 g(x i+1 ) µ i B(x i+1 ) (1.0, 0.5) (0.714, 0.357) (0.672, 0.336) (0.6672, ) (0.6666, ) Thus the optimal solution is x * = (2/3, 1/3).
10 158 Penalty Function Methods and Lagrange Multipliers Consider the penalty function approach to the problem below Minimize f(x) st gj(x) 0; j = 1,2,,m h j (x) = 0; j = m+1,m+2,,l x k R n. Suppose we use the usual interior penalty function described earlier, with p=2. The auxiliary function that we minimize is then given by P(x) = f(x) + µp(x) = f(x) + µ{ j [Max(0, gj(x))] 2 + j [h j (x)] 2 } = f(x) + { j µ[max(0, gj(x))] 2 + j µ[h j (x)] 2 } The necessary condition for this to have a minimum is that P(x) = f(x) + µ p(x) = 0, i.e., f(x) + j 2µ[Max(0, gj(x)] gj(x) + j 2µ[h j (x)] hj(x) = 0 (1) Suppose that the solution to (1) for a fixed µ (say µ k >0) is given by x k. Let us also designate 2µ k [Max{0, gj(x k )}] = λ j (µ k ); j=1,2,,m (2) 2µ k [h j (x k )] = λ j (µ k ); j=m+1,,l (3) so that for µ=µ k we may rewrite (1) as
11 159 f(x) + j λ j (µ k ) gj(x) + j λ j (µ k ) hj(x) = 0 (4) Now consider the Lagrangian for the original problem: L(x,λ) = f(x) + j λ j gj(x) + j λ j h j (x) The usual K-K-T necessary conditions yield f(x) + j=1..m {λ j gj(x)} + j=m+1..l {λ j hj(x)} = 0 (5) plus the original constraints, complementary slackness for the inequality constraints and λ j 0 for j=1,,m. Comparing (4) and (5) we can see that when we minimize the auxiliary function using µ=µ k, the λ j (µ k ) values given by (2) and (3) estimate the Lagrange multipliers in (5). In fact it may be shown that as the penalty function method proceeds and µ k and the x k x *, the optimum solution, the values of λ j (µ k ) λ * j, the optimum Lagrange multiplier value for constraint j. Consider our example on page 153 once again. For this problem the Lagrangian is given by L(x,λ) =x x λ(1-x 1 -x 2 ). The K-K-T conditions yield L/ x 1 = 2x 1 -λ = 0; L/ x 2 = 4x 2 -λ = 0; λ(1-x 1 -x 2 )=0 Solving these results in x 1 * =2/3; x 2 * =1/3; λ * =4/3; (λ=0 yields an infeasible solution).
12 160 Recall that for fixed µ k the optimum value of xk was [2µ k /(2+3µ k ) µ k /(2+3µ k )] T. As we saw, when µ k, these converge to the optimum solution of x * = (2/3,1/3). Now, suppose we use (2) to define λ(µ) = 2µ[Max(0, gj(x k )] = 2µ[1 - {2µ/(2+3µ)} - {µ/(2+3µ)}] (since gj(x k )>0 if µ>0) = 2µ[1 - {3µ/(2+3µ)}] = 4µ/(2+3µ) Then it is readily seen that Limµ λ(µ) = Limµ 4µ/(2+3µ) = 4/3 = λ * Similar statements can also be made for the barrier (interior penalty) function approach, e.g., if we use the log-barrier function we have P(x) = f(x) + µ p(x) = f(x) + µ j -log(-gj(x)), so that P(x) = f(x) - j {µ/gj(x)} gj(x) = 0 (6) If (like before) for a fixed µ k we denote the solution by x k and define -µ k /gj(x k ) = λ j (µ k ), then from (6) and (5) we see that λ j (µ k ) approximates λ j. Furthermore, as µ k 0 and the x k x *, it can be shown that λ j (µ k ) λ j * For our example (page 156) -µ k /gj(x k ) =µ k / 1 = -2µ k /( ) 4/3, as µ k 0.
13 161 Penalty and Barrier function methods have been referred to as SEQUENTIAL UNCONSTRAINED MINIMIZATION TECHNIQUES (SUMT) and studied in detail by Fiacco and McCormick. While they are attractive in the simplicity of the principle on which they are based, they also possess several undesirable properties. When the parameters µ are very large in value, the penalty functions tend to be ill-behaved near the boundary of the constraint set where the optimum points usually lie. Another problem is the choice of appropriate µ i and β values. The rate at which µ i change (i.e. the β values) can seriously affect the computational effort to find a solution. Also, as µ increases, the Hessian of the unconstrained function often becomes ill-conditioned. Some of these problems are addressed by the so-called "multiplier" methods. Here there is no need for µ to go to infinity, and the unconstrained function is better conditioned with no singularities. Furthermore, they also have faster rates of convergence than SUMT.
14 162 Ill-Conditioning of the Hessian Matrix Consider the Hessian matrix of the Auxiliary function P(x)=x x 2 2 +µ(1-x 1 -x 2 ) 2 : H = 2 + 2µ 2µ 2µ 4 + 2µ. Suppose we want to find its eigenvalues by solving H-λI = (2+2µ-λ)*(4+2µ-λ) - 4µ 2 = λ 2 (6+4µ)λ + (8+12µ) = 0 This quadratic equation yields λ = (3 + 2µ ) ± 4µ Taking the ratio of the largest and the smallest eigenvalue yields (3 + 2µ ) + (3 + 2µ ) 2 4µ 2 4µ It should be clear that as µ, the limit of the preceding ratio also goes to. This indicates that as the iterations proceed and we start to increase the value of µ, the Hessian of the unconstrained function that we are minimizing becomes increasingly ill-conditioned. This is a common situation and is especially problematic if we are using a method for the unconstrained optimization that requires the use of the Hessian.
15 163 MULTIPLIER METHODS Consider the problem: Min f(x), st g j (x) 0, j=1,2,...,m. In multiplier methods the auxiliary function is given by P(x) = f(x) + ½ Max{0, ( ) + } where µ j >0 and 0 are parameters associated with the j th constraint (i iteration no.). This is also often referred to as the AUGMENTED LAGRANGIAN function. Note that if =0 and µ j =µ, this reduces to the usual exterior penalty function. The basic idea behind multiplier methods is as follows: Let x i be the minimum of the auxiliary function P (x) at some iteration i. From the optimality conditions we have P = + 0, + ( ) Now, 0, + = +,. So, P = implies + ( ) +, ( ) = Let us assume for a moment that are chosen at each iteration i so that they satisfy the following:, = 0 (1) It is easily verified that (1) is equivalent to requiring that 0, 0 and = 0 (2)
16 164 Therefore as long as (2) is satisfied, we have for the point x i that minimize P (x) P = + ( ) = (3) If we let = and use the fact that µ j >0, then (2) reduces to 0, 0 and = 0, (A) and (3) reduces to + ( ) =. (B) It is readily seen that (A) and (B) are merely the KARUSH-KUHN-TUCKER necessary conditions for x i to be a solution to the original problem below! Min f(x), st g j (x) 0, j=1,2,...,m. We may therefore conclude that = and = = Here x * is the solution to the problem, and λ j * is the optimum Lagrange multiplier associated with constraint j. From the previous discussion it should be clear that at each iteration i, should be chosen in such a way that, 0, which results in x i x *. Note that x i is obtained here by minimizing the auxiliary function with respect to x.
17 ALGORITHM: A general algorithmic approach for the multiplier method may now be stated as follows: STEP 0: Set i=0, choose vectors x i, µ and. 165 STEP 1: Set i=i+1. STEP 2: Starting at x i-1, Minimize P (x) to find x i. STEP 3: Check convergence criteria and go to Step 4 only if the criteria are not satisfied. STEP 4: Modify based on satisfying (1). Also modify µ if necessary and go to Step 2. (Usually is set to 0 ) One example of a formula for changing is the following: = +,
18 166 PRIMAL METHODS By a primal method we mean one that works directly on the original constrained problem: Min f(x) st g j (x) 0 j=1,2,...,m. Almost all methods are based on the following general strategy: Let x i be a design at iteration i. A new design x i+1 is found by the expression x i+1 = x i + α i, where α i is a step size parameter and is a search direction (vector). The direction vector is typically determined by an expression of the form: = + where f and gj are gradient vectors of the objective and constraint functions and Jε is an active set" given by Jε ={j g j (x i )=0}, or more commonly ={j g j (x i )+ε 0, ε >0}. Unlike with transformation methods, it is evident that here the gradient vectors of individual constraint functions need to be evaluated. This is a characteristic of all primal methods.
19 167 METHOD OF FEASIBLE DIRECTIONS The first class of primal methods we look at are the feasible directions methods, where each x i is within the feasible region. An advantage is that if the process is terminated before reaching the true optimum solution (as is often necessary with large-scale problems) the terminating point is feasible and hence acceptable. The general strategy at iteration i is: (1) Let x i be a feasible point. (2) Choose a direction so that a) x i +α is feasible at least for some "sufficiently" small α>0, and b) f(x i +α ) < f(x i ). (3) Do an unconstrained optimization Minα f(x i +α ) to obtain α * =α i and hence x i+1 = x i +α i DEFINITION: For the problem Minimize f(x), st x X d ( 0) is called a feasible direction at x' X if δ>0 (x'+αd) X for α [0,δ) X x' X x' d feasible directions not feasible directions
20 168 Further, if we also have f(x'+αd) < f(x') for all α [0,α'), α' <δ, then d is called an improving feasible direction. Under differentiability, recall that this implies f T (x') d < 0 Let Fx' = {d f T (x') d < 0}, and Dx' = {d δ>0 (x'+αd) X for all α [0,δ)}, Thus Fx' is the set of improving directions at x', and Dx' is the set of feasible directions at x'. If x * is a local optimum then Fx* Dx* = φ as long as a constraint qualification is met. In general, for the problem Minimize f(x), st g j (x) 0, j=1,2,...,m. let Jx' ={j g j (x')=0} (Index set of active constraints) We can show that the set of feasible directions at x' is Dx' = {d gj T (x') d < 0 for all j Jx'} NOTE: If g j (x) is a linear function, then the strict inequality (<) used to define the set Dx' above can be relaxed to an inequality ( ).
21 169 Geometric Interpretation Minimize f(x), st g j (x) 0, j=1,2,,m gj(x * ) gj(x * ) x * d T gj(x * )=0 gj(x)=0 d x * gj(x)=0 d d T gj(x * )<0 d is a feasible direction (i) d is tangent to the constraint boundary (ii) In (ii) d any positive step in this direction is infeasible (however we will often consider this a feasible direction ) FARKAS LEMMA: Given A R m n, b R m, x R n, y R m, the following statements are equivalent to each other: 1. y T A 0 y T b 0 2. z such that Az=b, z 0 y T A 0 y jh j- where H j- is the closed half-space on the side of H j that does not contain A j So y T A 0 y T b 0 simply implies that ( jh j- ) H b- H 1 H 2 H 3 jh j- Suppose H j is the hyperplane through the origin that is orthogonal to A j H b A 1 b A 2 A 3
22 170 Application to NLP Let b - f(x * ) A j gj(x * ) y d (direction vector), z j λ j for j Jx* {j gj(x * )=0} Farkas Lemma then implies that (1) d T gj(x * ) 0 j Jx* -d T f(x * ) 0, i.e., d T f(x * ) 0 (2) λ j 0 such that j Jx* λ j gj(x * ) = - f(x * ) are equivalent! Note that (1) indicates that directions satisfying d T gj(x * ) 0 j Jx* are feasible directions directions satisfying d T f(x * ) 0 are ascent (non-improving) directions On the other hand (2) indicates the K-K-T conditions. They are equivalent! So at the optimum, there is no feasible direction that is improving - f (x * ) g3(x * ) g1(x * ) x * Cone generated by gradients of tight constraints g 3 (x)=0 Feasible region g 2 (x)=0 g 1 (x)=0 K-K-T implies that the steepest descent direction is in the cone generated by gradients of tight constraints
23 171 Similarly for the general NLP problem Minimize f(x) st g j (x) 0, j=1,2,...,m h k (x) = 0, k=1,2,...,p the set of feasible directions is Dx'= {d gj T (x') d < 0 j Jx', and hk T (x') d = 0 k} ==================================================== In a feasible directions algorithm we have the following: STEP 1 (direction finding) To generate an improving feasible direction from x', we need to find d Fx' Dx' φ, i.e., f T (x') d < 0 and gj T (x') d < 0 for j Jx', hk T (x') d = 0 k We therefore solve the subproblem Minimize f T (x') d st gj T (x') d < 0 for j Jx' ( 0 for j g j linear) hk T (x') d = 0 for all k, along with some normalizing constraints such as -1 d i 1 for i=1,2,...,n or d 2 = d T d 1. If the objective of this subproblem is negative, we have an improving feasible direction d i.
24 In practice, the actual subproblem we solve is slightly different since strict inequality (<) constraints are difficult to handle. Let z = Max [ f T (x') d, gj T (x') d for j Jx']. Then we solve: 172 Minimize z st f T (x') d z gj T (x') d z for j J hk T (x') d = 0 for k=1,2,...,p, -1 d i 1 for i=1,2,...,n If (z *,d * ) is the optimum solution for this then a) z * < 0 d =d * is an improving feasible direction b) z = 0 x * is a Fritz John point (or if a CQ is met, a K-K-T point) (Note that z * can never be greater than 0) STEP 2: (line search for step size) Assuming that z * from the first step is negative, we now solve Min f(x i +αd i ) st g j (x i +αd i ) 0, α 0 This can be rewritten as Min f(x i +αd i ) st 0 α α max where α max = supremum {α g j (x i +αd i ) 0, j=1,2,...,m}. Let the optimum solution to this be at α * =α i. Then we set x i+1 = (x i +αd i ) and return to Step 1.
25 NOTE: (1) In practice the set Jx' is usually defined as Jx'={ j g j (x i )+ε 0}, where the tolerance ε is referred to as the "constraint thickness," which may be reduced as the algorithm proceeds. (2) This procedure is usually attributed to ZOUTENDIJK EXAMPLE 173 Minimize 2x x 2 2-2x 1 x 2-4x 1-6x 2 st x 1 + x 2 8 For this, we have -x 1 + 2x x 1 0 -x 2 0 (A QP ) ( ) = 1 1, ( ) = 1 2, ( ) = 1 0, ( ) = 0 1, and ( ) = Let us begin with = 0 0, with f(x0 )=0 ITERATION 1 Min z J={3,4}. STEP 1 yields Min z st f T (x 1 ) d z d 1 (4x 1-2x 2-4) + d 2 (2x 2-2x 1-6) z g3 T (x i ) d z -d 1 z g4 T (x i ) d z -d 2 z -1 d 1, d 2 1 d 1,d 2 [-1,1]
26 174 i.e., Min z st -4d 1-6d 2 z, -d 1 z, -d 2 z d 1,d 2 [-1,1] yielding z * = -1 (<0), with d * = d 0 = 1 1 STEP 2 (line search): + = = Thus α max = {α 2α 8, α 10, -α 0, -α 0} = 4 We therefore solve: Minimize f(x 0 +αd 0 ), st 0 α 4 α = α 0 =4 = + = 4 4, with f(x1 )= -24. ITERATION 2 J={1}. STEP 1 yields Min z st d 1 (4x 1-2x 2-4) + d 2 (2x 2-2x 1-6) z d 1 + d 2 z -1 d 1, d 2 1 i.e., Min z st 4d 1-6d 2 z, d 1 + d 2 z, d 1,d 2 [-1,1] yielding z * = -4 (<0), with d * = d 1 = 1 0. STEP 2 (line search): + = = 4 4
27 175 Thus α max = {α 8-α 8, α , α-4 0, -4 0} = 4 We therefore solve: Minimize 2(4-α) (4-α)4 4(4-α) 6(4), st 0 α 4. α =α 1 =1 = + = 3 4, with f(x2 )= -26. ITERATION 3 J={φ}. STEP 1 yields Min z st d 1 (4x 1-2x 2-4) + d 2 (2x 2-2x 1-6) z -1 d 1, d 2 1 i.e., Min z st -4d 2 z, d 1,d 2 [-1,1] yielding z * = -4 (<0), with d * = d 2 = 0 1. STEP 2 (line search): + = = Thus α max = {α 7+α 8, 5+α 10, -3 0, -4-α 0} = 1 We therefore solve: Minimize f(x 2 +αd 2 ), st 0 α 1 i.e., Minimize 2(3) 2 + (4+α) 2 2(3)(4+α) 4(3) 6(4+α), st 0 α 1 α =α 2 =1 = + = 3 5, with f(x3 )= -29.
28 176 ITERATION 4 J={1}. STEP 1 yields Min z st d 1 (4x 1-2x 2-4) + d 2 (2x 2-2x 1-6) z d 1 + d 2 z -1 d 1, d 2 1 i.e., Min z st -2d 1-10d 2 z, d 1 + d 2 z, d 1,d 2 [-1,1] yielding z * = 0, with d * = 0 0. STOP Therefore the optimum solution is given by = 3 5, with f(x* )= path followed by algorithm 7 6 -x 1 +2 x 2 =10 5 x x 2 x 1 feasible region 2 x 1 + x 2 =8 1 x
29 177 GRADIENT PROJECTION METHOD (ROSEN) Recall that the steepest descent direction is - f(x). However, for constrained problems, moving along - f(x) may destroy feasibility. Rosen's method works by projecting - f(x) on to the hyperplane tangent to the set of active constraints. By doing so it tries to improve the objective while simultaneously maintaining feasibility. It uses d=-p f(x) as the search direction, where P is a projection matrix. An (n n) matrix is a projection matrix if P T = P and P T P = P (i.e., P is idempotent) Properties of P 1) P is positive semidefinite 2) P is a projection matrix if, and only if, I-P is also a projection matrix. 3) Let Q=I-P and p= Px 1, q=qx 2 where x 1, x 2 R n. Then (a) p T q= q T p =0 (p and q are orthogonal) (b) Any x R n can be uniquely expressed as x=p+q, where p=px, q=(i-p)x q x 2 x given x, we can find p and q p x 1
30 178 Consider the simpler case where all constraints are linear. Assume that there are 2 linear constraints intersecting along a line as shown below. P(- f) - f g 1 g 1 (x)=0 g 2 (x)=0 (I-P)(- f) g 2 Obviously, moving along - f would take us outside the feasible region. Say we project - f on to the feasible region using the matrix P. THEOREM: As long as P is a projection matrix and P f 0, d=-p f will be an improving direction. PROOF: f T d = f T (-P f) = - f T P T P f = - P f 2 < 0. (QED) For -P f to also be feasible we must have g1 T (-P f) and g2 T (-P f) 0 (by definition ).
31 179 Say we pick P such that g1 T (-P f) = g2 T (-P f) = 0, so that -P f is a feasible direction). Thus if we denote M as the (2 n) matrix where Row 1 is g1 T and Row 2 is g2 T, then we have M(-P f)= 0 (*) To find the form of the matrix P, we make use of Property 4(b) to write the vector - f as - f = P(- f) + [I-P](- f) = -P f - [I-P] f Now, - g1 and - g2 are both orthogonal to -P f, and so is [I-P](- f). Hence we must have [I-P](- f) = - [I-P] f = λ 1 g1 + λ 2 g2 -[I-P] f = M T, where = -M[I-P] f = MM T -(MM T ) -1 M[I-P] f = (MM T ) -1 MM T = = -{ (MM T ) -1 M f - (MM T ) -1 MP f } = -{ (MM T ) -1 M f + (MM T ) -1 M(-P f) } = -(MM T ) -1 M f (from (*) above M(-P f)=0 ) Hence we have - f = -P f - [I-P] f = -P f + M T = -P f - M T (MM T ) -1 M f i.e., P f = f - M T (MM T ) -1 M f = [I - M T (MM T ) -1 M] f i.e., P = [I - M T (MM T ) -1 M]
32 180 Question: What if P f(x) = 0? Answer: Then 0 = P f = [I M T (MM T ) -1 M] f = f + M T w, where w = -(MM T ) -1 M f. If w 0, then x satisfies the Karush-Kuhn-Tucker conditions and we may stop; if not, a new projection matrix could be identified such that d= - f is an improving feasible direction. In order to identify this matrix we merely pick any component of w that is negative, and drop the corresponding row from the matrix M. Then we use the usual formula to obtain. ==================== Now consider the general problem Minimize f(x) st g j (x) 0, j=1,2,...,m h k (x) = 0, k=1,2,...,p The gradient projection algorithm can be generalized to nonlinear constraints by projecting the gradient on to the tangent hyperplane at x (rather than on to the surface of the feasible region itself).
33 181 STEP 1: SEARCH DIRECTION Let x i be a feasible point and let J be the "active set", i.e., J = {j g j (x i )=0}, or more practically, J = {j g j (x i )+ε 0}. Suppose each j J is approximated using the Taylor series expansion by: g j (x) = g j (x i ) + (x-x i ) g j (x i ). Notice that this approximation is a linear function of x with slope gj(x i ). Let M matrix whose rows are gj T (x i ) for j J (active constraints) and hk T (x i ) for k=1,2,...,p (equality constraints) Let P=[I - M T (MM T ) -1 M] ( projection matrix). If M does not exist, let P=I. Let d i = -P f(x i ). a) If d i =0 and M does not exist STOP b) If d i =0 and M exists find = = -(MM T ) -1 M f(x i ) where u corresponds to g (inequality constraints) and v to h (equality constraints). If u>0 STOP, else delete the row of M corresponding to some u j <0 and recompute P. c) If d i 0 go to Step 2.
34 182 STEP 2. STEP SIZE (Line Search) Solve Minimize + where α max = Supremum {α g j (x i +αd i ) 0, h j (x i +αd i ) =0}. Call the optimal solution α * and find x i+1 = x i +α * d i. If all constraints are not linear make a "correction move" to return x into the feasible region. The need for this is shown below. Generally, it could be done by solving g(x)=0 starting at x i+1 using the Newton Raphson approach, or by moving orthogonally from x i+1 etc. x 2 O.F. contours x 2 O.F. contours f g 2 =0 P f f g1 g 1 =0 P f x i+1 correction move g 1 =0 x 1 x 1 g 1, g 2 are linear constraints NO CORRECTION REQD. g 1 is a nonlinear constraint REQUIRES CORRECTION
35 183 LINEARIZATION METHODS The general idea of linearization methods is to replace the solution of the NLP by the solution of a series of linear programs which approximate the original NLP. The simplest method is RECURSIVE LP: Transform Min f(x), st g j (x), j=1,2,...,m, into the LP Min {f(x i ) + (x-x i ) T f(x i )} st g j (x i ) + (x-x i ) T gj(x i ) 0, for all j. Thus we start with some initial x i where the objective and constraints (usually only the tight ones) are linearized, and then solve the LP to obtain a new x i+1 and continue... Note that f(x i ) is a constant and x-x i =d implies the problem is equivalent to Min d T f(x i ) st d T gj(x i ) -gj(x i ), for all j. and this is a direction finding LP problem, and x i+1 =x i +d implies that the step size=1.0 EXAMPLE: Min f(x)= 4x 1 x , st g 1 (x) = x x , g 2 (x) = -x x x x If this is linearized at the point x i =(2,4), then since
36 184 = 4 2 = 4 8 ; = 2 2 = 4 8 ; = = 6 2 it yields the following linear program (VERIFY ) Min (x) = 4x 1 8x 2 + 4, st 1(x) = 4x 1 + 8x , 2(x) = 6x 1 + 2x * =-30 f=-20 f=-10 =0 f=0 1=0 2=0 1=0 2=0 The method however has limited application since it does not converge if the local minimum occurs at a point that is not a vertex of the feasible region; in such cases it oscillates between adjacent vertices. To avoid this one may limit the step size (for instance, the Frank-Wolfe method minimizes f between x i and x i +d to get x i+1 ).
37 185 RECURSIVE QUADRATIC PROGRAMMING (RQP) RQP is similar in logic to RLP, except that it solves a sequence of quadratic programming problems. RQP is a better approach because the second order terms help capture curvature information. Min d T L(x i ) + ½ d T Bd st d T gj (x i ) -g j (x i ) j where B is the Hessian (or an approximation of the Hessian) of the Lagrangian function of the objective function f at the point x i. Notice that this a QP in d with linear constraints. Once the above QP is solved to get d i, we obtain x i by finding an optimum step size along x i +d i and continue. The matrix B is never explicitly computed --- it is usually updated by schemes such as the BFGS approach using information from f(x i ) and f(x i+1 ) only. (Recall the quasi-newton methods for unconstrained optimization)
38 186 KELLEY'S CUTTING PLANE METHOD One of the better linearization methods is the cutting plane algorithm of J. E. Kelley, developed for convex programming problems. Suppose we have a set of nonlinear constraints g j (x) 0 for j=1,2,...,m that determine the convex feasible region G. Suppose further that we can find a polyhedral set H that entirely contains the region determined by these constraints, i.e., G H. h 2 =0 h 3 =0 3=0 G 2=0 h 4 =0 1=0 H h 1 =0 G: j(x): nonlinear h j (x): linear; hence H is polyhedral In general, a cutting plane algorithm would operate as follows: 1. Solve the problem with linear constraints. 2. If the optimal solution to this problem is feasible in the original (nonlinear) constraints then STOP; it is also optimal for the original problem. 3. Otherwise add an extra linear constraint (i.e., an extra line/hyperplane) so that the current optimum point (for the linear constraints) becomes infeasible for the new problem with the added linear constraint (so we "cut out" part of the infeasibility). Now return to Step 1.
39 187 JUSTIFICATION Step 1: It is in general easier to solve problems with linear constraints than ones with nonlinear constraints. Step 2: G is determined by g j (x) 0, j=1,...,m, where each g j is convex, while H is determined by h k 0 k, where each h k is linear. G is wholly contained inside the polyhedral set H. Thus every point that satisfies g j (x) 0 automatically satisfies h k (x) 0, but not vice-versa. Thus G places "extra" constraints on the design variables and therefore the optimal solution with G can never be any better than the optimal solution with H. Therefore, if for some region H, the optimal solution x * is also feasible in G, it MUST be optimal for G also. (NOTE: In general, the optimal solution for H at some iteration will not be feasible in G). Step 3: Generating a CUT In Step 2, let = Maximum, where J is the set of violated constraints, J={j g j (x)>0}, i.e., is the value of the most violated constraint at x i. By the Taylor series approximation around x i, ( ) = + + [ ] ( )[ ] ( ) + ( ), for every x R n (since is convex).
40 188 If = then ( ) But since is violated at x i, > 0. Therefore ( ) > 0 x R n. is violated for all x, the original problem is infeasible. So, assuming then that, consider the following linear constraint: i. e., h( ) = + 0 ( ) (Note that x i, and are all known ) The current point x i is obviously infeasible in this constraint --- since h(x i ) 0 implies that + = 0, which contradicts the fact that is violated at x i, (i.e, (x i ) >0). Also, if x is feasible then ( ) 0 and + ( ) ( ) by the convexity of. So x remains feasible with this extra linear constraint and no part of the original feasible region is cut off. Hence ( ) defines a suitable cutting plane. If we add this extra linear constraint, then the optimal solution would change because the current optimal solution would no longer be feasible for the new LP with the added constraint. NOTE: If the objective for the original NLP was linear, then at each iteration we merely solve a linear program!
41 189 An Example (in R 2 ) of how a cutting plane algorithm might proceed. ( cutting plane). True optimum h x * 4 for G h 5 h 4 h 2 h 2 h 1 G h 1 G H h 3 H h 3 Nonnegativity + 3 linear constraints Nonnegativity + 4 linear constraints =x* h 6 h 4 h 5 h 2 h 4 h 5 h6 h 2 h 1 G h 1 G H h 3 H h 3 Nonnegativity + 5 linear constraints Nonnegativity + 6 linear constraints
5 Handling Constraints
5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest
More information2.3 Linear Programming
2.3 Linear Programming Linear Programming (LP) is the term used to define a wide range of optimization problems in which the objective function is linear in the unknown variables and the constraints are
More informationPenalty and Barrier Methods General classical constrained minimization problem minimize f(x) subject to g(x) 0 h(x) =0 Penalty methods are motivated by the desire to use unconstrained optimization techniques
More informationNumerical Optimization
Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationOptimization Problems with Constraints - introduction to theory, numerical Methods and applications
Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationLecture 13: Constrained optimization
2010-12-03 Basic ideas A nonlinearly constrained problem must somehow be converted relaxed into a problem which we can solve (a linear/quadratic or unconstrained problem) We solve a sequence of such problems
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More informationMultidisciplinary System Design Optimization (MSDO)
Multidisciplinary System Design Optimization (MSDO) Numerical Optimization II Lecture 8 Karen Willcox 1 Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Today s Topics Sequential
More informationISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints
ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained
More informationOptimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30
Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationCS-E4830 Kernel Methods in Machine Learning
CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationLecture V. Numerical Optimization
Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize
More informationConstrained Optimization
1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange
More information2.098/6.255/ Optimization Methods Practice True/False Questions
2.098/6.255/15.093 Optimization Methods Practice True/False Questions December 11, 2009 Part I For each one of the statements below, state whether it is true or false. Include a 1-3 line supporting sentence
More informationNONLINEAR. (Hillier & Lieberman Introduction to Operations Research, 8 th edition)
NONLINEAR PROGRAMMING (Hillier & Lieberman Introduction to Operations Research, 8 th edition) Nonlinear Programming g Linear programming has a fundamental role in OR. In linear programming all its functions
More informationBindel, Spring 2017 Numerical Analysis (CS 4220) Notes for So far, we have considered unconstrained optimization problems.
Consider constraints Notes for 2017-04-24 So far, we have considered unconstrained optimization problems. The constrained problem is minimize φ(x) s.t. x Ω where Ω R n. We usually define x in terms of
More informationAppendix A Taylor Approximations and Definite Matrices
Appendix A Taylor Approximations and Definite Matrices Taylor approximations provide an easy way to approximate a function as a polynomial, using the derivatives of the function. We know, from elementary
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationConstrained optimization: direct methods (cont.)
Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a
More informationComputational Optimization. Augmented Lagrangian NW 17.3
Computational Optimization Augmented Lagrangian NW 17.3 Upcoming Schedule No class April 18 Friday, April 25, in class presentations. Projects due unless you present April 25 (free extension until Monday
More informationDate: July 5, Contents
2 Lagrange Multipliers Date: July 5, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 14 2.3. Informative Lagrange Multipliers...........
More informationUNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems
UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction
More informationIntroduction to Nonlinear Optimization Paul J. Atzberger
Introduction to Nonlinear Optimization Paul J. Atzberger Comments should be sent to: atzberg@math.ucsb.edu Introduction We shall discuss in these notes a brief introduction to nonlinear optimization concepts,
More informationConstrained Optimization Theory
Constrained Optimization Theory Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Constrained Optimization Theory IMA, August
More informationAM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality
More informationMVE165/MMG631 Linear and integer optimization with applications Lecture 13 Overview of nonlinear programming. Ann-Brith Strömberg
MVE165/MMG631 Overview of nonlinear programming Ann-Brith Strömberg 2015 05 21 Areas of applications, examples (Ch. 9.1) Structural optimization Design of aircraft, ships, bridges, etc Decide on the material
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More information1 Computing with constraints
Notes for 2017-04-26 1 Computing with constraints Recall that our basic problem is minimize φ(x) s.t. x Ω where the feasible set Ω is defined by equality and inequality conditions Ω = {x R n : c i (x)
More informationPenalty and Barrier Methods. So we again build on our unconstrained algorithms, but in a different way.
AMSC 607 / CMSC 878o Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 3: Penalty and Barrier Methods Dianne P. O Leary c 2008 Reference: N&S Chapter 16 Penalty and Barrier
More informationNumerical optimization
Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal
More informationPart 4: Active-set methods for linearly constrained optimization. Nick Gould (RAL)
Part 4: Active-set methods for linearly constrained optimization Nick Gould RAL fx subject to Ax b Part C course on continuoue optimization LINEARLY CONSTRAINED MINIMIZATION fx subject to Ax { } b where
More informationNumerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems
1 Numerical optimization Alexander & Michael Bronstein, 2006-2009 Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Numerical optimization 048921 Advanced topics in vision Processing and Analysis of
More informationMathematical Foundations -1- Constrained Optimization. Constrained Optimization. An intuitive approach 2. First Order Conditions (FOC) 7
Mathematical Foundations -- Constrained Optimization Constrained Optimization An intuitive approach First Order Conditions (FOC) 7 Constraint qualifications 9 Formal statement of the FOC for a maximum
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationAM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α
More information8 Barrier Methods for Constrained Optimization
IOE 519: NL, Winter 2012 c Marina A. Epelman 55 8 Barrier Methods for Constrained Optimization In this subsection, we will restrict our attention to instances of constrained problem () that have inequality
More informationInterior-Point Methods for Linear Optimization
Interior-Point Methods for Linear Optimization Robert M. Freund and Jorge Vera March, 204 c 204 Robert M. Freund and Jorge Vera. All rights reserved. Linear Optimization with a Logarithmic Barrier Function
More informationTMA 4180 Optimeringsteori KARUSH-KUHN-TUCKER THEOREM
TMA 4180 Optimeringsteori KARUSH-KUHN-TUCKER THEOREM H. E. Krogstad, IMF, Spring 2012 Karush-Kuhn-Tucker (KKT) Theorem is the most central theorem in constrained optimization, and since the proof is scattered
More informationIn view of (31), the second of these is equal to the identity I on E m, while this, in view of (30), implies that the first can be written
11.8 Inequality Constraints 341 Because by assumption x is a regular point and L x is positive definite on M, it follows that this matrix is nonsingular (see Exercise 11). Thus, by the Implicit Function
More informationScientific Computing: Optimization
Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture
More informationCO 250 Final Exam Guide
Spring 2017 CO 250 Final Exam Guide TABLE OF CONTENTS richardwu.ca CO 250 Final Exam Guide Introduction to Optimization Kanstantsin Pashkovich Spring 2017 University of Waterloo Last Revision: March 4,
More informationChap 2. Optimality conditions
Chap 2. Optimality conditions Version: 29-09-2012 2.1 Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1
More informationOptimization. A first course on mathematics for economists
Optimization. A first course on mathematics for economists Xavier Martinez-Giralt Universitat Autònoma de Barcelona xavier.martinez.giralt@uab.eu II.3 Static optimization - Non-Linear programming OPT p.1/45
More informationMATH2070 Optimisation
MATH2070 Optimisation Nonlinear optimisation with constraints Semester 2, 2012 Lecturer: I.W. Guo Lecture slides courtesy of J.R. Wishart Review The full nonlinear optimisation problem with equality constraints
More information"SYMMETRIC" PRIMAL-DUAL PAIR
"SYMMETRIC" PRIMAL-DUAL PAIR PRIMAL Minimize cx DUAL Maximize y T b st Ax b st A T y c T x y Here c 1 n, x n 1, b m 1, A m n, y m 1, WITH THE PRIMAL IN STANDARD FORM... Minimize cx Maximize y T b st Ax
More informationInterior Point Methods. We ll discuss linear programming first, followed by three nonlinear problems. Algorithms for Linear Programming Problems
AMSC 607 / CMSC 764 Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 4: Introduction to Interior Point Methods Dianne P. O Leary c 2008 Interior Point Methods We ll discuss
More information5.6 Penalty method and augmented Lagrangian method
5.6 Penalty method and augmented Lagrangian method Consider a generic NLP problem min f (x) s.t. c i (x) 0 i I c i (x) = 0 i E (1) x R n where f and the c i s are of class C 1 or C 2, and I and E are the
More informationCONVERGENCE ANALYSIS OF AN INTERIOR-POINT METHOD FOR NONCONVEX NONLINEAR PROGRAMMING
CONVERGENCE ANALYSIS OF AN INTERIOR-POINT METHOD FOR NONCONVEX NONLINEAR PROGRAMMING HANDE Y. BENSON, ARUN SEN, AND DAVID F. SHANNO Abstract. In this paper, we present global and local convergence results
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationICS-E4030 Kernel Methods in Machine Learning
ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationDuality Theory of Constrained Optimization
Duality Theory of Constrained Optimization Robert M. Freund April, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 The Practical Importance of Duality Duality is pervasive
More informationA SHIFTED PRIMAL-DUAL PENALTY-BARRIER METHOD FOR NONLINEAR OPTIMIZATION
A SHIFTED PRIMAL-DUAL PENALTY-BARRIER METHOD FOR NONLINEAR OPTIMIZATION Philip E. Gill Vyacheslav Kungurtsev Daniel P. Robinson UCSD Center for Computational Mathematics Technical Report CCoM-19-3 March
More informationOutline. Roadmap for the NPP segment: 1 Preliminaries: role of convexity. 2 Existence of a solution
Outline Roadmap for the NPP segment: 1 Preliminaries: role of convexity 2 Existence of a solution 3 Necessary conditions for a solution: inequality constraints 4 The constraint qualification 5 The Lagrangian
More informationWritten Examination
Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More informationCE 191: Civil and Environmental Engineering Systems Analysis. LEC 05 : Optimality Conditions
CE 191: Civil and Environmental Engineering Systems Analysis LEC : Optimality Conditions Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 214 Prof. Moura
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationCONSTRAINED OPTIMALITY CRITERIA
5 CONSTRAINED OPTIMALITY CRITERIA In Chapters 2 and 3, we discussed the necessary and sufficient optimality criteria for unconstrained optimization problems. But most engineering problems involve optimization
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationMiscellaneous Nonlinear Programming Exercises
Miscellaneous Nonlinear Programming Exercises Henry Wolkowicz 2 08 21 University of Waterloo Department of Combinatorics & Optimization Waterloo, Ontario N2L 3G1, Canada Contents 1 Numerical Analysis Background
More informationConvex Optimization and Modeling
Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca
More informationOptimisation in Higher Dimensions
CHAPTER 6 Optimisation in Higher Dimensions Beyond optimisation in 1D, we will study two directions. First, the equivalent in nth dimension, x R n such that f(x ) f(x) for all x R n. Second, constrained
More informationSIMPLEX LIKE (aka REDUCED GRADIENT) METHODS. REDUCED GRADIENT METHOD (Wolfe)
19 SIMPLEX LIKE (aka REDUCED GRADIENT) METHODS The REDUCED GRADIENT algorithm and its variants such as the CONVEX SIMPLEX METHOD (CSM) and the GENERALIZED REDUCED GRADIENT (GRG) algorithm are approximation
More informationA SHIFTED PRIMAL-DUAL INTERIOR METHOD FOR NONLINEAR OPTIMIZATION
A SHIFTED RIMAL-DUAL INTERIOR METHOD FOR NONLINEAR OTIMIZATION hilip E. Gill Vyacheslav Kungurtsev Daniel. Robinson UCSD Center for Computational Mathematics Technical Report CCoM-18-1 February 1, 2018
More informationSeptember Math Course: First Order Derivative
September Math Course: First Order Derivative Arina Nikandrova Functions Function y = f (x), where x is either be a scalar or a vector of several variables (x,..., x n ), can be thought of as a rule which
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationPrimal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization
Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization Roger Behling a, Clovis Gonzaga b and Gabriel Haeser c March 21, 2013 a Department
More informationAM 205: lecture 18. Last time: optimization methods Today: conditions for optimality
AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality Existence of Global Minimum For example: f (x, y) = x 2 + y 2 is coercive on R 2 (global min. at (0, 0)) f (x) = x 3
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationINTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: CONVERGENCE ANALYSIS AND COMPUTATIONAL PERFORMANCE
INTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: CONVERGENCE ANALYSIS AND COMPUTATIONAL PERFORMANCE HANDE Y. BENSON, ARUN SEN, AND DAVID F. SHANNO Abstract. In this paper, we present global
More information3.10 Lagrangian relaxation
3.10 Lagrangian relaxation Consider a generic ILP problem min {c t x : Ax b, Dx d, x Z n } with integer coefficients. Suppose Dx d are the complicating constraints. Often the linear relaxation and the
More informationNonlinear Programming and the Kuhn-Tucker Conditions
Nonlinear Programming and the Kuhn-Tucker Conditions The Kuhn-Tucker (KT) conditions are first-order conditions for constrained optimization problems, a generalization of the first-order conditions we
More informationIMM OPTIMIZATION WITH CONSTRAINTS CONTENTS. 2nd Edition, March K. Madsen, H.B. Nielsen, O. Tingleff
IMM CONTENTS OPTIMIZATION WITH CONSTRAINTS 2nd Edition, March 2004 K. Madsen, H.B. Nielsen, O. Tingleff. INTRODUCTION..... Smoothness and Descent Directions.... 4.2. Convexity.... 7 2. LOCAL, CONSTRAINED
More informationOutline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems
Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction
More informationLagrange Relaxation and Duality
Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler
More informationLectures 9 and 10: Constrained optimization problems and their optimality conditions
Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained
More informationLecture Notes: Geometric Considerations in Unconstrained Optimization
Lecture Notes: Geometric Considerations in Unconstrained Optimization James T. Allison February 15, 2006 The primary objectives of this lecture on unconstrained optimization are to: Establish connections
More informationCS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares
CS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares Robert Bridson October 29, 2008 1 Hessian Problems in Newton Last time we fixed one of plain Newton s problems by introducing line search
More informationNONLINEAR PROGRAMMING WITH CONSTRAINTS
NONLINEAR PROGRAMMING WITH CONSTRAINTS 8.1 Directsubstitution... 265 8.2 First-Order ~ecessary Conditions for a Local Extremum... 267 8.3 Quadratic Programming... 284 8.4 Penalty. Barrier. and Augmented
More informationMore on Lagrange multipliers
More on Lagrange multipliers CE 377K April 21, 2015 REVIEW The standard form for a nonlinear optimization problem is min x f (x) s.t. g 1 (x) 0. g l (x) 0 h 1 (x) = 0. h m (x) = 0 The objective function
More information1 Newton s Method. Suppose we want to solve: x R. At x = x, f (x) can be approximated by:
Newton s Method Suppose we want to solve: (P:) min f (x) At x = x, f (x) can be approximated by: n x R. f (x) h(x) := f ( x)+ f ( x) T (x x)+ (x x) t H ( x)(x x), 2 which is the quadratic Taylor expansion
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods
More informationCONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS
CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS A Dissertation Submitted For The Award of the Degree of Master of Philosophy in Mathematics Neelam Patel School of Mathematics
More informationPart 2: NLP Constrained Optimization
Part 2: NLP Constrained Optimization James G. Shanahan 2 Independent Consultant and Lecturer UC Santa Cruz EMAIL: James_DOT_Shanahan_AT_gmail_DOT_com WIFI: SSID Student USERname ucsc-guest Password EnrollNow!
More informationStructural and Multidisciplinary Optimization. P. Duysinx and P. Tossings
Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be
More informationMATH 4211/6211 Optimization Basics of Optimization Problems
MATH 4211/6211 Optimization Basics of Optimization Problems Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 A standard minimization
More informationSeminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1
Seminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1 Session: 15 Aug 2015 (Mon), 10:00am 1:00pm I. Optimization with
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More informationThe general programming problem is the nonlinear programming problem where a given function is maximized subject to a set of inequality constraints.
1 Optimization Mathematical programming refers to the basic mathematical problem of finding a maximum to a function, f, subject to some constraints. 1 In other words, the objective is to find a point,
More information