Lagrange Relaxation and Duality
|
|
- Daniel Joseph
- 5 years ago
- Views:
Transcription
1 Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler one. If we choose suitable relaxed problem we can sometimes solve the original problem. We have already learned Lagrange relaxation technique in analysis course. In this chapter we present more theory and applications. To begin with we present a general theory on relaxation. 1. Relaxation in general Let us consider the general optimization problem { min f(x) (1) (G) s.t. x S and the following problem (2) (G r ) { min fr (x) s.t. x S r The second problem, (G r ) is called relaxation of (G) if (i) S S r, (ii) f r (x) f(x) for x S. Note that the inequality does not have to be satisfied outside the feasible region S even if f is defined. The following theorem is easy to prove but very useful. It says that the optimal value lies between those which normally are easy to compute. Theorem 1.1. Assume that (G r ) is a relaxation of (G). If x r and x are optimal solutions of (G r ) and (G) respectively, then Proof. f r ( x r ) f( x) f( x r ). f r ( x r ) = { x r is optimum of (G r )} f r ( x) = {by definition of the relaxation} f( x) = { x is optimum of (G)} f( x r ) Remark 1.1. The careful reader may notice that, in Theorem?? we assumed that there are points where the minimum is attained. This is of course not always the case. However, we can obtain a more general theorem by changing f( x) to inf x X f(x) and the counterpart of f r ( x r ). The price we pay is the clean idea, because many limit arguments will be involved. Therefore we choose to assume that the optimum is attained at some points.
2 Corollary Assume that (G r ) is a relaxation of (G) and x r is optimum to (G r ). If (i) x r S, (ii) f r ( x r ) = f( x r ) Then x r is optimum to (G). Proof. Let x be an arbitrary feasible point to (G). Then f(x) f r (x) f r ( x r ). But x r is also feasible, showing that x r is optimum to (G), for x is arbitrary. Common types of relaxation are to relax integer constraints or positivity requirements. Next we review the so-called Lagrange relaxation to relax more complicated constraints. 2. Lagrange relaxation First we study Lagrange relaxation for the optimization problem with inequality constraints described by (M). That is, for λ R m we construct the problem (3) (M λ ) min x f(x) + s.t x X. m λ i g i (x) The λ i are called lagrangians or Lagrange multipliers. Note that lagrangians are not variables but parameters. For every λ we have an optimization problem (M λ ) in the variables x. The function f λ (x) := f(x) + λ t g(x) = f(x) + λ i g i (x) is called the Lagrange function to (M). It is often denoted by L(x, λ). Proposition 2.1. If λ 0 then (M λ ) is a relaxation of (M). Proof. Note that the feasible region for (M) is S = {x X g(x) 0}. If x S then g( x) 0. Moreover, it is easy to see that f λ ( x) = f( x) + λi g i ( x) f( x) if λ 0. Now S X = S λ, by definition, (M λ ) is a i=1 relaxation of (M) if λ 0. With equality constrained problem min f(x) (4) (M = ) s.t. g(x) = 0 x X we construct Lagrange relaxation in a similar way and obtain again (M λ ). 2
3 Proposition 2.2. (M λ ) is a relaxation of (M = ) for arbitrary λ R m. Theorem 2.3. Assume that x λ is optimal to (M λ ). Then x λ is optimal to (M) if (i) g( x λ ) 0 (ii) λ i g i ( x λ ) = 0, i = 1,..., m (iii) λ 0. Proof. Condition (i) means that x λ S, which is the condition of Corollary?? (i). Condition (ii) says that f λ ( x λ ) = f( x λ ) + λ i g i ( x) = f( x λ ) which is condition (ii) in Corollary??. Finally, condition (iii) says that we have a relaxation. The statement follows then by Corollary??. Remark 2.1. Condition (ii) is called complementary condition because it requires either λ i = 0 or g i (x λ ) = 0. Example 2.1. Consider the following problem: n x 2 j min f(x) = 2 j=1 n s.t. a j x j b j=1 x j 0, j = 1,..., n where we assume that b and a j s are positive. Rewrite the the constraint in the form b a j x j 0 to get the inequality form in (M). Then we lift this into the objective function by one Lagrange multiplier λ 0 (since we only have one constraint) and obtain the Lagrange relaxed problem (P λ ) s.t. min x f λ (x) = n j=1 x j 0, j = 1,..., n ( x2 j 2 λa jx j ) + λb Because we minimize the objective function with respect to x, the term λb is constant. Note also that in Problem (P λ ) the variables x i do not affect each other. So the problem becomes n independent one dimensional optimization problems: Now min (P λ,j ) x j s.t. x j 0. f λ,j (x j ) = x2 j 2 λa jx j f λ,j (x j) = x j λa j 3
4 The necessary condition for a local optimum to (P λ,j ) gives x j = x j (λ) = λa j This is also a global optimum because of the convexity of f λ,j. Hence, (P λ ) has a global optimum at x(λ). Next we shall determine the parameter λ by Theorem??. That is we have to check that conditions (i)-(iii) in Theorem?? are satisfied. First we look at the complementary condition. It is λ(b a j ˆx j (λ)) = 0 i.e. either λ = 0 or a j x j x(λ) = b. So if λ = 0 then x(λ) = 0 But then a j x j b cannot hold, that is condition (i) cannot be satisfied. If aj (xj (λ) = b, then we have b = a j x j (λ) = a j λa j = λ a 2 j, Hence, λ = λ = b a 2 j = b n. k=1 a2 k This value is 0, that is, condition (i), and a j x( λ) = b, that is conditions (ii) and (iii) are satisfied. Thus we have the optimal solution to (P): x j = x j ( λ) = λa j = a j b/ a 2 k. The optimal value is f( x) = 1 2 a 2 j b 2 /( a 2 k )2 = 1 2 b2 a 2 k /( a 2 k )2 = 1 2 b2 / a 2 k. Now we turn to the equality constrained problem. Theorem 2.4. Assume that x λ is optimal to (M λ ). Then x λ is optimal to (M = ) if g( x λ ) = 0. Proof. The condition g( x λ ) = 0 is condition (i) in Corollary??. The second condition in Corollary?? is f( x λ ) = f λ ( x λ ) = f( x λ ) + λ i g i ( x λ ) holds automatically, for g i ( x λ ) = 0. Moreover, (??) is a relaxation for all λ R m. The theorem follows by Corollary??. As an application we are now going to prove an important theorem, that is, the KKT-conditions are also sufficient if all involved functions are convex. Theorem 2.5. Assume that f and g i are convex and differentiable. If x X and λ R m satisfy the KKT-conditions: (i) g( x) 0 (ii) λ i g i ( x) = 0, i = 1,..., m (iii) λ 0 (iv) f( x) + m j=1 λ i g i ( x) = 0 4
5 then x is optimal to (G). Proof. The proof is a routine check of conditions in Theorem??. Conditions (i)-(iii) in this theorem are (i)-(iii) in Theorem??. The last condition implies that x is optimal to (??). The proof goes as follows. f λ = f + λ i g i is a convex function, because λ i 0 implies that λ i g i is convex, and sum of convex functions is convex. Furthermore, f λ = f + λ i g i. (iv) says that f λ ( x) = 0. But then by convexity (two-point condition) for any point x X it holds that f λ (x) f λ ( x) + f λ ( x)(x x) = f λ ( x). Hence, x is optimal to (??). It completes the proof. Most of theorems in this section assume that x r S, that is, that the relaxed optimal solution lies in the feasible region of the original problem. This is not always the case. However, if x r S we can often start with x r to find a point ˆx r which lies in S and not so expansive. Then we can use the following theorem to estimate how good ˆx r is. Theorem 2.6. Assume that (G r )is a relaxation of (G) and x and x r are optimal solutions of (G) and (G r ), rspctively. If another point ˆx r has the property that ˆx S then f r ( x r ) f( x) f(ˆx r ). Proof. Exercise. 3. Lagrange duality 3.1. Formulation of Lagrange dual problems Consider the following problem (P) which we call the primal problem min f(x) (P ) s.t. g(x) 0 x X and its Lagrange relaxation (P λ ) { min f(x) + λ t g(x) s.t. x X 5
6 Let p denote the optimal value of (P ) and φ(λ) optimal value of (P λ ): p = min{f(x) g(x) 0, x X} φ(λ) = min{f(x) + λ t g(x) x X} (In fact, it should be infimum here.) Then if λ 0, (P λ ) is a relaxation of (P ), and hence, φ(λ) p. It is natural to choose the strongest relaxation, that is, choose λ 0 such that φ(λ) becomes maximum. This gives us the dual prolem: { max φ(λ) (D) s.t. λ 0 If we consider equality constrained problem, then we do not have the positivity requirement in the dual problem. The theorems below hold even in this case, for they only make use of φ(λ) p for those λ which gives relaxation. Let d be the optimal value of (D). Then we have Proposition 3.1. d p. Proof. According to above definition of p and d, we have φ(λ) p λ 0. But then d = max{φ(λ) λ 0} p. Observe that we cannot always have d = p. We call δ := p d the duality gap. Moreover, δ 0. Remark 3.1. Given a nonlinear programming problem, several Lagrange dual problems can be devised, depending on which constraints are handled as inequality constraints and equality constraints and which constraints are treated by the set X. This choice can affect both the optimal value of (D) (as in nonconvex situations) and the effort expended in evaluating and updating the dual function φ(λ) during the course of solving the dual problem. Hence, an appropriate selection of the set X must be made, depending on the structure of the problem and the purpose for solving (D). Example 3.1. Consider the following primal problem: min x x 2 2 s.t. x 1 x x 1, x 2 0 The related Lagrange relaxation problem is min x x λ( x 1 x 2 + 4) s.t. x 1, x 2 0 6
7 This problem has an optimal solution at x 1 = x 2 = λ/2. Hence, Hence, the Lagrange dual problem is φ(λ) = 1 2 λ2 + 4λ max λ 0 φ(λ) = 1 2 λ2 + 4λ Note that φ is a concave function, and its maximum over λ 0 occurs at λ = 4. Note also that p = d = 8. So the duality gap is 0. We can also note that the primal optimal solution is x j = x j ( λ) which is the optimal solution of the relaxed problem with optimal dual solution as Lagrange multiplier. We can then determine the primal optimal solution with help of the dual optimal solution Duality theorems Theorem 3.2. (Weak duality theorem) Let x be a feasible solution to Problem (P ), that is x X and g(x) 0. Also let λ be a feasible solution to problem (D), that is, λ 0. Then f(x) φ(λ). Proof. By the definition of φ, and since x X, we have φ(λ) d p f(x). Corollary Assume that x and λ are feasible solutions to (P ) and (D), respectively. If φ( λ) = f( x), then λ and x are optimal solutions to (D) and (P ), respectively. Proof. Assume that λ is another feasible dual solution. According to Theorem??, φ(λ) f(x) = {according to the assumption} = φ( λ). That is, λ is optimal. manner. The optimality of x can be argued in the similar This corollary shows a new way to look at Theorem??. Under the assumptions in the theorem it holds that x λ and λ are feasible to (P) and (D) respectively, and that φ(λ) = f( x λ ) + λ i g i ( x λ ) = f( x λ ). The optimality follows also from this corollary. The Lagrange dual problem has some nicer features than the primal problem. Here we give some of them. Proposition 3.3. The function φ(λ) is concave. 7
8 Proof. For fixed x the function l x (λ) := f(x) + λ t g(x) is a concave function of λ, for this is affine. But φ(λ) = inf x X l x (λ). So φ is concave. In conclusion we claim that the dual problem is quite nice: (i) It has nice objective function, (ii) It has nice constraints (positivity requirements), (iii) It has few variables (= number of primal constraints). For those problems where we explicitly (analytically) can determine the dual objective function φ it s often worthwhile solving the dual problem instead and thereafter find the primal solution using the following theorem. The gradient of φ can be obtained by the following theorem. Theorem 3.4. Assume that X is closed and bounded and (P λ ) has a unique optimal soluton x(λ). Then φ(λ) = g( x(λ)) t. Proof. Let ψ(x, λ) = f(x) + λ t g(x) = f(x) + g(x) t λ, and φ(λ) = min{f(x) + λ t g(x)} = min x ψ(x, λ). It s clear that λ ψ( x, λ) = g( x(λ)) t. Now we want to show that φ(λ) = λ ψ( x, λ). Since x(λ) is an optimal solution to (P λ ), we have By the chain rule φ(λ) = ψ( x(λ), λ). φ(λ) = x Ψ( x(λ), λ) λ x(λ) + λ ψ( x(λ), λ) But x(λ) minimizes ψ(x, λ) over x thus x ψ( x(λ), λ) = 0. Now since x(λ) is a unique optimum we obtain φ(λ) = λ ψ( x(λ), λ) = g( x(λ)) t. Example 3.2 (Example 2.1, revisited). We saw in Example 2.1 x j (λ) = λa j and φ(λ) = 1 2 j a2 j λ2 +bλ. Thus g( x λ ) = b j a j x j (λ) = b j a2 j λ. Then φ (λ) = j a2 j λ + b and we see that φ ( λ) = g( x λ ) as it should be. Note that at λ = λ it holds that 0 = φ ( λ) = g( x λ). So x λ is a feasible solution. Moreover it holds that λ g( x λ) = 0. Therefore, f( x λ) = f( x λ) + λ g( x λ) = φ( λ). Optimality of the x λ and λ follow from the Corollary above. 8
9 Of course we wonder if it is a coincidence or we can in general find an optimal solution to the primal problem by solving its dual. The following theorem gives us some hope. As we said the dual problem has nicer constraints. In particular, it satisfies the CQ condition (linear independace). Therefore the KKT conditions are valid at an optimal point. Theorem 3.5. Assume that X is closed and bounded. Assume further that λ is optimal to D and that (P λ) has a unique optimum x( λ). Then (i) λ t g( x( λ)) = 0 (ii) g( x( λ)) 0. Proof. Since x( λ) is a unique solution to (P λ) by the Theorem?? φ( λ) = g( x( λ)) t Since λ is optimal to D by Theorem 3.2 in the lecture notes on Optimality conditions (a) φ( λ) λ = 0 (b) φ( λ) 0. Substitute the equation above in (a) and (b) we get the (i) and (ii) as desired. This theorem, together with the Lagrange Relaxation Theorem, shows the following theorem Theorem 3.6. Assume that X is closed and bounded. Assume further that λ is optimal to (D), and that (P λ) has a unique optimum x( λ). Then x( λ) is optimum to (P). Yishao Zhou// December 5,
Constrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationQuiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006
Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationLinear and Combinatorial Optimization
Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More informationLagrangian Duality. Evelien van der Hurk. DTU Management Engineering
Lagrangian Duality Evelien van der Hurk DTU Management Engineering Topics Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality 2 DTU Management Engineering 42111: Static and Dynamic Optimization
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More informationLagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark
Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationDuality (Continued) min f ( x), X R R. Recall, the general primal problem is. The Lagrangian is a function. defined by
Duality (Continued) Recall, the general primal problem is min f ( x), xx g( x) 0 n m where X R, f : X R, g : XR ( X). he Lagrangian is a function L: XR R m defined by L( xλ, ) f ( x) λ g( x) Duality (Continued)
More informationPrimal-dual Subgradient Method for Convex Problems with Functional Constraints
Primal-dual Subgradient Method for Convex Problems with Functional Constraints Yurii Nesterov, CORE/INMA (UCL) Workshop on embedded optimization EMBOPT2014 September 9, 2014 (Lucca) Yu. Nesterov Primal-dual
More informationICS-E4030 Kernel Methods in Machine Learning
ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationOptimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers
Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More informationLectures 9 and 10: Constrained optimization problems and their optimality conditions
Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained
More informationLagrangian Duality Theory
Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual
More informationAdditional Homework Problems
Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.
More informationIntroduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module - 5 Lecture - 22 SVM: The Dual Formulation Good morning.
More informationIntroduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs
Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following
More informationSolving Dual Problems
Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationConstrained Optimization
1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More informationOn the Method of Lagrange Multipliers
On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should
More information- Well-characterized problems, min-max relations, approximate certificates. - LP problems in the standard form, primal and dual linear programs
LP-Duality ( Approximation Algorithms by V. Vazirani, Chapter 12) - Well-characterized problems, min-max relations, approximate certificates - LP problems in the standard form, primal and dual linear programs
More informationSupport Vector Machines for Regression
COMP-566 Rohan Shah (1) Support Vector Machines for Regression Provided with n training data points {(x 1, y 1 ), (x 2, y 2 ),, (x n, y n )} R s R we seek a function f for a fixed ɛ > 0 such that: f(x
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationLecture 8 Plus properties, merit functions and gap functions. September 28, 2008
Lecture 8 Plus properties, merit functions and gap functions September 28, 2008 Outline Plus-properties and F-uniqueness Equation reformulations of VI/CPs Merit functions Gap merit functions FP-I book:
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationCS711008Z Algorithm Design and Analysis
CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief
More informationConvex Optimization and Modeling
Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual
More informationLagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
More informationNonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More informationCS-E4830 Kernel Methods in Machine Learning
CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationIn the original knapsack problem, the value of the contents of the knapsack is maximized subject to a single capacity constraint, for example weight.
In the original knapsack problem, the value of the contents of the knapsack is maximized subject to a single capacity constraint, for example weight. In the multi-dimensional knapsack problem, additional
More informationPrimal/Dual Decomposition Methods
Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients
More informationSupport Vector Machines: Maximum Margin Classifiers
Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and Fu-Jie Huang 1 Outline What is behind
More informationOutline. Roadmap for the NPP segment: 1 Preliminaries: role of convexity. 2 Existence of a solution
Outline Roadmap for the NPP segment: 1 Preliminaries: role of convexity 2 Existence of a solution 3 Necessary conditions for a solution: inequality constraints 4 The constraint qualification 5 The Lagrangian
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationTMA947/MAN280 APPLIED OPTIMIZATION
Chalmers/GU Mathematics EXAM TMA947/MAN280 APPLIED OPTIMIZATION Date: 06 08 31 Time: House V, morning Aids: Text memory-less calculator Number of questions: 7; passed on one question requires 2 points
More informationLecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
More informationDuality. Geoff Gordon & Ryan Tibshirani Optimization /
Duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider
More informationKarush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725
Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 4. Subgradient
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 4 Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 2 4.1. Subgradients definition subgradient calculus duality and optimality conditions Shiqian
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationDuality Theory of Constrained Optimization
Duality Theory of Constrained Optimization Robert M. Freund April, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 The Practical Importance of Duality Duality is pervasive
More informationDuality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities
Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form
More informationApplications of Linear Programming
Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal
More informationLagrangian Duality and Convex Optimization
Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationLecture 18: Optimization Programming
Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationThe Lagrangian L : R d R m R r R is an (easier to optimize) lower bound on the original problem:
HT05: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford Convex Optimization and slides based on Arthur Gretton s Advanced Topics in Machine Learning course
More informationTutorial on Convex Optimization: Part II
Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation
More informationConstrained Optimization Theory
Constrained Optimization Theory Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Constrained Optimization Theory IMA, August
More informationNumerical Optimization
Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,
More informationCONSTRAINED OPTIMALITY CRITERIA
5 CONSTRAINED OPTIMALITY CRITERIA In Chapters 2 and 3, we discussed the necessary and sufficient optimality criteria for unconstrained optimization problems. But most engineering problems involve optimization
More informationSparse Optimization Lecture: Dual Certificate in l 1 Minimization
Sparse Optimization Lecture: Dual Certificate in l 1 Minimization Instructor: Wotao Yin July 2013 Note scriber: Zheng Sun Those who complete this lecture will know what is a dual certificate for l 1 minimization
More informationSupport Vector Machines
Support Vector Machines Support vector machines (SVMs) are one of the central concepts in all of machine learning. They are simply a combination of two ideas: linear classification via maximum (or optimal
More informationOptimality, Duality, Complementarity for Constrained Optimization
Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear
More informationSeminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1
Seminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1 Session: 15 Aug 2015 (Mon), 10:00am 1:00pm I. Optimization with
More informationCO 250 Final Exam Guide
Spring 2017 CO 250 Final Exam Guide TABLE OF CONTENTS richardwu.ca CO 250 Final Exam Guide Introduction to Optimization Kanstantsin Pashkovich Spring 2017 University of Waterloo Last Revision: March 4,
More informationMachine Learning And Applications: Supervised Learning-SVM
Machine Learning And Applications: Supervised Learning-SVM Raphaël Bournhonesque École Normale Supérieure de Lyon, Lyon, France raphael.bournhonesque@ens-lyon.fr 1 Supervised vs unsupervised learning Machine
More informationFinite Dimensional Optimization Part III: Convex Optimization 1
John Nachbar Washington University March 21, 2017 Finite Dimensional Optimization Part III: Convex Optimization 1 1 Saddle points and KKT. These notes cover another important approach to optimization,
More informationThe general programming problem is the nonlinear programming problem where a given function is maximized subject to a set of inequality constraints.
1 Optimization Mathematical programming refers to the basic mathematical problem of finding a maximum to a function, f, subject to some constraints. 1 In other words, the objective is to find a point,
More informationConvex Optimization and SVM
Convex Optimization and SVM Problem 0. Cf lecture notes pages 12 to 18. Problem 1. (i) A slab is an intersection of two half spaces, hence convex. (ii) A wedge is an intersection of two half spaces, hence
More informationsubject to (x 2)(x 4) u,
Exercises Basic definitions 5.1 A simple example. Consider the optimization problem with variable x R. minimize x 2 + 1 subject to (x 2)(x 4) 0, (a) Analysis of primal problem. Give the feasible set, the
More informationNonlinear Programming (Hillier, Lieberman Chapter 13) CHEM-E7155 Production Planning and Control
Nonlinear Programming (Hillier, Lieberman Chapter 13) CHEM-E7155 Production Planning and Control 19/4/2012 Lecture content Problem formulation and sample examples (ch 13.1) Theoretical background Graphical
More informationStructural and Multidisciplinary Optimization. P. Duysinx and P. Tossings
Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be
More informationDEPARTMENT OF STATISTICS AND OPERATIONS RESEARCH OPERATIONS RESEARCH DETERMINISTIC QUALIFYING EXAMINATION. Part I: Short Questions
DEPARTMENT OF STATISTICS AND OPERATIONS RESEARCH OPERATIONS RESEARCH DETERMINISTIC QUALIFYING EXAMINATION Part I: Short Questions August 12, 2008 9:00 am - 12 pm General Instructions This examination is
More informationDuality in Linear Programs. Lecturer: Ryan Tibshirani Convex Optimization /36-725
Duality in Linear Programs Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: proximal gradient descent Consider the problem x g(x) + h(x) with g, h convex, g differentiable, and
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationLecture 2: Linear SVM in the Dual
Lecture 2: Linear SVM in the Dual Stéphane Canu stephane.canu@litislab.eu São Paulo 2015 July 22, 2015 Road map 1 Linear SVM Optimization in 10 slides Equality constraints Inequality constraints Dual formulation
More informationLagrangian Duality for Dummies
Lagrangian Duality for Dummies David Knowles November 13, 2010 We want to solve the following optimisation problem: f 0 () (1) such that f i () 0 i 1,..., m (2) For now we do not need to assume conveity.
More informationLecture 8. Strong Duality Results. September 22, 2008
Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation
More informationEE 227A: Convex Optimization and Applications October 14, 2008
EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider
More informationCourse Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu
More informationLinear and non-linear programming
Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)
More informationGlobal Optimization of Polynomials
Semidefinite Programming Lecture 9 OR 637 Spring 2008 April 9, 2008 Scribe: Dennis Leventhal Global Optimization of Polynomials Recall we were considering the problem min z R n p(z) where p(z) is a degree
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationSECTION C: CONTINUOUS OPTIMISATION LECTURE 11: THE METHOD OF LAGRANGE MULTIPLIERS
SECTION C: CONTINUOUS OPTIMISATION LECTURE : THE METHOD OF LAGRANGE MULTIPLIERS HONOUR SCHOOL OF MATHEMATICS OXFORD UNIVERSITY HILARY TERM 005 DR RAPHAEL HAUSER. Examples. In this lecture we will take
More informationCONSTRAINED NONLINEAR PROGRAMMING
149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach
More informationON LICQ AND THE UNIQUENESS OF LAGRANGE MULTIPLIERS
ON LICQ AND THE UNIQUENESS OF LAGRANGE MULTIPLIERS GERD WACHSMUTH Abstract. Kyparisis proved in 1985 that a strict version of the Mangasarian- Fromovitz constraint qualification (MFCQ) is equivalent to
More informationMIT Algebraic techniques and semidefinite optimization February 14, Lecture 3
MI 6.97 Algebraic techniques and semidefinite optimization February 4, 6 Lecture 3 Lecturer: Pablo A. Parrilo Scribe: Pablo A. Parrilo In this lecture, we will discuss one of the most important applications
More informationConvex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming
E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program
More informationEnhanced Fritz John Optimality Conditions and Sensitivity Analysis
Enhanced Fritz John Optimality Conditions and Sensitivity Analysis Dimitri P. Bertsekas Laboratory for Information and Decision Systems Massachusetts Institute of Technology March 2016 1 / 27 Constrained
More informationUniversity of California, Davis Department of Agricultural and Resource Economics ARE 252 Lecture Notes 2 Quirino Paris
University of California, Davis Department of Agricultural and Resource Economics ARE 5 Lecture Notes Quirino Paris Karush-Kuhn-Tucker conditions................................................. page Specification
More informationLecture Notes on Support Vector Machine
Lecture Notes on Support Vector Machine Feng Li fli@sdu.edu.cn Shandong University, China 1 Hyperplane and Margin In a n-dimensional space, a hyper plane is defined by ω T x + b = 0 (1) where ω R n is
More informationAppendix A Taylor Approximations and Definite Matrices
Appendix A Taylor Approximations and Definite Matrices Taylor approximations provide an easy way to approximate a function as a polynomial, using the derivatives of the function. We know, from elementary
More informationConstrained optimization
Constrained optimization In general, the formulation of constrained optimization is as follows minj(w), subject to H i (w) = 0, i = 1,..., k. where J is the cost function and H i are the constraints. Lagrange
More informationCS711008Z Algorithm Design and Analysis
.. CS711008Z Algorithm Design and Analysis Lecture 9. Algorithm design technique: Linear programming and duality Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China
More informationExamination paper for TMA4180 Optimization I
Department of Mathematical Sciences Examination paper for TMA4180 Optimization I Academic contact during examination: Phone: Examination date: 26th May 2016 Examination time (from to): 09:00 13:00 Permitted
More informationKernel Machines. Pradeep Ravikumar Co-instructor: Manuela Veloso. Machine Learning
Kernel Machines Pradeep Ravikumar Co-instructor: Manuela Veloso Machine Learning 10-701 SVM linearly separable case n training points (x 1,, x n ) d features x j is a d-dimensional vector Primal problem:
More informationMATH2070 Optimisation
MATH2070 Optimisation Nonlinear optimisation with constraints Semester 2, 2012 Lecturer: I.W. Guo Lecture slides courtesy of J.R. Wishart Review The full nonlinear optimisation problem with equality constraints
More information