Duality (Continued) min f ( x), X R R. Recall, the general primal problem is. The Lagrangian is a function. defined by
|
|
- Arabella Cameron
- 5 years ago
- Views:
Transcription
1 Duality (Continued) Recall, the general primal problem is min f ( x), xx g( x) 0 n m where X R, f : X R, g : XR ( X). he Lagrangian is a function L: XR R m defined by L( xλ, ) f ( x) λ g( x)
2 Duality (Continued) and the dual function, L : R m R { } is defined by L ( λ) min L( x, λ) xx (where it is understood that max sup and min inf ) heorem 3: he dual function is concave (regardless of the nature of the general primal problem). L 2
3 Duality (Continued) 2 2 Proof: Let λ, λ 0 be such that L ( λ ) and L ( λ ). 2 Let [0,] and consider λ ( ) λ. L 2 2 λ ( ) λ min f ( x) λ ( ) λ g( x) xx 2 min f( ) ( ) ( ) f( ) ( ) x λ g x x λ g x x X 2 min f( x) λ g( x) ( ) min f( x) λ g( x) xx L 2 ( λ ) ( ) L( λ ) xx 3
4 Duality (Continued) 2 If, on the other hand, L ( λ ) or L ( λ ) then, automatically, the concave inequality holds. L L L 2 2 ( λ ( ) λ ) ( λ ) ( ) ( λ ) Alternative Proof: For a fixed x X we see that f ( x) λ g( x) is linear in λ 0 and is therefore concave in λ (for this fixed x ). herefore, G λ λr R x λ x x m (, ),, f ( ) g( ) 4
5 Duality (Continued) is a convex set. But G λ λr R λ L m (, ),, L ( ) m ( λ, ) λ R, R, min L( x, λ) xx λ λr R f x λ g x all xx m (, ),, ( ) ( ) f g xx xx G m ( λ, ) λ R, R, ( x) λ ( x) x But the intersection of any collection of convex sets is also convex; therefore G L is convex L is concave in λ 0. 5
6 Duality (Continued) We now introduce the primal function which may also be called the optimal value function or the perturbation function. Definition: he set of feasible right sides is defined to be m Y yr xx g( x) y X Y : YR (note: ). he primal function is defined as ( y) min f ( x), xx g( x) y. 6
7 Duality (Continued) heorem 4: If X is convex and g is a vector of concave functions then Y is a convex set. If, in addition, f is convex on X then is a convex function on Y. 2 2 Proof: Let y Y, y Y. hen there exist x X, x X 2 2 so that g( x ) y and g( x ) y, and then g( x ) y 2 2 and ( ) g ( x ) ( ) y for [0,]. By concavity of the vector g we have g( x ( ) x ) g( x ) ( ) g( x ) y (y y ( y Y, since 2 x x X (. 7
8 Duality (Continued) 2 Now, also assume f is convex on X. Let y Y, y Y be 2 such that ( y ) and ( y ). hen, by the definition of inf, for all 0 there exist x 2, x X so that g( x ) y g( x ) y 2 2 i i and f( x ) ( y ), i, 2. Also, by concavity of g, x ( ) x F xx g( x) y ( ) y 2 2 8
9 Duality (Continued) and, therefore, y y x x 2 2 ( ( ) ) f ( ( ) ) f x herefore, by letting 0 we get 2 ( ) ( ) f( ) y y x 2 ( ) ( ) ( ) 2 2 ( y ( ) y ) ( y ) ( ) ( y ) HW 46: Complete the proof for the case where 2 or ( y ). ( y ) 9
10 Let f : R R be defined by f( x) = x f ( x) x hen f is a convex function which is not differentiable at x = 0. However, note that at x = 0 we have x x ( xx) or x x 0
11 and this holds for all [,]. Such a function is said to be subdifferentiable at x. Definition: Let X R n be convex and let f : X R. f is said to be subdifferentiable at x X (in convex sense) if there is a vector R n so that x f( x) f( x) ( xx) for all x X. he vector x is called a subgradient or a support at x. x
12 [Note: f is subdifferentiable in concave sense at x if f is subdifferentiable in convex sense at x.] We say f is subdifferentiable on X (in either sense) if it is so at each x X. heorem 5: Let X be convex and let f : X R be subdifferentiable (in convex sense) (in concave sense). hen f is (convex on X ) (concave on X ). 2
13 2 Proof: Let x, x X and let x x ( ) x 2 for [0,]. hen so that R n x Subdifferentiability f f ( x ) ( x ) ( x x ) x f f 2 2 ( x ) ( x ) ( x x ) x f (x ) + ( ) f (x f (x ) x x x x 2-2 ) ( ) ( ) ( ) x x f (x ) // 3
14 HW 47 : Let X be convex and let f : X R be convex. Does this imply f is subdifferentiable, in convex sense, on X? HW 48 : Show that if f is differentiable at x and if x is a support at x then f ( x) x. [Hint : Simply use the notion of directional derivative.] heorem 6: Consider the general primal problem min f ( x) g( x) 0. Let λ 0 and let x X solve min L( xλ, ). hen g( x) is a support (concave sense) for at λ. L xx xx 4
15 Proof: o show L() λ L()- λ g( x) ( λ λ) for all λ 0. Now, L ( λ) inf ( f( x) λ g( x)) f( x) λ g( x) xx f ( x) λ g( x) λ g( x) λ g( x) L ( λ) g( x) ( λλ) [Note: Because of HW 48, if L is differentiable at λ then L ( λ) g( x).] he following is an often useful result. 5
16 heorem 7: Consider the general problem GP: min f ( x), xx g( x) 0 Let λ 0 and let x solve min L( xλ, ). hen x solves xx min f ( x), xx g( x) g( x) ( ) P g ( x ) 6
17 Proof : Let y g( x). he Lagrangian for this perturbed problem is L(, xλ ) f () x λ y ( g () x g ()) x f ( x) λ g( x) λ g( x)) L( xλ, ) λ g( x) L ( λ) min L ( x, λ) min L( x, λ) λ g( x) L( x, λ) λ g( x) y xx y xx 7
18 herefore, (i) x solves Subdifferentiability min L ( xλ, ) xx (ii) x X, g( x) g( x) y (iii) λ ( g( x) g( x)) 0 ( xλ, ) is a saddle-point for the Lagrangian L(, xλ ) P g ( x ) (which is the Lagrangian for.) HW 49 : Corollary: x is optimal for min f ( x), xx g( x) y where y g ( x) if 0, and y g ( x) if 0. i i i i i i y 8
19 We now want to demonstrate the result that if x solves min L( xλ, ) then λ is a support (convex sense) for at g( x). hat is, we want to show ( y) ( g( x)) λ ( yg( x)), for all y Y. herefore, if is differentiable at g( x) then xx ( g( x)) λ he most convenient way to lead up to this important result is through the following which relates the primal function,, with the dual function,. L 9
20 heorem 8: L( λ)= inf ( ( y) λ y). yy Proof: inf ( ( ) L( λ)= f x λ g( x)) inf ( f( x) λ g( x)) xx all yy { xx g( x) y } herefore, L( λ) inf ( ( y) λ y). yy inf ( f ( x) λ y) ( y) λ y { xx g( x) y } It remains to show the opposite inequality. Define Y { y R m xx g ( x ) y }. hen Y Y. Now, let x X and let y g ( x) ( yy ). 20
21 hen, f( x) λ g( x) inf ( f( x) λ g( x)) inf ( f( x) λ y) { xx g( x) y } { xx g( x) y} inf ( f ( x) λ y) ( y) λ y { xx g( x) y } herefore, since x X is arbitrary, we have L ( λ) min ( f ( x) λ g( x)) inf ( ( y) λ y) inf ( ( y) λ y) xx yy yy We can now derive the result dual to that of heorem 6. heorem 9: Let λ 0 and let x X solve min L( xλ, ) xx hen λ is a support (convex sense) for at g( x). 2
22 Proof: o show ( y) ( g( x)) λ ( y g( x)), for all y Y, by heorem 7, we know that ( xλ, ) is a saddle-point for min f ( x) g( x) g( x) xx and, therefore, ( g( x )) ( λ ) ( λ ) λ g( x ) inf ( ( y ) λ y ) λ g( x ) L y L yg( x) yy ( y) λ g( x) λ y all yy ( y) ( g( x)) λ ( y g ( x)), for all y Y Note: If is differentiable at g( x), then ( g( x)) λ 22
23 Let s define the sets of supports for and L, respectively, as and m ( z) { λr ( y) ( z) λ ( y z), forall y Y } m L( γ) { yr L( λ) L( γ) y ( λγ), forall λ 0 } In terms of this notation, heorems 6 and 9 can be summarized as follows. heorem 20: Let λ 0 and let x X solve min L( xλ, ). hen, and g( x) L ( λ) λ ( g( x)) xx 23
24 Moreover is differentiable at and is differentiable at g x, L λ ( ) g( x) L ( λ) and λ ( g( x)) 2 Example : min x, x 0. xr Recall, this problem has no saddle-point. Also, we showed L ( L ( 24
25 that max L( ) has no optimizing vector. Also, note that 0 and, for y Y, we have ( y) y Y { y y 0} y) y y) y And this function has no convex support at the origin ( y 0). 25
26 Example 2 : min x, x 0. xr Subdifferentiability Recall, this problem has no saddle-point. We also showed L ( ) 0, if 0, if 0 herefore, 0 solves max L ( ). Also, note that Y R and 0 ( y) y, if y 0, if y 26
27 y) y Note that has no convex support at the origin ( y 0). Also, note that L( x, ) x and therefore only x 0 solves min L( x, ) and, of course, is not even feasible. Also, note that xr max L( x, ) xr x 0 (so this is no help either). 27
28 Example 3 : min x, x 0 where xx hen L( x, ) ( ) x and x R, if 0 L ( ), if 28
29 * * * herefore, solves max L ( ). Also, note that ( x, ) is a saddle-point, 0 * where x (show this!). However, L( x, ) for all xr * * * and, therefore, x is not the only optimizer for the Lagrangian (parameterized by the optimal dual vector). In particular, the operation * min L( x, ) xr does not automatically provide an optimal primal solution. Note further * that the dual function,, is not differentiable at. L hese examples lead to the following. 29
30 heorem 2: Consider the general primal problem min f ( x), xx g( x) 0 (GP) and assume GP has an optimal vector x. hen, GP has a saddle point if, and only if, the primal function has a nonnegative support (convex sense) at the origin ( y 0). Proof: Suppose ( x, λ ) is a saddle-point. o show λ (0) or to show ( y) (0) λ y, for all yy 30
31 Now, (0) f ( x) L ( λ) inf ( ( y) λ y) yy ( y) λ y, all yy or ( y) (0) λ y, all yy Conversely, suppose λ (0), λ 0. hen, ( y) (0) λ y, all yy L ( λ) inf ( ( y) λ y) (0) yy 3
32 But the Weak Duality heorem states that L ( λ) and, therefore, L ( λ). herefore, λ solves the dual problem and f ( x) L ( λ) implies that ( xλ, ) is a saddle-point. heorem 22: Let λ 0 solve max L ( λ and further assume that L is λ0 differentiable at λ. hen any x * X which solves min L( xλ, ) is also optimal for the general primal GP. xx 32
33 Proof: Since λ is optimal we must have L ( λ) ( λλ) 0, all λ 0 (since { λλ λ 0 } is the set of feasible directions at λ ). By HW48 * and heorem 6 we have L ( λ) g( x ) and therefore * gx ( ) ( λ λ) 0, all λ0 or * * inf λ ( x ) λ ( x ) λ0 g g (*) * herefore, g( x ) 0 since if, say, g ( x * ) 0 * then inf λ g( x ). * * herefore, λ g( x ) 0. But, by setting λ=0 in (*), we have λ g( x ) 0. * Hence λ g( x ) 0. herefore, λ0 33
34 and solves * (i) x X min L( x, λ) xx * * (ii), g( ) x X x 0 * (iii) λ g( x ) 0 * ( x, λ) * is a saddle-point for GP which, in turn, implies x is optimal for GP. HW 50 Consider the problem min xr x x 2 x
35 [ Note: his problem is equivalent to Example.] Show whether this problem has a saddle-point. HW 5: Consider GP and assume there is a vector x X so that (i.e., g( x) 0, g2( x) 0,, g m ( x) 0). Show that the set of " λ " components of saddle-points (if any) is bounded. hat is, show that the set { λ 0 xx ( x, λ is a saddle -point } is a bounded set [Hint: Use the definition of the saddle point.] Is the set for HW 50 bounded? [Note: he proof of heorem 22 is not entirely rigorous since λ may have some zero components (i.e., λ may be on boundary of { λ λ 0} and we have not said what we mean by L being differentiable at a boundary point.] 35
Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers
Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex
More informationLagrange Relaxation and Duality
Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler
More informationIntroduction to Optimization Techniques. Nonlinear Programming
Introduction to Optiization echniques Nonlinear Prograing Optial Solutions Consider the optiization proble in f ( x) where F R n xf Definition : x F is optial (global iniu) for this proble, if f( x ) f(
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationSome Properties of the Augmented Lagrangian in Cone Constrained Optimization
MATHEMATICS OF OPERATIONS RESEARCH Vol. 29, No. 3, August 2004, pp. 479 491 issn 0364-765X eissn 1526-5471 04 2903 0479 informs doi 10.1287/moor.1040.0103 2004 INFORMS Some Properties of the Augmented
More informationPrimal-dual Subgradient Method for Convex Problems with Functional Constraints
Primal-dual Subgradient Method for Convex Problems with Functional Constraints Yurii Nesterov, CORE/INMA (UCL) Workshop on embedded optimization EMBOPT2014 September 9, 2014 (Lucca) Yu. Nesterov Primal-dual
More informationCS Lecture 8 & 9. Lagrange Multipliers & Varitional Bounds
CS 6347 Lecture 8 & 9 Lagrange Multipliers & Varitional Bounds General Optimization subject to: min ff 0() R nn ff ii 0, h ii = 0, ii = 1,, mm ii = 1,, pp 2 General Optimization subject to: min ff 0()
More informationConvex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationAdditional Homework Problems
Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.
More informationPractice Exam 1: Continuous Optimisation
Practice Exam : Continuous Optimisation. Let f : R m R be a convex function and let A R m n, b R m be given. Show that the function g(x) := f(ax + b) is a convex function of x on R n. Suppose that f is
More informationCourse Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 4. Subgradient
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 4 Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 2 4.1. Subgradients definition subgradient calculus duality and optimality conditions Shiqian
More informationStructural and Multidisciplinary Optimization. P. Duysinx and P. Tossings
Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be
More informationDuality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities
Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationCONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS
CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS A Dissertation Submitted For The Award of the Degree of Master of Philosophy in Mathematics Neelam Patel School of Mathematics
More informationLecture 3 January 28
EECS 28B / STAT 24B: Advanced Topics in Statistical LearningSpring 2009 Lecture 3 January 28 Lecturer: Pradeep Ravikumar Scribe: Timothy J. Wheeler Note: These lecture notes are still rough, and have only
More informationLinear Programming. Larry Blume Cornell University, IHS Vienna and SFI. Summer 2016
Linear Programming Larry Blume Cornell University, IHS Vienna and SFI Summer 2016 These notes derive basic results in finite-dimensional linear programming using tools of convex analysis. Most sources
More informationFinite Dimensional Optimization Part III: Convex Optimization 1
John Nachbar Washington University March 21, 2017 Finite Dimensional Optimization Part III: Convex Optimization 1 1 Saddle points and KKT. These notes cover another important approach to optimization,
More informationsubject to (x 2)(x 4) u,
Exercises Basic definitions 5.1 A simple example. Consider the optimization problem with variable x R. minimize x 2 + 1 subject to (x 2)(x 4) 0, (a) Analysis of primal problem. Give the feasible set, the
More informationLP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra
LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality
More informationSubgradients. subgradients. strong and weak subgradient calculus. optimality conditions via subgradients. directional derivatives
Subgradients subgradients strong and weak subgradient calculus optimality conditions via subgradients directional derivatives Prof. S. Boyd, EE364b, Stanford University Basic inequality recall basic inequality
More informationHW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.
HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard
More information1. f(β) 0 (that is, β is a feasible point for the constraints)
xvi 2. The lasso for linear models 2.10 Bibliographic notes Appendix Convex optimization with constraints In this Appendix we present an overview of convex optimization concepts that are particularly useful
More informationLagrangian Duality and Convex Optimization
Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?
More informationIntroduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs
Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationOn duality theory of conic linear problems
On duality theory of conic linear problems Alexander Shapiro School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 3332-25, USA e-mail: ashapiro@isye.gatech.edu
More informationPrimal/Dual Decomposition Methods
Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients
More informationEE364b Homework 2. λ i f i ( x) = 0, i=1
EE364b Prof. S. Boyd EE364b Homework 2 1. Subgradient optimality conditions for nondifferentiable inequality constrained optimization. Consider the problem minimize f 0 (x) subject to f i (x) 0, i = 1,...,m,
More informationMath 273a: Optimization Convex Conjugacy
Math 273a: Optimization Convex Conjugacy Instructor: Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com Convex conjugate (the Legendre transform) Let f be a closed proper
More informationConvex analysis and profit/cost/support functions
Division of the Humanities and Social Sciences Convex analysis and profit/cost/support functions KC Border October 2004 Revised January 2009 Let A be a subset of R m Convex analysts may give one of two
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More informationEE364a Review Session 5
EE364a Review Session 5 EE364a Review announcements: homeworks 1 and 2 graded homework 4 solutions (check solution to additional problem 1) scpd phone-in office hours: tuesdays 6-7pm (650-723-1156) 1 Complementary
More informationConvex Optimization and Modeling
Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual
More information3.10 Lagrangian relaxation
3.10 Lagrangian relaxation Consider a generic ILP problem min {c t x : Ax b, Dx d, x Z n } with integer coefficients. Suppose Dx d are the complicating constraints. Often the linear relaxation and the
More informationSubgradients. subgradients and quasigradients. subgradient calculus. optimality conditions via subgradients. directional derivatives
Subgradients subgradients and quasigradients subgradient calculus optimality conditions via subgradients directional derivatives Prof. S. Boyd, EE392o, Stanford University Basic inequality recall basic
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationDUALITY, OPTIMALITY CONDITIONS AND PERTURBATION ANALYSIS
1 DUALITY, OPTIMALITY CONDITIONS AND PERTURBATION ANALYSIS Alexander Shapiro 1 School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332-0205, USA, E-mail: ashapiro@isye.gatech.edu
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationMaximal Monotone Inclusions and Fitzpatrick Functions
JOTA manuscript No. (will be inserted by the editor) Maximal Monotone Inclusions and Fitzpatrick Functions J. M. Borwein J. Dutta Communicated by Michel Thera. Abstract In this paper, we study maximal
More informationLagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark
Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More informationSymmetric and Asymmetric Duality
journal of mathematical analysis and applications 220, 125 131 (1998) article no. AY975824 Symmetric and Asymmetric Duality Massimo Pappalardo Department of Mathematics, Via Buonarroti 2, 56127, Pisa,
More informationEE 227A: Convex Optimization and Applications October 14, 2008
EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider
More informationConvex Optimization. (EE227A: UC Berkeley) Lecture 28. Suvrit Sra. (Algebra + Optimization) 02 May, 2013
Convex Optimization (EE227A: UC Berkeley) Lecture 28 (Algebra + Optimization) 02 May, 2013 Suvrit Sra Admin Poster presentation on 10th May mandatory HW, Midterm, Quiz to be reweighted Project final report
More informationLecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
More informationLecture 1: Background on Convex Analysis
Lecture 1: Background on Convex Analysis John Duchi PCMI 2016 Outline I Convex sets 1.1 Definitions and examples 2.2 Basic properties 3.3 Projections onto convex sets 4.4 Separating and supporting hyperplanes
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More informationw-(h,ω) CONJUGATE DUALITY THEORY IN MULTIOBJECTIVE NONLINEAR OPTIMIZATION 1
Management Science and Engineering Vol.1 No.2 December 2007 w-(h,ω) CONJUGATE DUALITY THEORY IN MULTIOBJECTIVE NONLINEAR OPTIMIZATION 1 Feng Junwen 2 Abstract: The duality in multiobjective optimization
More informationEnhanced Fritz John Optimality Conditions and Sensitivity Analysis
Enhanced Fritz John Optimality Conditions and Sensitivity Analysis Dimitri P. Bertsekas Laboratory for Information and Decision Systems Massachusetts Institute of Technology March 2016 1 / 27 Constrained
More informationSubgradient. Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes. definition. subgradient calculus
1/41 Subgradient Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes definition subgradient calculus duality and optimality conditions directional derivative Basic inequality
More informationLagrange Multipliers
Optimization with Constraints As long as algebra and geometry have been separated, their progress have been slow and their uses limited; but when these two sciences have been united, they have lent each
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationDuality Theory of Constrained Optimization
Duality Theory of Constrained Optimization Robert M. Freund April, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 The Practical Importance of Duality Duality is pervasive
More informationA notion of Total Dual Integrality for Convex, Semidefinite and Extended Formulations
A notion of for Convex, Semidefinite and Extended Formulations Marcel de Carli Silva Levent Tunçel April 26, 2018 A vector in R n is integral if each of its components is an integer, A vector in R n is
More informationSolving Dual Problems
Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem
More informationFranco Giannessi, Giandomenico Mastroeni. Institute of Mathematics University of Verona, Verona, Italy
ON THE THEORY OF VECTOR OPTIMIZATION AND VARIATIONAL INEQUALITIES. IMAGE SPACE ANALYSIS AND SEPARATION 1 Franco Giannessi, Giandomenico Mastroeni Department of Mathematics University of Pisa, Pisa, Italy
More informationIE 521 Convex Optimization Homework #1 Solution
IE 521 Convex Optimization Homework #1 Solution your NAME here your NetID here February 13, 2019 Instructions. Homework is due Wednesday, February 6, at 1:00pm; no late homework accepted. Please use the
More informationCHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008.
1 ECONOMICS 594: LECTURE NOTES CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS W. Erwin Diewert January 31, 2008. 1. Introduction Many economic problems have the following structure: (i) a linear function
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationBASICS OF CONVEX ANALYSIS
BASICS OF CONVEX ANALYSIS MARKUS GRASMAIR 1. Main Definitions We start with providing the central definitions of convex functions and convex sets. Definition 1. A function f : R n R + } is called convex,
More informationLecture 25: Subgradient Method and Bundle Methods April 24
IE 51: Convex Optimization Spring 017, UIUC Lecture 5: Subgradient Method and Bundle Methods April 4 Instructor: Niao He Scribe: Shuanglong Wang Courtesy warning: hese notes do not necessarily cover everything
More informationLecture 8 Plus properties, merit functions and gap functions. September 28, 2008
Lecture 8 Plus properties, merit functions and gap functions September 28, 2008 Outline Plus-properties and F-uniqueness Equation reformulations of VI/CPs Merit functions Gap merit functions FP-I book:
More information4. Algebra and Duality
4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone
More informationMIT Algebraic techniques and semidefinite optimization February 14, Lecture 3
MI 6.97 Algebraic techniques and semidefinite optimization February 4, 6 Lecture 3 Lecturer: Pablo A. Parrilo Scribe: Pablo A. Parrilo In this lecture, we will discuss one of the most important applications
More informationOptimization and Optimal Control in Banach Spaces
Optimization and Optimal Control in Banach Spaces Bernhard Schmitzer October 19, 2017 1 Convex non-smooth optimization with proximal operators Remark 1.1 (Motivation). Convex optimization: easier to solve,
More informationLecture 2: Linear SVM in the Dual
Lecture 2: Linear SVM in the Dual Stéphane Canu stephane.canu@litislab.eu São Paulo 2015 July 22, 2015 Road map 1 Linear SVM Optimization in 10 slides Equality constraints Inequality constraints Dual formulation
More informationSubgradient Descent. David S. Rosenberg. New York University. February 7, 2018
Subgradient Descent David S. Rosenberg New York University February 7, 2018 David S. Rosenberg (New York University) DS-GA 1003 / CSCI-GA 2567 February 7, 2018 1 / 43 Contents 1 Motivation and Review:
More informationA convergence result for an Outer Approximation Scheme
A convergence result for an Outer Approximation Scheme R. S. Burachik Engenharia de Sistemas e Computação, COPPE-UFRJ, CP 68511, Rio de Janeiro, RJ, CEP 21941-972, Brazil regi@cos.ufrj.br J. O. Lopes Departamento
More informationJENSEN S OPERATOR INEQUALITY AND ITS CONVERSES
MAH. SCAND. 100 (007, 61 73 JENSEN S OPERAOR INEQUALIY AND IS CONVERSES FRANK HANSEN, JOSIP PEČARIĆ and IVAN PERIĆ (Dedicated to the memory of Gert K. Pedersen Abstract We give a general formulation of
More informationLinear and non-linear programming
Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)
More informationCourse Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course otes for EE7C (Spring 018): Conve Optimization and Approimation Instructor: Moritz Hardt Email: hardt+ee7c@berkeley.edu Graduate Instructor: Ma Simchowitz Email: msimchow+ee7c@berkeley.edu October
More informationDuality. for The New Palgrave Dictionary of Economics, 2nd ed. Lawrence E. Blume
Duality for The New Palgrave Dictionary of Economics, 2nd ed. Lawrence E. Blume Headwords: CONVEXITY, DUALITY, LAGRANGE MULTIPLIERS, PARETO EFFICIENCY, QUASI-CONCAVITY 1 Introduction The word duality is
More informationLagrange Relaxation: Introduction and Applications
1 / 23 Lagrange Relaxation: Introduction and Applications Operations Research Anthony Papavasiliou 2 / 23 Contents 1 Context 2 Applications Application in Stochastic Programming Unit Commitment 3 / 23
More informationLecture 8. Strong Duality Results. September 22, 2008
Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation
More information106 CHAPTER 3. TOPOLOGY OF THE REAL LINE. 2. The set of limit points of a set S is denoted L (S)
106 CHAPTER 3. TOPOLOGY OF THE REAL LINE 3.3 Limit Points 3.3.1 Main Definitions Intuitively speaking, a limit point of a set S in a space X is a point of X which can be approximated by points of S other
More informationResearch Article Sufficient Optimality and Sensitivity Analysis of a Parameterized Min-Max Programming
Applied Mathematics Volume 2012, Article ID 692325, 9 pages doi:10.1155/2012/692325 Research Article Sufficient Optimality and Sensitivity Analysis of a Parameterized Min-Max Programming Huijuan Xiong,
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationExample: feasibility. Interpretation as formal proof. Example: linear inequalities and Farkas lemma
4-1 Algebra and Duality P. Parrilo and S. Lall 2006.06.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone of valid
More informationICS-E4030 Kernel Methods in Machine Learning
ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationLagrangian Duality Theory
Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual
More informationIteration-complexity of first-order penalty methods for convex programming
Iteration-complexity of first-order penalty methods for convex programming Guanghui Lan Renato D.C. Monteiro July 24, 2008 Abstract This paper considers a special but broad class of convex programing CP)
More informationLagrangian Duality. Evelien van der Hurk. DTU Management Engineering
Lagrangian Duality Evelien van der Hurk DTU Management Engineering Topics Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality 2 DTU Management Engineering 42111: Static and Dynamic Optimization
More informationA SET OF LECTURE NOTES ON CONVEX OPTIMIZATION WITH SOME APPLICATIONS TO PROBABILITY THEORY INCOMPLETE DRAFT. MAY 06
A SET OF LECTURE NOTES ON CONVEX OPTIMIZATION WITH SOME APPLICATIONS TO PROBABILITY THEORY INCOMPLETE DRAFT. MAY 06 CHRISTIAN LÉONARD Contents Preliminaries 1 1. Convexity without topology 1 2. Convexity
More informationOptimization Theory. Lectures 4-6
Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x$0} Existence of a maximum:
More informationINVEX FUNCTIONS AND CONSTRAINED LOCAL MINIMA
BULL. AUSRAL. MAH. SOC. VOL. 24 (1981), 357-366. 9C3 INVEX FUNCIONS AND CONSRAINED LOCAL MINIMA B.D. CRAVEN If a certain weakening of convexity holds for the objective and all constraint functions in a
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More informationLecture 2: Convex Sets and Functions
Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are
More informationVictoria Martín-Márquez
A NEW APPROACH FOR THE CONVEX FEASIBILITY PROBLEM VIA MONOTROPIC PROGRAMMING Victoria Martín-Márquez Dep. of Mathematical Analysis University of Seville Spain XIII Encuentro Red de Análisis Funcional y
More informationRobust Duality in Parametric Convex Optimization
Robust Duality in Parametric Convex Optimization R.I. Boţ V. Jeyakumar G.Y. Li Revised Version: June 20, 2012 Abstract Modelling of convex optimization in the face of data uncertainty often gives rise
More informationPrimal Solutions and Rate Analysis for Subgradient Methods
Primal Solutions and Rate Analysis for Subgradient Methods Asu Ozdaglar Joint work with Angelia Nedić, UIUC Conference on Information Sciences and Systems (CISS) March, 2008 Department of Electrical Engineering
More informationSupport Vector Machines for Regression
COMP-566 Rohan Shah (1) Support Vector Machines for Regression Provided with n training data points {(x 1, y 1 ), (x 2, y 2 ),, (x n, y n )} R s R we seek a function f for a fixed ɛ > 0 such that: f(x
More informationA New Fenchel Dual Problem in Vector Optimization
A New Fenchel Dual Problem in Vector Optimization Radu Ioan Boţ Anca Dumitru Gert Wanka Abstract We introduce a new Fenchel dual for vector optimization problems inspired by the form of the Fenchel dual
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming
E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program
More information