Enhanced Fritz John Optimality Conditions and Sensitivity Analysis
|
|
- Belinda Lloyd
- 5 years ago
- Views:
Transcription
1 Enhanced Fritz John Optimality Conditions and Sensitivity Analysis Dimitri P. Bertsekas Laboratory for Information and Decision Systems Massachusetts Institute of Technology March / 27
2 Constrained Optimization Problem We focus on the problem minimize f (x) subject to x X, g j (x) 0, j = 1,..., r, where f : R n R, g j : R n R, and X R n. The Lagrangian function L(x, µ) = f (x) + involves the multiplier vector µ = (µ 1,..., µ r ). r µ j g j (x) j=1 We aim to find µ (Langange multipliers) that facilitate analysis or algorithms. 2 / 27
3 Salient Properties of Lagrange Multipliers They enter in optimality conditions: They convert the problem to an unconstrained or less constrained optimization" of the Lagrangian They are central in sensitivity analysis: They quantify the rate of cost improvement as the constraint level is perturbed. Lagrange multipliers are some sort of derivative" of the primal function p(u) = inf f (x) x X, g(x) u This talk will revolve around these two properties. 3 / 27
4 Fritz John Conditions: A Classical Line of Development of L-Multiplier Theory There are several different lines of development of L-multiplier theory (forms of the implicit function theorem, forms of Farkas lemma, penalty functions, etc) FJ conditions is a classical line but not the most popular This talk will be in the direction of strengthening this line Our starting point A more powerful version of the classical FJ conditions They include extra conditions that narrow down the candidate multipliers It is based on the combination of two related but complementary works: Hestenes (1975) and Rockafellar (1993) The line of proof is based on penalty functions; goes back to a 4-page paper by McShane (1973) Allows: An easy" development of unifying constraint qualifications A direct connection to sensitivity 4 / 27
5 References for this Overview Talk Joint and individual works with Asuman Ozdaglar and Paul Tseng Bertsekas and Ozdaglar, Pseudonormality and a Lagrange Multiplier Theory for Constrained Optimization," J. Opt. Th. and Appl. Ozdaglar and Bertsekas, The Relation Between Pseudonormality and Quasiregularity in Constrained Optimization," Optimization Methods and Software. Bertsekas, Lagrange Multipliers with Optimal Sensitivity Properties in Constrained Optimization," in Proc. of the 2004 Erice Workshop on Large Scale Nonlinear Optimization, Erice, Italy, Kluwer. Bertsekas, Ozdaglar, and Tseng, Enhanced Fritz John Optimality Conditions for Convex Programming," SIAM J. on Optimization. An umbrella reference is the book Bertsekas, D. P., Nedić, A., and Ozdaglar, A. E., Convex Analysis and Optimization, Athena Scientific, Belmont, MA. But it does not contain some refinements in the last three papers. 5 / 27
6 Outline 1 Enhanced FJ Conditions for Nonconvex/Differentiable Problems 2 Pseudonormality: A Unifying Constraint Qualification 3 Sensitivity for Nonconvex/Differentiable Problems 4 Convex Problems 5 Sensitivity Analysis for Convex Problems 6 / 27
7 Lagrange Multipliers for Nonconvex Differentiable Problems minimize f (x) subject to x X, g j (x) 0, j = 1,..., r, where f : R n R, g j : R n R, are cont. differentiable, and X R n is closed. The Lagrangian function is r L(x, µ) = f (x) + µ j g j (x) j=1 We focus at a local minimum x A L-multiplier at x is a µ = (µ 1,..., µ r ) 0 such that L(, µ ) is stationary at x and g j (x ) = 0 j with µ j > 0, Complementary Slackness (CS) Meaning of stationarity Case X = R n : xl(x, µ ) = 0 Case X R n : xl(x, µ ) y 0 for all y T x(x ), the tangent cone of X at x (we will define this later). 8 / 27
8 FJ Necessary Conditions for Case X = R n (Hestenes 1975) There exists (µ 0, µ ) (0, 0) such that (µ 0, µ ) (0, 0) and 1 µ 0 f (x ) + r j=1 µ j g j (x ) = 0 2 In every neighborhood of x there exists x such that f (x) < f (x ), and g j (x) > 0 j with µ j > 0 Condition (2) (called CV, for complementary violation) = CS Two Cases µ 0 = 0. Then µ 0 and satisfies r j=1 µ j g j (x ) = 0 and the stronger CV condition. This greatly facilitates proofs that µ 0 0 under various constraint qualifications. µ 0 = 1. Then µ is a L-multiplier and the positive µ j indicate the constraints j that need to be violated to effect cost improvement - a sensitivity property. 9 / 27
9 An Example A problem with equality constraint h(x) = 0 split as h(x) 0 and h(x) 0 apple x f(x) =f(x ) g 2 (x) =h(x) apple 0 g 1 (x) = h(x) apple 0 rf(x ) x rh(x ) µ 2 =0 µ 1 > 0 CS requires that µ 1 0 and µ 2 0 (there is an infinite number of these) CV requires that µ 1 > 0 and µ 2 = 0 (we cannot violate simultaneously both constraints) The multiplier satisfying CV is unique. Through its sign pattern indicates that the constraint g 1 (x) 0 should be violated for cost reduction. 10 / 27
10 Case X R n (Rockafellar 1993) Extension of FJ conditions with the Lagrangian stationarity at the local min x expressed in terms of N X (x ), the normal cone of X at x. Definitions of Tangent Cone T X (x ) and Normal Cone N X (x ) The tangent cone T X (x) at some x X is the set of all y such that y = 0 or there exists a sequence {x k } X such that x k x for all k and x k x, x k x x k x y y The normal cone N X (x) at some x X is the set of all z such that there exist sequences {x k } X and {z k } such that x k x, z k z, and z k T X (x k ) [the polar of T X (x k )]. Note that T X (x ) N X (x ). If equality holds, X is called regular at x (if X is convex, it is regular at all points). Convex X Not Regular X = TX(x) TX(x) TX(x) = {0} NX(x) x = 0 x = 0 NX(x) 11 / 27
11 FJ Conditions for Case X R n (Rockafellar 1993) Constrained optimization problem minimize f (x) subject to x X, g j (x) 0, j = 1,..., r, where f : R n R, g : R n R, are cont. differentiable, and X R n is closed. FJ Conditions There exists (µ 0, µ ) (0, 0) such that (µ 0, µ ) (0, 0) and ( 1 µ 0 f (x ) + ) r j=1 µ j g j (x ) N X (x ) 2 g j (x ) = 0 j with µ j > 0, i.e., CS holds. If µ 0 = 1 and X is regular at x, then N X (x ) = T X (x ), and condition (1) becomes r f (x ) + µ j g j (x ) y 0, y T X (x ), j=1 so µ is a L-multiplier satisfying CS. 12 / 27
12 Enhanced FJ Necessary Conditions for Case X R n (B+O 2002) Constrained optimization problem minimize f (x) subject to x X, g j (x) 0, j = 1,..., r, where f : R n R, g : R n R, are cont. differentiable, and X R n is closed. There exists (µ 0, µ ) (0, 0) such that (µ 0, µ ) (0, 0) and 1 ( µ 0 f (x ) + r j=1 µ j g j (x ) ) N X (x ) 2 In every neighborhood of x there exists x such that i.e., CV holds. f (x) < f (x ), and g j (x) > 0 j with µ j > 0 So if µ 0 = 1 and X is regular at x, then µ is a L-multiplier satisfying CV. We call such a multiplier informative. 13 / 27
13 Constraint Qualifications: Conditions Under Which µ 0 0 Pseudonormality: An umbrella constraint qualification (B+O 2002) x is a pseudonormal local min if there is no µ 0 and sequence {x k } X with x k x such that r r µ j g j (x ) N X (x ), µ j g j (x k ) > 0, k. j=1 j=1 Note: If x is pseudonormal and X is regular at x there is an informative L-multiplier. All the principal constraint qualifications for existence of L-multipliers and some new ones can be shown (very easily) to imply pseudonormality. Linear constraints Linear independence A Slater constraint qualification Concave inequalities Arrow-Hurwitz- Uzawa-Mangasarian Pseudonormality 15 / 27
14 Sensitivity: A Hierarchy of L-Multipliers (B+O 2002, 2005) Assume T X (x ) is convex and the set of L-multipliers is nonempty. Then: L-Multipliers (satisfy CS) Informative L-Multipliers (satisfy CV) Sensitivity Result Min-Norm L-Multiplier µ Cost improvement bound: For every sequence of infeasible vectors {x k } X with x k x we have f (x ) f (x k ) µ g + (x k ) + o ( x k x ), where g + (x) = ( g + 1 (x),..., g+ r (x) ) : constraint violation vector. Optimal constraint violation direction: If µ 0, there exists infeasible sequence {x k } X with x k x and such that lim k f (x ) f (x k ) g+ (x k ) g + = µ j (x k ), lim k g+ (x k ) = µ j, j = 1,..., r. µ 17 / 27
15 The Convex Case (B+O+T 2006) Consider the problem minimize f (x) subject to x X, g j (x) 0, j = 1,..., r, where f : R n R, g : R n R, are convex, and X R n is convex and closed. Primal function: p(u) = inf x X, g(x) u f (x), Dual function: q(µ) = inf x X L(x, µ) Optimal primal value: f = p(0), Optimal dual value: q = sup µ 0 q(µ) Duality gap: f q µ is Lagrange multiplier if f q = 0 and µ is a dual optimal solution Primal and dual FJ conditions adapted to convex programming There exists (µ 0, µ ) (0, 0) such that (µ 0, µ ) (0, 0), and { µ 0f = inf x X µ 0f (x) + } r j=1 µ j g j (x) (Primal FJ Conditions) { µ 0q = inf x X µ 0f (x) + } r j=1 µ j g j (x) (Dual FJ Conditions) and CS holds (there are versions with CV also). 19 / 27
16 Visualization with the Primal Function p(u) = inf x X, g(x) u f (x) p(u) µ 0 = 1 (0, f ) (µ 0, µ ) 0 g(x k ) u The most favorable case: p(0) exists, and µ = p(0) is the unique L-multiplier Rate of cost improvement: The slope of the support hyperplane at (0, f ) 21 / 27
17 Visualization with the Primal Function p(u) = inf x X, g(x) u f (x) p(u) µ 0 = 1 (0, f ) (µ 0, µ ) 0 g(x k ) u p is subdifferentiable at 0. Set of L-multipliers = p(0). Optimal rate of cost improvement: The slope of the min norm support hyperplane at (0, f ) 22 / 27
18 Visualization with the Primal Function p(u) = inf x X, g(x) u f (x) p(u) p(u) µ 0 = 1 (0, f ) µ 0 = 0 (0, f ) (µ 0, µ ) (µ 0, µ ) 0 g(x k ) u 0 g(x k ) u 23 / 27
19 Visualization with the Primal Function p(u) = inf x X, g(x) u f (x) p(u) µ 0 = 1 (0, f ) p(u) µ 0 = 0 (0, f ) (µ 0, µ ) (µ 0, µ ) 0 g(x k ) u 0 g(x k ) u p(u) (0, f ) (0, q ) µ 0 = 1 (µ 0, µ ) 0 g(x k ) u 24 / 27
20 Visualization with the Primal Function p(u) = inf x X, g(x) u f (x) p(u) p(u) µ 0 = 1 (0, f ) µ 0 = 0 (0, f ) (µ 0, µ ) (µ 0, µ ) 0 g(x k ) u 0 g(x k ) u p(u) p(u) (0, f ) µ 0 = 1 (0, f ) µ 0 = 0 (0, q ) (µ 0, µ ) (0, q ) (µ 0, µ ) 0 g(x k ) u 0 g(x k ) u 25 / 27
21 Dual Sensitivity Result Assume that < q f <, and that the set of dual optimal solutions is nonempty. Let µ be the min norm dual optimal solution (may not be a L-multiplier): Cost improvement bound: For all x X q f (x) µ g + (x) If µ 0, there exists infeasible sequence {x k } X such that the bound is asymptotically sharp, f (x k ) q, g + (x k ) 0, q f (x k ) g+ (x k ) µ. Also µ is the optimal constraint violation direction, g + (x k ) g+ (x k ) µ µ [Note: { g + (x k ) } may need to lie on a curve]. 26 / 27
22 Thank you! 27 / 27
Date: July 5, Contents
2 Lagrange Multipliers Date: July 5, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 14 2.3. Informative Lagrange Multipliers...........
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationThe Relation Between Pseudonormality and Quasiregularity in Constrained Optimization 1
October 2003 The Relation Between Pseudonormality and Quasiregularity in Constrained Optimization 1 by Asuman E. Ozdaglar and Dimitri P. Bertsekas 2 Abstract We consider optimization problems with equality,
More informationLagrange Multipliers with Optimal Sensitivity Properties in Constrained Optimization
Lagrange Multipliers with Optimal Sensitivity Properties in Constrained Optimization Dimitri P. Bertsekasl Dept. of Electrical Engineering and Computer Science, M.I.T., Cambridge, Mass., 02139, USA. (dimitribhit.
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationA Geometric Framework for Nonconvex Optimization Duality using Augmented Lagrangian Functions
A Geometric Framework for Nonconvex Optimization Duality using Augmented Lagrangian Functions Angelia Nedić and Asuman Ozdaglar April 15, 2006 Abstract We provide a unifying geometric framework for the
More informationA Unified Analysis of Nonconvex Optimization Duality and Penalty Methods with General Augmenting Functions
A Unified Analysis of Nonconvex Optimization Duality and Penalty Methods with General Augmenting Functions Angelia Nedić and Asuman Ozdaglar April 16, 2006 Abstract In this paper, we study a unifying framework
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More informationNumerical Optimization
Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,
More informationNonlinear Programming 3rd Edition. Theoretical Solutions Manual Chapter 6
Nonlinear Programming 3rd Edition Theoretical Solutions Manual Chapter 6 Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts 1 NOTE This manual contains
More informationConvex Optimization Theory. Chapter 4 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 4 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationPrimal Solutions and Rate Analysis for Subgradient Methods
Primal Solutions and Rate Analysis for Subgradient Methods Asu Ozdaglar Joint work with Angelia Nedić, UIUC Conference on Information Sciences and Systems (CISS) March, 2008 Department of Electrical Engineering
More informationLECTURE 10 LECTURE OUTLINE
LECTURE 10 LECTURE OUTLINE Min Common/Max Crossing Th. III Nonlinear Farkas Lemma/Linear Constraints Linear Programming Duality Convex Programming Duality Optimality Conditions Reading: Sections 4.5, 5.1,5.2,
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More information1. f(β) 0 (that is, β is a feasible point for the constraints)
xvi 2. The lasso for linear models 2.10 Bibliographic notes Appendix Convex optimization with constraints In this Appendix we present an overview of convex optimization concepts that are particularly useful
More informationOptimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers
Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex
More informationSome Properties of the Augmented Lagrangian in Cone Constrained Optimization
MATHEMATICS OF OPERATIONS RESEARCH Vol. 29, No. 3, August 2004, pp. 479 491 issn 0364-765X eissn 1526-5471 04 2903 0479 informs doi 10.1287/moor.1040.0103 2004 INFORMS Some Properties of the Augmented
More informationLagrangian Duality Theory
Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationAdditional Homework Problems
Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.
More informationConvex Optimization and Modeling
Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual
More information4. Algebra and Duality
4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone
More informationSubgradient Methods in Network Resource Allocation: Rate Analysis
Subgradient Methods in Networ Resource Allocation: Rate Analysis Angelia Nedić Department of Industrial and Enterprise Systems Engineering University of Illinois Urbana-Champaign, IL 61801 Email: angelia@uiuc.edu
More informationConvex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationPrimal-dual Subgradient Method for Convex Problems with Functional Constraints
Primal-dual Subgradient Method for Convex Problems with Functional Constraints Yurii Nesterov, CORE/INMA (UCL) Workshop on embedded optimization EMBOPT2014 September 9, 2014 (Lucca) Yu. Nesterov Primal-dual
More informationUNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems
UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationLecture 18: Optimization Programming
Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming
More informationConstrained Optimization Theory
Constrained Optimization Theory Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Constrained Optimization Theory IMA, August
More informationLecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
More informationKarush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725
Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =
More informationNonlinear Programming and the Kuhn-Tucker Conditions
Nonlinear Programming and the Kuhn-Tucker Conditions The Kuhn-Tucker (KT) conditions are first-order conditions for constrained optimization problems, a generalization of the first-order conditions we
More informationLecture 8. Strong Duality Results. September 22, 2008
Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationExample: feasibility. Interpretation as formal proof. Example: linear inequalities and Farkas lemma
4-1 Algebra and Duality P. Parrilo and S. Lall 2006.06.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone of valid
More informationsubject to (x 2)(x 4) u,
Exercises Basic definitions 5.1 A simple example. Consider the optimization problem with variable x R. minimize x 2 + 1 subject to (x 2)(x 4) 0, (a) Analysis of primal problem. Give the feasible set, the
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More informationMachine Learning And Applications: Supervised Learning-SVM
Machine Learning And Applications: Supervised Learning-SVM Raphaël Bournhonesque École Normale Supérieure de Lyon, Lyon, France raphael.bournhonesque@ens-lyon.fr 1 Supervised vs unsupervised learning Machine
More informationEE/AA 578, Univ of Washington, Fall Duality
7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationPseudonormality and a Lagrange Multiplier Theory for Constrained Optimization 1
November 2000 Submitted for publication to JOTA Pseudonormality and a Lagrange Multiplier Theory for Constrained Optimization 1 by Dimitri P. Bertsekas and Asuman E. Ozdaglar 2 Abstract We consider optimization
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationDuality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities
Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form
More informationConstrained Optimization
1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange
More informationIntroduction to Optimization Techniques. Nonlinear Optimization in Function Spaces
Introduction to Optimization Techniques Nonlinear Optimization in Function Spaces X : T : Gateaux and Fréchet Differentials Gateaux and Fréchet Differentials a vector space, Y : a normed space transformation
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationExtended Monotropic Programming and Duality 1
March 2006 (Revised February 2010) Report LIDS - 2692 Extended Monotropic Programming and Duality 1 by Dimitri P. Bertsekas 2 Abstract We consider the problem minimize f i (x i ) subject to x S, where
More informationApproximate Primal Solutions and Rate Analysis for Dual Subgradient Methods
Approximate Primal Solutions and Rate Analysis for Dual Subgradient Methods Angelia Nedich Department of Industrial and Enterprise Systems Engineering University of Illinois at Urbana-Champaign 117 Transportation
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationOptimality, Duality, Complementarity for Constrained Optimization
Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear
More informationLagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
More informationON LICQ AND THE UNIQUENESS OF LAGRANGE MULTIPLIERS
ON LICQ AND THE UNIQUENESS OF LAGRANGE MULTIPLIERS GERD WACHSMUTH Abstract. Kyparisis proved in 1985 that a strict version of the Mangasarian- Fromovitz constraint qualification (MFCQ) is equivalent to
More informationLectures 9 and 10: Constrained optimization problems and their optimality conditions
Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained
More informationEE364a Review Session 5
EE364a Review Session 5 EE364a Review announcements: homeworks 1 and 2 graded homework 4 solutions (check solution to additional problem 1) scpd phone-in office hours: tuesdays 6-7pm (650-723-1156) 1 Complementary
More informationCONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS
CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS A Dissertation Submitted For The Award of the Degree of Master of Philosophy in Mathematics Neelam Patel School of Mathematics
More informationLinear and non-linear programming
Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationPrimal/Dual Decomposition Methods
Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients
More informationChap 2. Optimality conditions
Chap 2. Optimality conditions Version: 29-09-2012 2.1 Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1
More informationPart IB Optimisation
Part IB Optimisation Theorems Based on lectures by F. A. Fischer Notes taken by Dexter Chua Easter 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after
More informationLagrange Relaxation and Duality
Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler
More informationSolutions Chapter 5. The problem of finding the minimum distance from the origin to a line is written as. min 1 2 kxk2. subject to Ax = b.
Solutions Chapter 5 SECTION 5.1 5.1.4 www Throughout this exercise we will use the fact that strong duality holds for convex quadratic problems with linear constraints (cf. Section 3.4). The problem of
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationHomework Set #6 - Solutions
EE 15 - Applications of Convex Optimization in Signal Processing and Communications Dr Andre Tkacenko JPL Third Term 11-1 Homework Set #6 - Solutions 1 a The feasible set is the interval [ 4] The unique
More informationNonlinear Optimization
Nonlinear Optimization Etienne de Klerk (UvT)/Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos Course WI3031 (Week 4) February-March, A.D. 2005 Optimization Group 1 Outline
More informationSupport Vector Machines
Support Vector Machines Support vector machines (SVMs) are one of the central concepts in all of machine learning. They are simply a combination of two ideas: linear classification via maximum (or optimal
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationLagrangian Duality and Convex Optimization
Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?
More informationLagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark
Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian
More informationLecture 5. Theorems of Alternatives and Self-Dual Embedding
IE 8534 1 Lecture 5. Theorems of Alternatives and Self-Dual Embedding IE 8534 2 A system of linear equations may not have a solution. It is well known that either Ax = c has a solution, or A T y = 0, c
More informationIntroduction to Nonlinear Stochastic Programming
School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS
More informationSymmetric and Asymmetric Duality
journal of mathematical analysis and applications 220, 125 131 (1998) article no. AY975824 Symmetric and Asymmetric Duality Massimo Pappalardo Department of Mathematics, Via Buonarroti 2, 56127, Pisa,
More informationSolution Methods for Stochastic Programs
Solution Methods for Stochastic Programs Huseyin Topaloglu School of Operations Research and Information Engineering Cornell University ht88@cornell.edu August 14, 2010 1 Outline Cutting plane methods
More informationSemi-infinite programming, duality, discretization and optimality conditions
Semi-infinite programming, duality, discretization and optimality conditions Alexander Shapiro School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332-0205,
More informationSupport Vector Machines: Maximum Margin Classifiers
Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and Fu-Jie Huang 1 Outline What is behind
More informationCONSTRAINED NONLINEAR PROGRAMMING
149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach
More informationLinear and Combinatorial Optimization
Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality
More informationConvexity, Duality, and Lagrange Multipliers
LECTURE NOTES Convexity, Duality, and Lagrange Multipliers Dimitri P. Bertsekas with assistance from Angelia Geary-Nedic and Asuman Koksal Massachusetts Institute of Technology Spring 2001 These notes
More informationOn the Method of Lagrange Multipliers
On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should
More informationLecture Note 5: Semidefinite Programming for Stability Analysis
ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State
More informationExistence of Global Minima for Constrained Optimization 1
Existence of Global Minima for Constrained Optimization 1 A. E. Ozdaglar 2 and P. Tseng 3 Communicated by A. Miele 1 We thank Professor Dimitri Bertsekas for his comments and support in the writing of
More informationA FRITZ JOHN APPROACH TO FIRST ORDER OPTIMALITY CONDITIONS FOR MATHEMATICAL PROGRAMS WITH EQUILIBRIUM CONSTRAINTS
A FRITZ JOHN APPROACH TO FIRST ORDER OPTIMALITY CONDITIONS FOR MATHEMATICAL PROGRAMS WITH EQUILIBRIUM CONSTRAINTS Michael L. Flegel and Christian Kanzow University of Würzburg Institute of Applied Mathematics
More informationNonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More informationLECTURE 12 LECTURE OUTLINE. Subgradients Fenchel inequality Sensitivity in constrained optimization Subdifferential calculus Optimality conditions
LECTURE 12 LECTURE OUTLINE Subgradients Fenchel inequality Sensitivity in constrained optimization Subdifferential calculus Optimality conditions Reading: Section 5.4 All figures are courtesy of Athena
More informationCONVEX OPTIMIZATION VIA LINEARIZATION. Miguel A. Goberna. Universidad de Alicante. Iberian Conference on Optimization Coimbra, November, 2006
CONVEX OPTIMIZATION VIA LINEARIZATION Miguel A. Goberna Universidad de Alicante Iberian Conference on Optimization Coimbra, 16-18 November, 2006 Notation X denotes a l.c. Hausdorff t.v.s and X its topological
More informationLecture 2: Linear SVM in the Dual
Lecture 2: Linear SVM in the Dual Stéphane Canu stephane.canu@litislab.eu São Paulo 2015 July 22, 2015 Road map 1 Linear SVM Optimization in 10 slides Equality constraints Inequality constraints Dual formulation
More informationMinimax Problems. Daniel P. Palomar. Hong Kong University of Science and Technolgy (HKUST)
Mini Problems Daniel P. Palomar Hong Kong University of Science and Technolgy (HKUST) ELEC547 - Convex Optimization Fall 2009-10, HKUST, Hong Kong Outline of Lecture Introduction Matrix games Bilinear
More informationLP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra
LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality
More informationLecture 7: Convex Optimizations
Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1
More informationApproximate Farkas Lemmas in Convex Optimization
Approximate Farkas Lemmas in Convex Optimization Imre McMaster University Advanced Optimization Lab AdvOL Graduate Student Seminar October 25, 2004 1 Exact Farkas Lemma Motivation 2 3 Future plans The
More informationCS-E4830 Kernel Methods in Machine Learning
CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationLagrangian Duality. Evelien van der Hurk. DTU Management Engineering
Lagrangian Duality Evelien van der Hurk DTU Management Engineering Topics Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality 2 DTU Management Engineering 42111: Static and Dynamic Optimization
More informationSupport Vector Machines
EE 17/7AT: Optimization Models in Engineering Section 11/1 - April 014 Support Vector Machines Lecturer: Arturo Fernandez Scribe: Arturo Fernandez 1 Support Vector Machines Revisited 1.1 Strictly) Separable
More information