# Convex Optimization & Lagrange Duality

Save this PDF as:

Size: px
Start display at page: ## Transcription

1 Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010

2 Outline Convex optimization Optimality condition Lagrange duality KKT optimality condition Sensitivity analysis Acknowledgement: Thanks to Mung Chiang (Princeton), Stephen Boyd (Stanford) and Steven Low (Caltech) for the course materials in this class. 1

3 Convex Optimization A convex optimization problem with variables x: minimize subject to f 0 (x) f i (x) 0, i = 1, 2,..., m a T i x = b i, i = 1, 2,..., p where f 0, f 1,..., f m are convex functions. Minimize convex objective function (or maximize concave objective function) Upper bound inequality constraints on convex functions ( Constraint set is convex) Equality constraints must be affine 2

4 Convex Optimization Epigraph form: minimize t subject to f 0 (x) t 0 f i (x) 0, i = 1, 2,..., m a T i x = b i, i = 1, 2,..., p Not in convex optimization form: minimize x x 2 2 subject to 0 x 1 1+x 2 2 (x 1 + x 2 ) 2 = 0 3

5 Now transformed into a convex optimization problem: minimize x x 2 2 subject to x 1 0 x 1 + x 2 = 0 4

6 Locally Optimal Globally Optimal Given x is locally optimal for a convex optimization problem, i.e., x is feasible and for some R > 0, f 0 (x) = inf{f 0 (z) z is feasible, z x 2 R} Suppose x is not globally optimal, i.e., there is a feasible y such that f 0 (y) < f 0 (x) Since y x 2 > R, we can construct a point z = (1 θ)x + θy where R θ = 2 y x 2. By convexity of feasible set, z is feasible. By convexity of f 0, we have f 0 (z) (1 θ)f 0 (x) + θf 0 (y) < f 0 (x) which contradicts locally optimality of x Therefore, there exists no feasible y such that f 0 (y) < f 0 (x) 5

7 Optimality Condition for Differentiable f 0 x is optimal for a convex optimization problem iff x is feasible and for all feasible y: f 0 (x) T (y x) 0 f 0 (x) is supporting hyperplane to feasible set Unconstrained convex optimization: condition reduces to: f 0 (x) = 0 Proof: take y = x t f 0 (x) where t R +. For small enough t, y is feasible, so f 0 (x) T (y x) = t f 0 (x) Thus f 0 (x) = 0 6

8 Unconstrained Quadratic Optimization Minimize f 0 (x) = 1 2 xt P x + q T x + r P is positive semidefinite. So it s a convex optimization problem x minimizes f 0 iff (P, q) satisfy this linear equality: f 0 (x) = P x + q = 0 If q / R(P ), no solution. f 0 unbounded below If q R(P ) and P 0, there is a unique minimizer x = P 1 q If q R(P ) and P is singular, set of optimal x: P q + N (P ) 7

9 Equality Constrained Convex Optimization minimize subject to f 0 (x) Ax = b Assume nonempty feasible set. Since f 0 (x) T (y x) 0, y : Ay = b, and every feasible y can be written as y = x + v for some v N (A), optimality condition becomes: f 0 (x) T v 0, v N (A) which implies f 0 (x) T v = 0, i.e., f 0 (x) is orthogonal to N (A). Thus f 0 (x) R(A T ), i.e., there exists ν such that f 0 (x) + A T ν = 0 8

10 Conclusion: x is optimal iff Ax = b and ν s.t. f 0 (x) + A T ν = 0 9

11 Duality Mentality Bound or solve an optimization problem via a different optimization problem! We ll develop the basic Lagrange duality theory for a general optimization problem, then specialize for convex optimization 10

12 Lagrange Dual Function An optimization problem in standard form: minimize subject to f 0 (x) f i (x) 0, i = 1, 2,..., m h i (x) = 0, i = 1, 2,..., p Variables: x R n. Assume nonempty feasible set Optimal value: p. Optimizer: x Idea: augment objective with a weighted sum of constraints Lagrangian L(x, λ, ν) = f 0 (x) + m i=1 λ if i (x) + p i=1 ν ih i (x) Lagrange multipliers (dual variables): λ 0, ν Lagrange dual function: g(λ, ν) = inf x L(x, λ, ν) 11

13 Lower Bound on Optimal Value Claim: g(λ, ν) p, λ 0, ν Proof: Consider feasible x: L( x, λ, ν) = f 0 ( x) + m λ i f i ( x) + p ν i h i ( x) f 0 ( x) i=1 i=1 since f i ( x) 0 and λ i 0 Hence, g(λ, ν) L( x, λ, ν) f 0 ( x) for all feasible x Therefore, g(λ, ν) p 12

14 Lagrange Dual Function and Conjugate Function Lagrange dual function g(λ, ν) Conjugate function: f (y) = sup x dom f (y T x f(x)) Consider linearly constrained optimization: minimize subject to f 0 (x) Ax b Cx = d g(λ, ν) = inf x ( f0 (x) + λ T (Ax b) + ν T (Cx d) ) = b T λ d T ν + inf x ( f0 (x) + (A T λ + C T ν) T x ) = b T λ d T ν f 0 ( A T λ C T ν) 13

15 Example We ll use the simplest version of entropy maximization as our example for the rest of this lecture on duality. Entropy maximization is an important basic problem in information theory: minimize subject to f 0 (x) = n i=1 x i log x i Ax b 1 T x = 1 Since the conjugate function of u log u is e y 1, by independence of the sum, we have f 0 (y) = n i=1 e y i 1 14

16 Therefore, dual function of entropy maximization is g(λ, ν) = b T λ ν e ν 1 n i=1 e at i λ where a i are columns of A 15

17 Lagrange Dual Problem Lower bound from Lagrange dual function depends on (λ, ν). What s the best lower bound that can be obtained from Lagrange dual function? maximize g(λ, ν) subject to λ 0 This is the Lagrange dual problem with dual variables (λ, ν) Always a convex optimization! (Dual objective function always a concave function since it s the infimum of a family of affine functions in (λ, ν)) Denote the optimal value of Lagrange dual problem by d 16

18 Weak Duality What s the relationship between d and p? Weak duality always hold (even if primal problem is not convex): d p Optimal duality gap: p d Efficient generation of lower bounds through (convex) dual problem 17

19 Strong Duality Strong duality (zero optimal duality gap): d = p If strong duality holds, solving dual is equivalent to solving primal. But strong duality does not always hold Convexity and constraint qualifications Strong duality A simple constraint qualification: Slater s condition (there exists strictly feasible primal variables f i (x) < 0 for non-affine f i ) Another reason why convex optimization is easy 18

20 Example Primal optimization problem (variables x): minimize subject to f 0 (x) = n i=1 x i log x i Ax b 1 T x = 1 Dual optimization problem (variables λ, ν): maximize b T λ ν e ν 1 n i=1 e at i λ subject to λ 0 Analytically maximize over the unconstrained ν Simplified dual 19

21 optimization problem (variables λ): maximize b T λ log n i=1 exp( at i λ) subject to λ 0 Strong duality holds 20

22 Saddle Point Interpretation Assume no equality constraints. We can express primal optimal value as p = inf x sup L(x, λ) λ 0 By definition of dual optimal value: d = sup inf λ 0 x L(x, λ) Weak duality (max min inequality): sup inf λ 0 x L(x, λ) inf x sup L(x, λ) λ 0 21

23 Strong duality (saddle point property): sup inf λ 0 x L(x, λ) = inf x sup L(x, λ) λ 0 22

24 Economics Interpretation Primal objective: cost of operation Primal constraints: can be violated Dual variables: price for violating the corresponding constraint (dollar per unit violation). For the same price, can sell unused violation for revenue Lagrangian: total cost Lagrange dual function: optimal cost as a function of violation prices Weak duality: optimal cost when constraints can be violated is less 23

25 than or equal to optimal cost when constraints cannot be violated, for any violation prices Duality gap: minimum possible arbitrage advantage Strong duality: can price the violations so that there is no arbitrage advantages 24

26 Complementary Slackness f 0 (x ) = g(λ, ν ) ( = inf x Assume strong duality holds: f 0 (x ) + f 0 (x ) f 0 (x) + m λ i f i (x) + i=1 m λ i f i (x ) + i=1 ) p νi h i (x) i=1 p νi h i (x ) i=1 So the two inequalities must hold with equality. This implies: λ i f i (x ) = 0, i = 1, 2,..., m 25

27 Complementary Slackness Property: λ i > 0 f i (x ) = 0 f i (x ) < 0 λ i = 0 26

28 KKT Optimality Conditions Since x minimizes L(x, λ, ν ) over x, we have f 0 (x ) + m λ i f i (x ) + p ν i h i (x ) = 0 i=1 i=1 Karush-Kuhn-Tucker optimality conditions: f i (x ) 0, h i (x ) = 0, λ i 0 λ i f i(x ) = 0 f 0 (x ) + m i=1 λ i f i(x ) + p i=1 ν i h i(x ) = 0 Any optimization (with differentiable objective and constraint functions) with strong duality, KKT condition is necessary condition 27

29 for primal-dual optimality Convex optimization (with differentiable objective and constraint functions) with Slater s condition, KKT condition is also sufficient condition for primal-dual optimality (useful for theoretical and numerical purposes) 28

30 Example Primal optimization problem (variables x): minimize f 0 (x) = n i=1 x i log x i subject to Ax b, 1 T x = 1 Dual optimization problem (variables λ, ν): maximize b T λ ν e ν 1 n i=1 e at i λ subject to λ 0 Having solved dual problem, recover optimal primal variables as minimizer of n L(x, λ, ν ) = x i log x i + λ T (Ax b) + ν (1 T x 1) i=1 29

31 i.e., x i = 1/ exp(at i λ + ν + 1) If x above is primal feasible, it s the optimal primal. Otherwise, the primal optimum is not attained 30

32 Water-filling n maximize i=1 log(α i + x i ) subject to x 0, 1 T x = 1 Variables: x (powers). Constants: α (noise) KKT conditions: x 0, 1 T x = 1, λ 0 λ i x i = 0, 1/(α i + x i ) λ i + ν = 0 Since λ are slack variables, reduce to x 0, 1 T x = 1 x i (ν 1/(αi + x i )) = 0, ν 1/(α i + x i ) 31

33 If ν < 1/α i, x i > 0. So x i = 1/ν α i. Otherwise, x i = 0 Thus, x i = [1/ν α i ] + where ν is such that i x i = 1 32

34 Global Sensitivity Analysis Perturbed optimization problem: minimize subject to f 0 (x) f i (x) u i, i = 1, 2,..., m h i (x) = v i i = 1, 2,..., p Optimal value p (u, v) as a function of parameters (u, v) Assume strong duality and that dual optimum is attained: p (0, 0) = g(λ, ν ) f 0 (x) + i λ i f i(x) + i ν i h i(x) f 0 (x) + λ T u + ν T v p (u, v) p (0, 0) λ T u ν T v If λ i is large, tightening ith constraint (u i < 0) will increase optimal value greatly If λ i is small, loosening ith constraint (u i > 0) will reduce optimal value only slightly 33

35 Local Sensitivity Analysis Assume that p (u, v) is differentiable at (0, 0): λ i = p (0, 0) u i, ν i = p (0, 0) v i Shadow price interpretation of Lagrange dual variables Small λ i means tightening or loosening ith constraint will not change optimal value by much 34

36 Summary Convexity mentality. Convex optimization is nice for several reasons: local optimum is global optimum, zero optimal duality gap (under technical conditions), KKT optimality conditions are necessary and sufficient Duality mentality. Can always bound primal through dual, sometimes solve primal through dual Primal-dual: where is the optimum, how sensitive it is to perturbations Reading assignment: Sections and of textbook. 35

### 5. Duality. Lagrangian 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Convex Optimization Boyd & Vandenberghe. 5. Duality 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

### Convex Optimization M2 Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization

### Lecture: Duality. Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong

### EE/AA 578, Univ of Washington, Fall Duality 7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Constrained Optimization and Lagrangian Duality CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

### Interior Point Algorithms for Constrained Convex Optimization Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems

### On the Method of Lagrange Multipliers On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should

### I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010 I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0

### Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem: CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

### Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form

### Lecture: Duality of LP, SOCP and SDP 1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:

### Lecture 18: Optimization Programming Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming

### Lagrangian Duality and Convex Optimization Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?

### CS-E4830 Kernel Methods in Machine Learning CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This

### The Lagrangian L : R d R m R r R is an (easier to optimize) lower bound on the original problem: HT05: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford Convex Optimization and slides based on Arthur Gretton s Advanced Topics in Machine Learning course

### Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,

### ICS-E4030 Kernel Methods in Machine Learning ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This

### Lagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST) Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual

### Lagrange duality. The Lagrangian. We consider an optimization program of the form Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization

### 14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness. CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity

### Optimization for Machine Learning Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html

### CSCI : Optimization and Control of Networks. Review on Convex Optimization CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one

### Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725 Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =

### A Brief Review on Convex Optimization A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review

### Linear and Combinatorial Optimization Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality

### 4TE3/6TE3. Algorithms for. Continuous Optimization 4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality

### Convex Optimization and Modeling Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

### Lagrangian Duality Theory Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual

### subject to (x 2)(x 4) u, Exercises Basic definitions 5.1 A simple example. Consider the optimization problem with variable x R. minimize x 2 + 1 subject to (x 2)(x 4) 0, (a) Analysis of primal problem. Give the feasible set, the

### EE364a Review Session 5 EE364a Review Session 5 EE364a Review announcements: homeworks 1 and 2 graded homework 4 solutions (check solution to additional problem 1) scpd phone-in office hours: tuesdays 6-7pm (650-723-1156) 1 Complementary

### Homework Set #6 - Solutions EE 15 - Applications of Convex Optimization in Signal Processing and Communications Dr Andre Tkacenko JPL Third Term 11-1 Homework Set #6 - Solutions 1 a The feasible set is the interval [ 4] The unique

### Support Vector Machines: Maximum Margin Classifiers Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and Fu-Jie Huang 1 Outline What is behind

### Nonlinear Programming and the Kuhn-Tucker Conditions Nonlinear Programming and the Kuhn-Tucker Conditions The Kuhn-Tucker (KT) conditions are first-order conditions for constrained optimization problems, a generalization of the first-order conditions we

### Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

### Lecture 6: Conic Optimization September 8 IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

### Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following

### Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014 Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,

### Lecture 7: Convex Optimizations Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1

### Generalization to inequality constrained problem. Maximize Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum

### Primal/Dual Decomposition Methods Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients

### Numerical Optimization Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,

### Part IB Optimisation Part IB Optimisation Theorems Based on lectures by F. A. Fischer Notes taken by Dexter Chua Easter 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after

### Convex Optimization Lecture 6: KKT Conditions, and applications Convex Optimization Lecture 6: KKT Conditions, and applications Dr. Michel Baes, IFOR / ETH Zürich Quick recall of last week s lecture Various aspects of convexity: The set of minimizers is convex. Convex

### Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 4. Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 4 Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 2 4.1. Subgradients definition subgradient calculus duality and optimality conditions Shiqian

### Tutorial on Convex Optimization for Engineers Part II Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

### Optimality, Duality, Complementarity for Constrained Optimization Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

### Machine Learning. Support Vector Machines. Manfred Huber Machine Learning Support Vector Machines Manfred Huber 2015 1 Support Vector Machines Both logistic regression and linear discriminant analysis learn a linear discriminant function to separate the data

### ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained

### Chap 2. Optimality conditions Chap 2. Optimality conditions Version: 29-09-2012 2.1 Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1

### Lecture 8. Strong Duality Results. September 22, 2008 Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation

### Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu

### 4. Convex optimization problems (part 1: general) EE/AA 578, Univ of Washington, Fall 2016 4. Convex optimization problems (part 1: general) optimization problem in standard form convex optimization problems quasiconvex optimization 4 1 Optimization problem

### 12. Interior-point methods 12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

### Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian

### Optimality Conditions for Constrained Optimization 72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)

### The Karush-Kuhn-Tucker (KKT) conditions The Karush-Kuhn-Tucker (KKT) conditions In this section, we will give a set of sufficient (and at most times necessary) conditions for a x to be the solution of a given convex optimization problem. These

### Convex Optimization in Communications and Signal Processing Convex Optimization in Communications and Signal Processing Prof. Dr.-Ing. Wolfgang Gerstacker 1 University of Erlangen-Nürnberg Institute for Digital Communications National Technical University of Ukraine,

### Lagrange Relaxation and Duality Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler

### 4. Convex optimization problems Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization

### 4. Algebra and Duality 4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone

### Rate Control in Communication Networks From Models to Algorithms Department of Computer Science & Engineering The Chinese University of Hong Kong February 29, 2008 Outline Preliminaries 1 Preliminaries Convex Optimization TCP Congestion Control

### Tutorial on Convex Optimization: Part II Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation

### Linear Programming. Larry Blume Cornell University, IHS Vienna and SFI. Summer 2016 Linear Programming Larry Blume Cornell University, IHS Vienna and SFI Summer 2016 These notes derive basic results in finite-dimensional linear programming using tools of convex analysis. Most sources

### Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module - 5 Lecture - 22 SVM: The Dual Formulation Good morning.

### Support vector machines Support vector machines Guillaume Obozinski Ecole des Ponts - ParisTech SOCN course 2014 SVM, kernel methods and multiclass 1/23 Outline 1 Constrained optimization, Lagrangian duality and KKT 2 Support

### Support Vector Machines for Regression COMP-566 Rohan Shah (1) Support Vector Machines for Regression Provided with n training data points {(x 1, y 1 ), (x 2, y 2 ),, (x n, y n )} R s R we seek a function f for a fixed ɛ > 0 such that: f(x

### Linear and non-linear programming Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)

### Constrained optimization Constrained optimization DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Compressed sensing Convex constrained

### Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006 Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

### Lagrangian Duality. Evelien van der Hurk. DTU Management Engineering Lagrangian Duality Evelien van der Hurk DTU Management Engineering Topics Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality 2 DTU Management Engineering 42111: Static and Dynamic Optimization

### A Tutorial on Convex Optimization II: Duality and Interior Point Methods A Tutorial on Convex Optimization II: Duality and Interior Point Methods Haitham Hindi Palo Alto Research Center (PARC), Palo Alto, California 94304 email: hhindi@parc.com Abstract In recent years, convex

### EE 227A: Convex Optimization and Applications October 14, 2008 EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider

### CS711008Z Algorithm Design and Analysis .. CS711008Z Algorithm Design and Analysis Lecture 9. Algorithm design technique: Linear programming and duality Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China

### ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications Professor M. Chiang Electrical Engineering Department, Princeton University February

### Convex Optimization. Newton s method. ENSAE: Optimisation 1/44 Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)

### Lecture 8 Plus properties, merit functions and gap functions. September 28, 2008 Lecture 8 Plus properties, merit functions and gap functions September 28, 2008 Outline Plus-properties and F-uniqueness Equation reformulations of VI/CPs Merit functions Gap merit functions FP-I book:

### Convex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST) Convex Functions Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Definition convex function Examples

### Lecture Notes on Support Vector Machine Lecture Notes on Support Vector Machine Feng Li fli@sdu.edu.cn Shandong University, China 1 Hyperplane and Margin In a n-dimensional space, a hyper plane is defined by ω T x + b = 0 (1) where ω R n is Subgradients subgradients and quasigradients subgradient calculus optimality conditions via subgradients directional derivatives Prof. S. Boyd, EE392o, Stanford University Basic inequality recall basic

### In view of (31), the second of these is equal to the identity I on E m, while this, in view of (30), implies that the first can be written 11.8 Inequality Constraints 341 Because by assumption x is a regular point and L x is positive definite on M, it follows that this matrix is nonsingular (see Exercise 11). Thus, by the Implicit Function

### Lectures 9 and 10: Constrained optimization problems and their optimality conditions Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained

### Structural and Multidisciplinary Optimization. P. Duysinx and P. Tossings Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be

### Lecture 3 January 28 EECS 28B / STAT 24B: Advanced Topics in Statistical LearningSpring 2009 Lecture 3 January 28 Lecturer: Pradeep Ravikumar Scribe: Timothy J. Wheeler Note: These lecture notes are still rough, and have only

### Primal-dual Subgradient Method for Convex Problems with Functional Constraints Primal-dual Subgradient Method for Convex Problems with Functional Constraints Yurii Nesterov, CORE/INMA (UCL) Workshop on embedded optimization EMBOPT2014 September 9, 2014 (Lucca) Yu. Nesterov Primal-dual

### Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex

### Subgradients. subgradients. strong and weak subgradient calculus. optimality conditions via subgradients. directional derivatives Subgradients subgradients strong and weak subgradient calculus optimality conditions via subgradients directional derivatives Prof. S. Boyd, EE364b, Stanford University Basic inequality recall basic inequality

### Convex Optimization and Support Vector Machine Convex Optimization and Support Vector Machine Problem 0. Consider a two-class classification problem. The training data is L n = {(x 1, t 1 ),..., (x n, t n )}, where each t i { 1, 1} and x i R p. We

### CS711008Z Algorithm Design and Analysis CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief

### Convex Optimization and SVM Convex Optimization and SVM Problem 0. Cf lecture notes pages 12 to 18. Problem 1. (i) A slab is an intersection of two half spaces, hence convex. (ii) A wedge is an intersection of two half spaces, hence

### minimize x subject to (x 2)(x 4) u, Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for

### Convex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com

### CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS A Dissertation Submitted For The Award of the Degree of Master of Philosophy in Mathematics Neelam Patel School of Mathematics

### Nonlinear Optimization Nonlinear Optimization Etienne de Klerk (UvT)/Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos Course WI3031 (Week 4) February-March, A.D. 2005 Optimization Group 1 Outline Enhanced Fritz John Optimality Conditions and Sensitivity Analysis Dimitri P. Bertsekas Laboratory for Information and Decision Systems Massachusetts Institute of Technology March 2016 1 / 27 Constrained Journal of the Operations Research Society of Japan Vol. 60, No. 2, April 2017, pp. 44 49 c The Operations Research Society of Japan DYNAMIC DUALIZATION IN A GENERAL SETTING Hidefumi Kawasaki Kyushu University