# 14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

Size: px
Start display at page:

Download "14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness."

## Transcription

1 CS/ECE/ISyE 524 Introduction to Optimization Spring Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity analysis Laurent Lessard (

2 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r 14-2

3 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ D is the domain of all functions involved. ˆ Suppose the optimal value is p. 14-2

4 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ D is the domain of all functions involved. ˆ Suppose the optimal value is p. ˆ Upper bounds: if x D satisfies f i (x) 0 and h j (x) = 0 for all i and j, then: p f 0 (x). 14-2

5 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ D is the domain of all functions involved. ˆ Suppose the optimal value is p. ˆ Upper bounds: if x D satisfies f i (x) 0 and h j (x) = 0 for all i and j, then: p f 0 (x). ˆ Any feasible x yields an upper bound for p. 14-2

6 Lower bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ As with LPs, use the constraints to find lower bounds 14-3

7 Lower bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ As with LPs, use the constraints to find lower bounds ˆ For any λ i 0 and ν j R, if x D is feasible, then f 0 (x) f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 14-3

8 Lower bounds f 0 (x) f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 } {{ } Lagrangian L(x, λ, ν) This is a lower bound on f 0, but we want a lower bound on p. Minimize right side over x D and left side over feasible x. 14-4

9 Lower bounds f 0 (x) f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 } {{ } Lagrangian L(x, λ, ν) This is a lower bound on f 0, but we want a lower bound on p. Minimize right side over x D and left side over feasible x. p { inf x D } L(x, λ, ν) = g(λ, ν) This inequality holds whenever λ

10 Lower bounds L(x, λ, ν) := f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 Whenever λ 0, we have: g(λ, ν) := { inf x D } L(x, λ, ν) p Useful fact: g(λ, ν) is a concave function. This is true even if the original optimization problem is not convex! (because g is a pointwise minimum of affine functions) 14-5

11 General duality Primal problem (P) x D f 0 (x) subject to: f i (x) 0 i h j (x) = 0 j Dual problem (D) maximize λ,ν g(λ, ν) subject to: λ

12 General duality Primal problem (P) x D f 0 (x) subject to: f i (x) 0 i h j (x) = 0 j Dual problem (D) maximize λ,ν g(λ, ν) subject to: λ 0 If x and λ are feasible points of (P) and (D) respectively: g(λ, ν) d p f 0 (x) 14-6

13 General duality Primal problem (P) x D f 0 (x) subject to: f i (x) 0 i h j (x) = 0 j Dual problem (D) maximize λ,ν g(λ, ν) subject to: λ 0 If x and λ are feasible points of (P) and (D) respectively: g(λ, ν) d p f 0 (x) This is called the Lagrange dual. Bad news: strong duality (p = d ) does not always hold! 14-6

14 Example (Srikant) x x subject to: (x 2)(x 4) x -5 x (x - 2) (x - 4) 14-7

15 Example (Srikant) x x subject to: (x 2)(x 4) x -5 x (x - 2) (x - 4) ˆ optimum occurs at x = 2, has value p =

16 Example (Srikant) Lagrangian: L(x, λ) = x λ(x 2)(x 4) x -5 ˆ Plot for different values of λ 0 ˆ Lower bounds for x on the feasible set 2 x

17 Example (Srikant) Lagrangian: L(x, λ) = x λ(x 2)(x 4) ˆ Minimize the Lagrangian: g(λ) = inf x = inf x L(x, λ) (λ + 1)x 2 6λx + (8λ + 1) If λ 1, it is unbounded. If λ > 1, the minimum occurs when 2(λ + 1)x 6λ = 0, so ˆx = 3λ g(λ) = { 9λ 2 /(1 + λ) λ λ+1. λ > 1 λ

18 Example (Srikant) maximize λ 9λ 2 /(1 + λ) λ subject to: λ 0 g(λ) λ

19 Example (Srikant) maximize λ 9λ 2 /(1 + λ) λ subject to: λ 0 g(λ) λ ˆ optimum occurs at λ = 2, has value d = 5 ˆ same optimal value as primal problem! (strong duality) 14-10

20 Constraint qualifications ˆ weak duality (d p ) always holds. Even when the optimization problem is not convex. ˆ strong duality (d = p ) often holds for convex problems (but not always)

21 Constraint qualifications ˆ weak duality (d p ) always holds. Even when the optimization problem is not convex. ˆ strong duality (d = p ) often holds for convex problems (but not always). A constraint qualification is a condition that guarantees strong duality

22 Constraint qualifications ˆ weak duality (d p ) always holds. Even when the optimization problem is not convex. ˆ strong duality (d = p ) often holds for convex problems (but not always). A constraint qualification is a condition that guarantees strong duality. An example we ve already seen: ˆ If the optimization problem is an LP, strong duality holds 14-11

23 Slater s constraint qualification x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r Slater s constraint qualification: If the optimization problem is convex and strictly feasible, then strong duality holds. ˆ convexity requires: D and f i are convex and h j are affine. ˆ strict feasibility means there exists some x in the interior of D such that f i ( x) < 0 for i = 1,..., m

24 Slater s constraint qualification If the optimization problem is convex and strictly feasible, then strong duality holds. ˆ Good news: Slater s constraint qualification is rather weak. i.e. it is usually satisfied by convex problems. ˆ Can be relaxed so that strict feasibility is not required for the linear constraints

25 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ The function x 2 /y is convex for y > 0 (see plot) ˆ The objective e x is convex ˆ Feasible set: {(0, y) y > 0} ˆ Solution is trivial (p = 1) 14-14

26 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y 14-15

27 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y ˆ Dual function: g(λ) = inf x,y>0 (e x + λx 2 /y) =

28 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y ˆ Dual function: g(λ) = inf x,y>0 (e x + λx 2 /y) = 0. ˆ The dual problem is: maximize λ 0 0 So we have d = 0 < 1 = p

29 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y ˆ Dual function: g(λ) = inf x,y>0 (e x + λx 2 /y) = 0. ˆ The dual problem is: maximize λ 0 0 So we have d = 0 < 1 = p. ˆ Slater s constraint qualification is not satisfied! 14-15

30 About Slater s constraint qualification Slater s condition is only sufficient. (Slater) = (strong duality) 14-16

31 About Slater s constraint qualification Slater s condition is only sufficient. (Slater) = (strong duality) ˆ There exist problems where Slater s condition fails, yet strong duality holds. ˆ There exist nonconvex problems with strong duality

32 Complementary slackness Assume strong duality holds. If x is primal optimal and (λ, ν ) is dual optimal, then we have: g(λ, ν ) = d = p = f 0 (x ) 14-17

33 Complementary slackness Assume strong duality holds. If x is primal optimal and (λ, ν ) is dual optimal, then we have: f 0 (x ) = g(λ, ν ) = inf x D g(λ, ν ) = d = p = f 0 (x ) ( f 0 (x ) + f 0 (x ) f 0 (x) + m λ i f i (x) + i=1 m λ i f i (x ) + i=1 ) r νj h j (x) j=1 r νj h j (x ) The last inequality holds because x is primal feasible. We conclude that the inequalities must all be equalities. j=

34 Complementary slackness ˆ We concluded that: f 0 (x ) = f 0 (x ) + m λ i f i (x ) + i=1 r νj h j (x ) j=1 But f i (x ) 0 and h j (x ) = 0. Therefore: λ i f i (x ) = 0 for i = 1,..., m 14-18

35 Complementary slackness ˆ We concluded that: f 0 (x ) = f 0 (x ) + m λ i f i (x ) + i=1 r νj h j (x ) j=1 But f i (x ) 0 and h j (x ) = 0. Therefore: λ i f i (x ) = 0 for i = 1,..., m ˆ This property is called complementary slackness. We ve seen it before for linear programs

36 Complementary slackness ˆ We concluded that: f 0 (x ) = f 0 (x ) + m λ i f i (x ) + i=1 r νj h j (x ) j=1 But f i (x ) 0 and h j (x ) = 0. Therefore: λ i f i (x ) = 0 for i = 1,..., m ˆ This property is called complementary slackness. We ve seen it before for linear programs. λ i > 0 = f i (x ) = 0 and f i (x ) < 0 = λ i =

37 Dual of an LP x 0 subject to: c T x Ax b 14-19

38 Dual of an LP x 0 subject to: c T x Ax b ˆ Lagrangian: L(x, λ) = c T x + λ T (b Ax) 14-19

39 Dual of an LP x 0 subject to: c T x Ax b ˆ Lagrangian: L(x, λ) = c T x + λ T (b Ax) ˆ Dual function: g(λ) = min x 0 (c AT λ) T x + λ T b 14-19

40 Dual of an LP x 0 subject to: c T x Ax b ˆ Lagrangian: L(x, λ) = c T x + λ T (b Ax) ˆ Dual function: g(λ) = min x 0 (c AT λ) T x + λ T b g(λ) = { λ T b if A T λ c otherwise 14-19

41 Dual of an LP x 0 subject to: c T x Ax b ˆ Dual is: maximize λ 0 subject to: λ T b A T λ c ˆ This is the same result that we found when we were studying duality for linear programs

42 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x

43 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Lagrangian: L(x, λ, µ) = c T x + λ T (b Ax) µ T x 14-21

44 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Lagrangian: L(x, λ, µ) = c T x + λ T (b Ax) µ T x ˆ Dual function: g(λ, µ) = min (c A T λ µ) T x + λ T b x 14-21

45 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Lagrangian: L(x, λ, µ) = c T x + λ T (b Ax) µ T x ˆ Dual function: g(λ, µ) = min (c A T λ µ) T x + λ T b x { λ T b if A T λ + µ = c g(λ) = otherwise 14-21

46 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Dual is: maximize λ 0, µ 0 subject to: λ T b A T λ + µ = c ˆ Solution is the same, µ acts as the slack variable

47 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b 14-23

48 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Lagrangian: L(x, λ) = 1 2 x T Qx + λ T (b Ax) 14-23

49 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Lagrangian: L(x, λ) = 1x T Qx + λ T (b Ax) 2 ( ˆ Dual function: g(λ) = min 1 x T Qx + λ T (b Ax) ) x 2 Minimum occurs at: ˆx = Q 1 A T λ 14-23

50 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Lagrangian: L(x, λ) = 1x T Qx + λ T (b Ax) 2 ( ˆ Dual function: g(λ) = min 1 x T Qx + λ T (b Ax) ) x 2 Minimum occurs at: ˆx = Q 1 A T λ g(λ) = 1 2 λt AQ 1 A T λ + λ T b 14-23

51 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Dual is also a QP: maximize λ 1 2 λt AQ 1 A T λ + λ T b subject to: λ 0 ˆ It s still easy to solve (maximizing a concave function) 14-24

52 Sensitivity analysis min x D f 0 (x) s.t. f i (x) u i i h j (x) = v j j max λ,ν g(λ, ν) λ T u ν T v s.t. λ 0 ˆ As with LPs, dual variables quantify the sensitivity of the optimal cost to changes in each of the constraints

53 Sensitivity analysis min x D f 0 (x) s.t. f i (x) u i i h j (x) = v j j max λ,ν g(λ, ν) λ T u ν T v s.t. λ 0 ˆ As with LPs, dual variables quantify the sensitivity of the optimal cost to changes in each of the constraints. ˆ A change in u i causes a bigger change in p if λ i ˆ A change in v j causes a bigger change in p if ν j is larger. is larger

54 Sensitivity analysis min x D f 0 (x) s.t. f i (x) u i i h j (x) = v j j max λ,ν g(λ, ν) λ T u ν T v s.t. λ 0 ˆ As with LPs, dual variables quantify the sensitivity of the optimal cost to changes in each of the constraints. ˆ A change in u i causes a bigger change in p if λ i ˆ A change in v j causes a bigger change in p if ν j is larger. is larger. ˆ If p (u, v) is differentiable, then: λ i = p (0, 0) u i and ν j = p (0, 0) v j 14-25

55 Next class... ˆ Review! 14-26

### Lagrangian Duality and Convex Optimization

Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?

### 5. Duality. Lagrangian

5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Convex Optimization Boyd & Vandenberghe. 5. Duality

5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Convex Optimization M2

Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization

### Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities

Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form

### Lecture 18: Optimization Programming

Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming

### Lagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)

Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual

### Lecture: Duality.

Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong

### EE364a Review Session 5

EE364a Review Session 5 EE364a Review announcements: homeworks 1 and 2 graded homework 4 solutions (check solution to additional problem 1) scpd phone-in office hours: tuesdays 6-7pm (650-723-1156) 1 Complementary

### Lagrange duality. The Lagrangian. We consider an optimization program of the form

Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization

### The Lagrangian L : R d R m R r R is an (easier to optimize) lower bound on the original problem:

HT05: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford Convex Optimization and slides based on Arthur Gretton s Advanced Topics in Machine Learning course

### Lecture: Duality of LP, SOCP and SDP

1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:

### Duality. Geoff Gordon & Ryan Tibshirani Optimization /

Duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider

### EE/AA 578, Univ of Washington, Fall Duality

7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Constrained Optimization and Lagrangian Duality

CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

### Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

### Convex Optimization & Lagrange Duality

Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT

### I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010

I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0

### Convex Optimization and SVM

Convex Optimization and SVM Problem 0. Cf lecture notes pages 12 to 18. Problem 1. (i) A slab is an intersection of two half spaces, hence convex. (ii) A wedge is an intersection of two half spaces, hence

### Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014

Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,

### Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs

Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following

### Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers

Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex

### Applications of Linear Programming

Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal

### Lagrangian Duality Theory

Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual

### On the Method of Lagrange Multipliers

On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should

### CSCI : Optimization and Control of Networks. Review on Convex Optimization

CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one

### A Brief Review on Convex Optimization

A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review

### CS-E4830 Kernel Methods in Machine Learning

CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This

### Linear and Combinatorial Optimization

Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality

### Homework Set #6 - Solutions

EE 15 - Applications of Convex Optimization in Signal Processing and Communications Dr Andre Tkacenko JPL Third Term 11-1 Homework Set #6 - Solutions 1 a The feasible set is the interval [ 4] The unique

### Primal/Dual Decomposition Methods

Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients

### Optimization for Machine Learning

Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html

### subject to (x 2)(x 4) u,

Exercises Basic definitions 5.1 A simple example. Consider the optimization problem with variable x R. minimize x 2 + 1 subject to (x 2)(x 4) 0, (a) Analysis of primal problem. Give the feasible set, the

### ICS-E4030 Kernel Methods in Machine Learning

ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This

### Tutorial on Convex Optimization for Engineers Part II

Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

### Tutorial on Convex Optimization: Part II

Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation

### Lecture Note 5: Semidefinite Programming for Stability Analysis

ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State

### Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research

Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,

### Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:

CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

### Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725

Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =

### Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

### 4TE3/6TE3. Algorithms for. Continuous Optimization

4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality

### Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

### Lecture 7: Convex Optimizations

Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1

### Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark

Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian

### Convex Optimization M2

Convex Optimization M2 Lecture 8 A. d Aspremont. Convex Optimization M2. 1/57 Applications A. d Aspremont. Convex Optimization M2. 2/57 Outline Geometrical problems Approximation problems Combinatorial

### E5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming

E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program

### Sparse Optimization Lecture: Dual Certificate in l 1 Minimization

Sparse Optimization Lecture: Dual Certificate in l 1 Minimization Instructor: Wotao Yin July 2013 Note scriber: Zheng Sun Those who complete this lecture will know what is a dual certificate for l 1 minimization

### Maximum likelihood estimation

Maximum likelihood estimation Guillaume Obozinski Ecole des Ponts - ParisTech Master MVA Maximum likelihood estimation 1/26 Outline 1 Statistical concepts 2 A short review of convex analysis and optimization

### Lecture 8. Strong Duality Results. September 22, 2008

Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation

### HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.

HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

### Interior Point Algorithms for Constrained Convex Optimization

Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems

### Support vector machines

Support vector machines Guillaume Obozinski Ecole des Ponts - ParisTech SOCN course 2014 SVM, kernel methods and multiclass 1/23 Outline 1 Constrained optimization, Lagrangian duality and KKT 2 Support

### EE 227A: Convex Optimization and Applications October 14, 2008

EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider

### Duality in Linear Programs. Lecturer: Ryan Tibshirani Convex Optimization /36-725

Duality in Linear Programs Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: proximal gradient descent Consider the problem x g(x) + h(x) with g, h convex, g differentiable, and

### Lectures 9 and 10: Constrained optimization problems and their optimality conditions

Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained

### 12. Interior-point methods

12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

### Nonlinear Programming and the Kuhn-Tucker Conditions

Nonlinear Programming and the Kuhn-Tucker Conditions The Kuhn-Tucker (KT) conditions are first-order conditions for constrained optimization problems, a generalization of the first-order conditions we

### CS711008Z Algorithm Design and Analysis

CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief

### Finite Dimensional Optimization Part III: Convex Optimization 1

John Nachbar Washington University March 21, 2017 Finite Dimensional Optimization Part III: Convex Optimization 1 1 Saddle points and KKT. These notes cover another important approach to optimization,

### 10 Numerical methods for constrained problems

10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside

### Convex Optimization and Modeling

Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

### Support Vector Machines: Maximum Margin Classifiers

Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and Fu-Jie Huang 1 Outline What is behind

### Nonlinear Programming

Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week

### Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016

Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 1 Entropy Since this course is about entropy maximization,

### Optimality Conditions for Constrained Optimization

72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)

### Constrained optimization

Constrained optimization DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Compressed sensing Convex constrained

### Introduction to Nonlinear Stochastic Programming

School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS

### Convexification of Mixed-Integer Quadratically Constrained Quadratic Programs

Convexification of Mixed-Integer Quadratically Constrained Quadratic Programs Laura Galli 1 Adam N. Letchford 2 Lancaster, April 2011 1 DEIS, University of Bologna, Italy 2 Department of Management Science,

### Duality Uses and Correspondences. Ryan Tibshirani Convex Optimization

Duality Uses and Correspondences Ryan Tibshirani Conve Optimization 10-725 Recall that for the problem Last time: KKT conditions subject to f() h i () 0, i = 1,... m l j () = 0, j = 1,... r the KKT conditions

### The fundamental theorem of linear programming

The fundamental theorem of linear programming Michael Tehranchi June 8, 2017 This note supplements the lecture notes of Optimisation The statement of the fundamental theorem of linear programming and the

### Convex Optimization and Support Vector Machine

Convex Optimization and Support Vector Machine Problem 0. Consider a two-class classification problem. The training data is L n = {(x 1, t 1 ),..., (x n, t n )}, where each t i { 1, 1} and x i R p. We

### Lecture 6: Conic Optimization September 8

IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

### UC Berkeley Department of Electrical Engineering and Computer Science. EECS 227A Nonlinear and Convex Optimization. Solutions 5 Fall 2009

UC Berkeley Department of Electrical Engineering and Computer Science EECS 227A Nonlinear and Convex Optimization Solutions 5 Fall 2009 Reading: Boyd and Vandenberghe, Chapter 5 Solution 5.1 Note that

### 4. Algebra and Duality

4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone

### LECTURE 10 LECTURE OUTLINE

LECTURE 10 LECTURE OUTLINE Min Common/Max Crossing Th. III Nonlinear Farkas Lemma/Linear Constraints Linear Programming Duality Convex Programming Duality Optimality Conditions Reading: Sections 4.5, 5.1,5.2,

### Lecture 14: Optimality Conditions for Conic Problems

EE 227A: Conve Optimization and Applications March 6, 2012 Lecture 14: Optimality Conditions for Conic Problems Lecturer: Laurent El Ghaoui Reading assignment: 5.5 of BV. 14.1 Optimality for Conic Problems

### LINEAR PROGRAMMING. Relation to the Text (cont.) Relation to Material in Text. Relation to the Text. Relation to the Text (cont.

LINEAR PROGRAMMING Relation to Material in Text After a brief introduction to linear programming on p. 3, Cornuejols and Tϋtϋncϋ give a theoretical discussion including duality, and the simplex solution

### Lecture Notes on Support Vector Machine

Lecture Notes on Support Vector Machine Feng Li fli@sdu.edu.cn Shandong University, China 1 Hyperplane and Margin In a n-dimensional space, a hyper plane is defined by ω T x + b = 0 (1) where ω R n is

### Understanding the Simplex algorithm. Standard Optimization Problems.

Understanding the Simplex algorithm. Ma 162 Spring 2011 Ma 162 Spring 2011 February 28, 2011 Standard Optimization Problems. A standard maximization problem can be conveniently described in matrix form

### Lagrangian Duality. Evelien van der Hurk. DTU Management Engineering

Lagrangian Duality Evelien van der Hurk DTU Management Engineering Topics Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality 2 DTU Management Engineering 42111: Static and Dynamic Optimization

### Lagrange Relaxation and Duality

Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler

### A Tutorial on Convex Optimization II: Duality and Interior Point Methods

A Tutorial on Convex Optimization II: Duality and Interior Point Methods Haitham Hindi Palo Alto Research Center (PARC), Palo Alto, California 94304 email: hhindi@parc.com Abstract In recent years, convex

### Math 273a: Optimization Subgradients of convex functions

Math 273a: Optimization Subgradients of convex functions Made by: Damek Davis Edited by Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com 1 / 42 Subgradients Assumptions

### Interior Point Methods for LP

11.1 Interior Point Methods for LP Katta G. Murty, IOE 510, LP, U. Of Michigan, Ann Arbor, Winter 1997. Simplex Method - A Boundary Method: Starting at an extreme point of the feasible set, the simplex

### 7. Statistical estimation

7. Statistical estimation Convex Optimization Boyd & Vandenberghe maximum likelihood estimation optimal detector design experiment design 7 1 Parametric distribution estimation distribution estimation

### Introduction to Machine Learning Spring 2018 Note Duality. 1.1 Primal and Dual Problem

CS 189 Introduction to Machine Learning Spring 2018 Note 22 1 Duality As we have seen in our discussion of kernels, ridge regression can be viewed in two ways: (1) an optimization problem over the weights

### Example: feasibility. Interpretation as formal proof. Example: linear inequalities and Farkas lemma

4-1 Algebra and Duality P. Parrilo and S. Lall 2006.06.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone of valid

### Lecture #21. c T x Ax b. maximize subject to

COMPSCI 330: Design and Analysis of Algorithms 11/11/2014 Lecture #21 Lecturer: Debmalya Panigrahi Scribe: Samuel Haney 1 Overview In this lecture, we discuss linear programming. We first show that the

### CS711008Z Algorithm Design and Analysis

.. CS711008Z Algorithm Design and Analysis Lecture 9. Algorithm design technique: Linear programming and duality Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China

### Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation

Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu

### Optimization. Yuh-Jye Lee. March 28, Data Science and Machine Intelligence Lab National Chiao Tung University 1 / 40

Optimization Yuh-Jye Lee Data Science and Machine Intelligence Lab National Chiao Tung University March 28, 2017 1 / 40 The Key Idea of Newton s Method Let f : R n R be a twice differentiable function

### ECE Optimization for wireless networks Final. minimize f o (x) s.t. Ax = b,

ECE 788 - Optimization for wireless networks Final Please provide clear and complete answers. PART I: Questions - Q.. Discuss an iterative algorithm that converges to the solution of the problem minimize

### Part IB Optimisation

Part IB Optimisation Theorems Based on lectures by F. A. Fischer Notes taken by Dexter Chua Easter 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after

### LP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra

LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality

### Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

### Conic Linear Optimization and its Dual. yyye

Conic Linear Optimization and Appl. MS&E314 Lecture Note #04 1 Conic Linear Optimization and its Dual Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.

### Duality of LPs and Applications

Lecture 6 Duality of LPs and Applications Last lecture we introduced duality of linear programs. We saw how to form duals, and proved both the weak and strong duality theorems. In this lecture we will