14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness."

Transcription

1 CS/ECE/ISyE 524 Introduction to Optimization Spring Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity analysis Laurent Lessard (

2 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r 14-2

3 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ D is the domain of all functions involved. ˆ Suppose the optimal value is p. 14-2

4 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ D is the domain of all functions involved. ˆ Suppose the optimal value is p. ˆ Upper bounds: if x D satisfies f i (x) 0 and h j (x) = 0 for all i and j, then: p f 0 (x). 14-2

5 Upper bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ D is the domain of all functions involved. ˆ Suppose the optimal value is p. ˆ Upper bounds: if x D satisfies f i (x) 0 and h j (x) = 0 for all i and j, then: p f 0 (x). ˆ Any feasible x yields an upper bound for p. 14-2

6 Lower bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ As with LPs, use the constraints to find lower bounds 14-3

7 Lower bounds Optimization problem (not necessarily convex!): x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r ˆ As with LPs, use the constraints to find lower bounds ˆ For any λ i 0 and ν j R, if x D is feasible, then f 0 (x) f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 14-3

8 Lower bounds f 0 (x) f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 } {{ } Lagrangian L(x, λ, ν) This is a lower bound on f 0, but we want a lower bound on p. Minimize right side over x D and left side over feasible x. 14-4

9 Lower bounds f 0 (x) f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 } {{ } Lagrangian L(x, λ, ν) This is a lower bound on f 0, but we want a lower bound on p. Minimize right side over x D and left side over feasible x. p { inf x D } L(x, λ, ν) = g(λ, ν) This inequality holds whenever λ

10 Lower bounds L(x, λ, ν) := f 0 (x) + m λ i f i (x) + i=1 r ν j h j (x) j=1 Whenever λ 0, we have: g(λ, ν) := { inf x D } L(x, λ, ν) p Useful fact: g(λ, ν) is a concave function. This is true even if the original optimization problem is not convex! (because g is a pointwise minimum of affine functions) 14-5

11 General duality Primal problem (P) x D f 0 (x) subject to: f i (x) 0 i h j (x) = 0 j Dual problem (D) maximize λ,ν g(λ, ν) subject to: λ

12 General duality Primal problem (P) x D f 0 (x) subject to: f i (x) 0 i h j (x) = 0 j Dual problem (D) maximize λ,ν g(λ, ν) subject to: λ 0 If x and λ are feasible points of (P) and (D) respectively: g(λ, ν) d p f 0 (x) 14-6

13 General duality Primal problem (P) x D f 0 (x) subject to: f i (x) 0 i h j (x) = 0 j Dual problem (D) maximize λ,ν g(λ, ν) subject to: λ 0 If x and λ are feasible points of (P) and (D) respectively: g(λ, ν) d p f 0 (x) This is called the Lagrange dual. Bad news: strong duality (p = d ) does not always hold! 14-6

14 Example (Srikant) x x subject to: (x 2)(x 4) x -5 x (x - 2) (x - 4) 14-7

15 Example (Srikant) x x subject to: (x 2)(x 4) x -5 x (x - 2) (x - 4) ˆ optimum occurs at x = 2, has value p =

16 Example (Srikant) Lagrangian: L(x, λ) = x λ(x 2)(x 4) x -5 ˆ Plot for different values of λ 0 ˆ Lower bounds for x on the feasible set 2 x

17 Example (Srikant) Lagrangian: L(x, λ) = x λ(x 2)(x 4) ˆ Minimize the Lagrangian: g(λ) = inf x = inf x L(x, λ) (λ + 1)x 2 6λx + (8λ + 1) If λ 1, it is unbounded. If λ > 1, the minimum occurs when 2(λ + 1)x 6λ = 0, so ˆx = 3λ g(λ) = { 9λ 2 /(1 + λ) λ λ+1. λ > 1 λ

18 Example (Srikant) maximize λ 9λ 2 /(1 + λ) λ subject to: λ 0 g(λ) λ

19 Example (Srikant) maximize λ 9λ 2 /(1 + λ) λ subject to: λ 0 g(λ) λ ˆ optimum occurs at λ = 2, has value d = 5 ˆ same optimal value as primal problem! (strong duality) 14-10

20 Constraint qualifications ˆ weak duality (d p ) always holds. Even when the optimization problem is not convex. ˆ strong duality (d = p ) often holds for convex problems (but not always)

21 Constraint qualifications ˆ weak duality (d p ) always holds. Even when the optimization problem is not convex. ˆ strong duality (d = p ) often holds for convex problems (but not always). A constraint qualification is a condition that guarantees strong duality

22 Constraint qualifications ˆ weak duality (d p ) always holds. Even when the optimization problem is not convex. ˆ strong duality (d = p ) often holds for convex problems (but not always). A constraint qualification is a condition that guarantees strong duality. An example we ve already seen: ˆ If the optimization problem is an LP, strong duality holds 14-11

23 Slater s constraint qualification x D f 0 (x) subject to: f i (x) 0 for i = 1,..., m h j (x) = 0 for j = 1,..., r Slater s constraint qualification: If the optimization problem is convex and strictly feasible, then strong duality holds. ˆ convexity requires: D and f i are convex and h j are affine. ˆ strict feasibility means there exists some x in the interior of D such that f i ( x) < 0 for i = 1,..., m

24 Slater s constraint qualification If the optimization problem is convex and strictly feasible, then strong duality holds. ˆ Good news: Slater s constraint qualification is rather weak. i.e. it is usually satisfied by convex problems. ˆ Can be relaxed so that strict feasibility is not required for the linear constraints

25 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ The function x 2 /y is convex for y > 0 (see plot) ˆ The objective e x is convex ˆ Feasible set: {(0, y) y > 0} ˆ Solution is trivial (p = 1) 14-14

26 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y 14-15

27 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y ˆ Dual function: g(λ) = inf x,y>0 (e x + λx 2 /y) =

28 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y ˆ Dual function: g(λ) = inf x,y>0 (e x + λx 2 /y) = 0. ˆ The dual problem is: maximize λ 0 0 So we have d = 0 < 1 = p

29 Counterexample (Boyd) x R, y>0 e x subject to: x 2 /y 0 ˆ Lagrangian: L(x, y, λ) = e x + λx 2 /y ˆ Dual function: g(λ) = inf x,y>0 (e x + λx 2 /y) = 0. ˆ The dual problem is: maximize λ 0 0 So we have d = 0 < 1 = p. ˆ Slater s constraint qualification is not satisfied! 14-15

30 About Slater s constraint qualification Slater s condition is only sufficient. (Slater) = (strong duality) 14-16

31 About Slater s constraint qualification Slater s condition is only sufficient. (Slater) = (strong duality) ˆ There exist problems where Slater s condition fails, yet strong duality holds. ˆ There exist nonconvex problems with strong duality

32 Complementary slackness Assume strong duality holds. If x is primal optimal and (λ, ν ) is dual optimal, then we have: g(λ, ν ) = d = p = f 0 (x ) 14-17

33 Complementary slackness Assume strong duality holds. If x is primal optimal and (λ, ν ) is dual optimal, then we have: f 0 (x ) = g(λ, ν ) = inf x D g(λ, ν ) = d = p = f 0 (x ) ( f 0 (x ) + f 0 (x ) f 0 (x) + m λ i f i (x) + i=1 m λ i f i (x ) + i=1 ) r νj h j (x) j=1 r νj h j (x ) The last inequality holds because x is primal feasible. We conclude that the inequalities must all be equalities. j=

34 Complementary slackness ˆ We concluded that: f 0 (x ) = f 0 (x ) + m λ i f i (x ) + i=1 r νj h j (x ) j=1 But f i (x ) 0 and h j (x ) = 0. Therefore: λ i f i (x ) = 0 for i = 1,..., m 14-18

35 Complementary slackness ˆ We concluded that: f 0 (x ) = f 0 (x ) + m λ i f i (x ) + i=1 r νj h j (x ) j=1 But f i (x ) 0 and h j (x ) = 0. Therefore: λ i f i (x ) = 0 for i = 1,..., m ˆ This property is called complementary slackness. We ve seen it before for linear programs

36 Complementary slackness ˆ We concluded that: f 0 (x ) = f 0 (x ) + m λ i f i (x ) + i=1 r νj h j (x ) j=1 But f i (x ) 0 and h j (x ) = 0. Therefore: λ i f i (x ) = 0 for i = 1,..., m ˆ This property is called complementary slackness. We ve seen it before for linear programs. λ i > 0 = f i (x ) = 0 and f i (x ) < 0 = λ i =

37 Dual of an LP x 0 subject to: c T x Ax b 14-19

38 Dual of an LP x 0 subject to: c T x Ax b ˆ Lagrangian: L(x, λ) = c T x + λ T (b Ax) 14-19

39 Dual of an LP x 0 subject to: c T x Ax b ˆ Lagrangian: L(x, λ) = c T x + λ T (b Ax) ˆ Dual function: g(λ) = min x 0 (c AT λ) T x + λ T b 14-19

40 Dual of an LP x 0 subject to: c T x Ax b ˆ Lagrangian: L(x, λ) = c T x + λ T (b Ax) ˆ Dual function: g(λ) = min x 0 (c AT λ) T x + λ T b g(λ) = { λ T b if A T λ c otherwise 14-19

41 Dual of an LP x 0 subject to: c T x Ax b ˆ Dual is: maximize λ 0 subject to: λ T b A T λ c ˆ This is the same result that we found when we were studying duality for linear programs

42 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x

43 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Lagrangian: L(x, λ, µ) = c T x + λ T (b Ax) µ T x 14-21

44 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Lagrangian: L(x, λ, µ) = c T x + λ T (b Ax) µ T x ˆ Dual function: g(λ, µ) = min (c A T λ µ) T x + λ T b x 14-21

45 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Lagrangian: L(x, λ, µ) = c T x + λ T (b Ax) µ T x ˆ Dual function: g(λ, µ) = min (c A T λ µ) T x + λ T b x { λ T b if A T λ + µ = c g(λ) = otherwise 14-21

46 Dual of an LP What if we treat x 0 as a constraint instead? (D = R n ). x subject to: c T x Ax b x 0 ˆ Dual is: maximize λ 0, µ 0 subject to: λ T b A T λ + µ = c ˆ Solution is the same, µ acts as the slack variable

47 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b 14-23

48 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Lagrangian: L(x, λ) = 1 2 x T Qx + λ T (b Ax) 14-23

49 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Lagrangian: L(x, λ) = 1x T Qx + λ T (b Ax) 2 ( ˆ Dual function: g(λ) = min 1 x T Qx + λ T (b Ax) ) x 2 Minimum occurs at: ˆx = Q 1 A T λ 14-23

50 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Lagrangian: L(x, λ) = 1x T Qx + λ T (b Ax) 2 ( ˆ Dual function: g(λ) = min 1 x T Qx + λ T (b Ax) ) x 2 Minimum occurs at: ˆx = Q 1 A T λ g(λ) = 1 2 λt AQ 1 A T λ + λ T b 14-23

51 Dual of a convex QP Suppose Q 0. Let s find the dual of the QP: x subject to: 1 x T Qx 2 Ax b ˆ Dual is also a QP: maximize λ 1 2 λt AQ 1 A T λ + λ T b subject to: λ 0 ˆ It s still easy to solve (maximizing a concave function) 14-24

52 Sensitivity analysis min x D f 0 (x) s.t. f i (x) u i i h j (x) = v j j max λ,ν g(λ, ν) λ T u ν T v s.t. λ 0 ˆ As with LPs, dual variables quantify the sensitivity of the optimal cost to changes in each of the constraints

53 Sensitivity analysis min x D f 0 (x) s.t. f i (x) u i i h j (x) = v j j max λ,ν g(λ, ν) λ T u ν T v s.t. λ 0 ˆ As with LPs, dual variables quantify the sensitivity of the optimal cost to changes in each of the constraints. ˆ A change in u i causes a bigger change in p if λ i ˆ A change in v j causes a bigger change in p if ν j is larger. is larger

54 Sensitivity analysis min x D f 0 (x) s.t. f i (x) u i i h j (x) = v j j max λ,ν g(λ, ν) λ T u ν T v s.t. λ 0 ˆ As with LPs, dual variables quantify the sensitivity of the optimal cost to changes in each of the constraints. ˆ A change in u i causes a bigger change in p if λ i ˆ A change in v j causes a bigger change in p if ν j is larger. is larger. ˆ If p (u, v) is differentiable, then: λ i = p (0, 0) u i and ν j = p (0, 0) v j 14-25

55 Next class... ˆ Review! 14-26

Lagrangian Duality and Convex Optimization

Lagrangian Duality and Convex Optimization Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?

More information

5. Duality. Lagrangian

5. Duality. Lagrangian 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

Convex Optimization Boyd & Vandenberghe. 5. Duality

Convex Optimization Boyd & Vandenberghe. 5. Duality 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

Convex Optimization M2

Convex Optimization M2 Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization

More information

Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities

Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form

More information

Lecture 18: Optimization Programming

Lecture 18: Optimization Programming Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming

More information

Lagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)

Lagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST) Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual

More information

Lecture: Duality.

Lecture: Duality. Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong

More information

EE364a Review Session 5

EE364a Review Session 5 EE364a Review Session 5 EE364a Review announcements: homeworks 1 and 2 graded homework 4 solutions (check solution to additional problem 1) scpd phone-in office hours: tuesdays 6-7pm (650-723-1156) 1 Complementary

More information

Lagrange duality. The Lagrangian. We consider an optimization program of the form

Lagrange duality. The Lagrangian. We consider an optimization program of the form Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization

More information

The Lagrangian L : R d R m R r R is an (easier to optimize) lower bound on the original problem:

The Lagrangian L : R d R m R r R is an (easier to optimize) lower bound on the original problem: HT05: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford Convex Optimization and slides based on Arthur Gretton s Advanced Topics in Machine Learning course

More information

Lecture: Duality of LP, SOCP and SDP

Lecture: Duality of LP, SOCP and SDP 1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:

More information

Duality. Geoff Gordon & Ryan Tibshirani Optimization /

Duality. Geoff Gordon & Ryan Tibshirani Optimization / Duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider

More information

EE/AA 578, Univ of Washington, Fall Duality

EE/AA 578, Univ of Washington, Fall Duality 7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

Constrained Optimization and Lagrangian Duality

Constrained Optimization and Lagrangian Duality CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

More information

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

More information

Convex Optimization & Lagrange Duality

Convex Optimization & Lagrange Duality Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT

More information

I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010

I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010 I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0

More information

Convex Optimization and SVM

Convex Optimization and SVM Convex Optimization and SVM Problem 0. Cf lecture notes pages 12 to 18. Problem 1. (i) A slab is an intersection of two half spaces, hence convex. (ii) A wedge is an intersection of two half spaces, hence

More information

Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014

Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014 Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,

More information

Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs

Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following

More information

Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers

Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex

More information

Applications of Linear Programming

Applications of Linear Programming Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal

More information

Lagrangian Duality Theory

Lagrangian Duality Theory Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual

More information

On the Method of Lagrange Multipliers

On the Method of Lagrange Multipliers On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should

More information

CSCI : Optimization and Control of Networks. Review on Convex Optimization

CSCI : Optimization and Control of Networks. Review on Convex Optimization CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one

More information

A Brief Review on Convex Optimization

A Brief Review on Convex Optimization A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review

More information

CS-E4830 Kernel Methods in Machine Learning

CS-E4830 Kernel Methods in Machine Learning CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This

More information

Linear and Combinatorial Optimization

Linear and Combinatorial Optimization Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality

More information

Homework Set #6 - Solutions

Homework Set #6 - Solutions EE 15 - Applications of Convex Optimization in Signal Processing and Communications Dr Andre Tkacenko JPL Third Term 11-1 Homework Set #6 - Solutions 1 a The feasible set is the interval [ 4] The unique

More information

Primal/Dual Decomposition Methods

Primal/Dual Decomposition Methods Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients

More information

Optimization for Machine Learning

Optimization for Machine Learning Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html

More information

subject to (x 2)(x 4) u,

subject to (x 2)(x 4) u, Exercises Basic definitions 5.1 A simple example. Consider the optimization problem with variable x R. minimize x 2 + 1 subject to (x 2)(x 4) 0, (a) Analysis of primal problem. Give the feasible set, the

More information

ICS-E4030 Kernel Methods in Machine Learning

ICS-E4030 Kernel Methods in Machine Learning ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This

More information

Tutorial on Convex Optimization for Engineers Part II

Tutorial on Convex Optimization for Engineers Part II Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

More information

Tutorial on Convex Optimization: Part II

Tutorial on Convex Optimization: Part II Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation

More information

Lecture Note 5: Semidefinite Programming for Stability Analysis

Lecture Note 5: Semidefinite Programming for Stability Analysis ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State

More information

Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research

Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,

More information

Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:

Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem: CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

More information

Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725

Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725 Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =

More information

Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

More information

4TE3/6TE3. Algorithms for. Continuous Optimization

4TE3/6TE3. Algorithms for. Continuous Optimization 4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality

More information

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

More information

Lecture 7: Convex Optimizations

Lecture 7: Convex Optimizations Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1

More information

Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark

Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian

More information

Convex Optimization M2

Convex Optimization M2 Convex Optimization M2 Lecture 8 A. d Aspremont. Convex Optimization M2. 1/57 Applications A. d Aspremont. Convex Optimization M2. 2/57 Outline Geometrical problems Approximation problems Combinatorial

More information

E5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming

E5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program

More information

Sparse Optimization Lecture: Dual Certificate in l 1 Minimization

Sparse Optimization Lecture: Dual Certificate in l 1 Minimization Sparse Optimization Lecture: Dual Certificate in l 1 Minimization Instructor: Wotao Yin July 2013 Note scriber: Zheng Sun Those who complete this lecture will know what is a dual certificate for l 1 minimization

More information

Maximum likelihood estimation

Maximum likelihood estimation Maximum likelihood estimation Guillaume Obozinski Ecole des Ponts - ParisTech Master MVA Maximum likelihood estimation 1/26 Outline 1 Statistical concepts 2 A short review of convex analysis and optimization

More information

Lecture 8. Strong Duality Results. September 22, 2008

Lecture 8. Strong Duality Results. September 22, 2008 Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation

More information

HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.

HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given. HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

More information

Interior Point Algorithms for Constrained Convex Optimization

Interior Point Algorithms for Constrained Convex Optimization Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems

More information

Support vector machines

Support vector machines Support vector machines Guillaume Obozinski Ecole des Ponts - ParisTech SOCN course 2014 SVM, kernel methods and multiclass 1/23 Outline 1 Constrained optimization, Lagrangian duality and KKT 2 Support

More information

EE 227A: Convex Optimization and Applications October 14, 2008

EE 227A: Convex Optimization and Applications October 14, 2008 EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider

More information

Duality in Linear Programs. Lecturer: Ryan Tibshirani Convex Optimization /36-725

Duality in Linear Programs. Lecturer: Ryan Tibshirani Convex Optimization /36-725 Duality in Linear Programs Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: proximal gradient descent Consider the problem x g(x) + h(x) with g, h convex, g differentiable, and

More information

Lectures 9 and 10: Constrained optimization problems and their optimality conditions

Lectures 9 and 10: Constrained optimization problems and their optimality conditions Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained

More information

12. Interior-point methods

12. Interior-point methods 12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

More information

Nonlinear Programming and the Kuhn-Tucker Conditions

Nonlinear Programming and the Kuhn-Tucker Conditions Nonlinear Programming and the Kuhn-Tucker Conditions The Kuhn-Tucker (KT) conditions are first-order conditions for constrained optimization problems, a generalization of the first-order conditions we

More information

CS711008Z Algorithm Design and Analysis

CS711008Z Algorithm Design and Analysis CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief

More information

Finite Dimensional Optimization Part III: Convex Optimization 1

Finite Dimensional Optimization Part III: Convex Optimization 1 John Nachbar Washington University March 21, 2017 Finite Dimensional Optimization Part III: Convex Optimization 1 1 Saddle points and KKT. These notes cover another important approach to optimization,

More information

10 Numerical methods for constrained problems

10 Numerical methods for constrained problems 10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside

More information

Convex Optimization and Modeling

Convex Optimization and Modeling Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

More information

Support Vector Machines: Maximum Margin Classifiers

Support Vector Machines: Maximum Margin Classifiers Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and Fu-Jie Huang 1 Outline What is behind

More information

Nonlinear Programming

Nonlinear Programming Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week

More information

Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016

Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 1 Entropy Since this course is about entropy maximization,

More information

Optimality Conditions for Constrained Optimization

Optimality Conditions for Constrained Optimization 72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)

More information

Constrained optimization

Constrained optimization Constrained optimization DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Compressed sensing Convex constrained

More information

Introduction to Nonlinear Stochastic Programming

Introduction to Nonlinear Stochastic Programming School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS

More information

Convexification of Mixed-Integer Quadratically Constrained Quadratic Programs

Convexification of Mixed-Integer Quadratically Constrained Quadratic Programs Convexification of Mixed-Integer Quadratically Constrained Quadratic Programs Laura Galli 1 Adam N. Letchford 2 Lancaster, April 2011 1 DEIS, University of Bologna, Italy 2 Department of Management Science,

More information

Duality Uses and Correspondences. Ryan Tibshirani Convex Optimization

Duality Uses and Correspondences. Ryan Tibshirani Convex Optimization Duality Uses and Correspondences Ryan Tibshirani Conve Optimization 10-725 Recall that for the problem Last time: KKT conditions subject to f() h i () 0, i = 1,... m l j () = 0, j = 1,... r the KKT conditions

More information

The fundamental theorem of linear programming

The fundamental theorem of linear programming The fundamental theorem of linear programming Michael Tehranchi June 8, 2017 This note supplements the lecture notes of Optimisation The statement of the fundamental theorem of linear programming and the

More information

Convex Optimization and Support Vector Machine

Convex Optimization and Support Vector Machine Convex Optimization and Support Vector Machine Problem 0. Consider a two-class classification problem. The training data is L n = {(x 1, t 1 ),..., (x n, t n )}, where each t i { 1, 1} and x i R p. We

More information

Lecture 6: Conic Optimization September 8

Lecture 6: Conic Optimization September 8 IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

More information

UC Berkeley Department of Electrical Engineering and Computer Science. EECS 227A Nonlinear and Convex Optimization. Solutions 5 Fall 2009

UC Berkeley Department of Electrical Engineering and Computer Science. EECS 227A Nonlinear and Convex Optimization. Solutions 5 Fall 2009 UC Berkeley Department of Electrical Engineering and Computer Science EECS 227A Nonlinear and Convex Optimization Solutions 5 Fall 2009 Reading: Boyd and Vandenberghe, Chapter 5 Solution 5.1 Note that

More information

4. Algebra and Duality

4. Algebra and Duality 4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone

More information

LECTURE 10 LECTURE OUTLINE

LECTURE 10 LECTURE OUTLINE LECTURE 10 LECTURE OUTLINE Min Common/Max Crossing Th. III Nonlinear Farkas Lemma/Linear Constraints Linear Programming Duality Convex Programming Duality Optimality Conditions Reading: Sections 4.5, 5.1,5.2,

More information

Lecture 14: Optimality Conditions for Conic Problems

Lecture 14: Optimality Conditions for Conic Problems EE 227A: Conve Optimization and Applications March 6, 2012 Lecture 14: Optimality Conditions for Conic Problems Lecturer: Laurent El Ghaoui Reading assignment: 5.5 of BV. 14.1 Optimality for Conic Problems

More information

LINEAR PROGRAMMING. Relation to the Text (cont.) Relation to Material in Text. Relation to the Text. Relation to the Text (cont.

LINEAR PROGRAMMING. Relation to the Text (cont.) Relation to Material in Text. Relation to the Text. Relation to the Text (cont. LINEAR PROGRAMMING Relation to Material in Text After a brief introduction to linear programming on p. 3, Cornuejols and Tϋtϋncϋ give a theoretical discussion including duality, and the simplex solution

More information

Lecture Notes on Support Vector Machine

Lecture Notes on Support Vector Machine Lecture Notes on Support Vector Machine Feng Li fli@sdu.edu.cn Shandong University, China 1 Hyperplane and Margin In a n-dimensional space, a hyper plane is defined by ω T x + b = 0 (1) where ω R n is

More information

Understanding the Simplex algorithm. Standard Optimization Problems.

Understanding the Simplex algorithm. Standard Optimization Problems. Understanding the Simplex algorithm. Ma 162 Spring 2011 Ma 162 Spring 2011 February 28, 2011 Standard Optimization Problems. A standard maximization problem can be conveniently described in matrix form

More information

Lagrangian Duality. Evelien van der Hurk. DTU Management Engineering

Lagrangian Duality. Evelien van der Hurk. DTU Management Engineering Lagrangian Duality Evelien van der Hurk DTU Management Engineering Topics Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality 2 DTU Management Engineering 42111: Static and Dynamic Optimization

More information

Lagrange Relaxation and Duality

Lagrange Relaxation and Duality Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler

More information

A Tutorial on Convex Optimization II: Duality and Interior Point Methods

A Tutorial on Convex Optimization II: Duality and Interior Point Methods A Tutorial on Convex Optimization II: Duality and Interior Point Methods Haitham Hindi Palo Alto Research Center (PARC), Palo Alto, California 94304 email: hhindi@parc.com Abstract In recent years, convex

More information

Math 273a: Optimization Subgradients of convex functions

Math 273a: Optimization Subgradients of convex functions Math 273a: Optimization Subgradients of convex functions Made by: Damek Davis Edited by Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com 1 / 42 Subgradients Assumptions

More information

Interior Point Methods for LP

Interior Point Methods for LP 11.1 Interior Point Methods for LP Katta G. Murty, IOE 510, LP, U. Of Michigan, Ann Arbor, Winter 1997. Simplex Method - A Boundary Method: Starting at an extreme point of the feasible set, the simplex

More information

7. Statistical estimation

7. Statistical estimation 7. Statistical estimation Convex Optimization Boyd & Vandenberghe maximum likelihood estimation optimal detector design experiment design 7 1 Parametric distribution estimation distribution estimation

More information

Introduction to Machine Learning Spring 2018 Note Duality. 1.1 Primal and Dual Problem

Introduction to Machine Learning Spring 2018 Note Duality. 1.1 Primal and Dual Problem CS 189 Introduction to Machine Learning Spring 2018 Note 22 1 Duality As we have seen in our discussion of kernels, ridge regression can be viewed in two ways: (1) an optimization problem over the weights

More information

Example: feasibility. Interpretation as formal proof. Example: linear inequalities and Farkas lemma

Example: feasibility. Interpretation as formal proof. Example: linear inequalities and Farkas lemma 4-1 Algebra and Duality P. Parrilo and S. Lall 2006.06.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone of valid

More information

Lecture #21. c T x Ax b. maximize subject to

Lecture #21. c T x Ax b. maximize subject to COMPSCI 330: Design and Analysis of Algorithms 11/11/2014 Lecture #21 Lecturer: Debmalya Panigrahi Scribe: Samuel Haney 1 Overview In this lecture, we discuss linear programming. We first show that the

More information

CS711008Z Algorithm Design and Analysis

CS711008Z Algorithm Design and Analysis .. CS711008Z Algorithm Design and Analysis Lecture 9. Algorithm design technique: Linear programming and duality Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China

More information

Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation

Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu

More information

Optimization. Yuh-Jye Lee. March 28, Data Science and Machine Intelligence Lab National Chiao Tung University 1 / 40

Optimization. Yuh-Jye Lee. March 28, Data Science and Machine Intelligence Lab National Chiao Tung University 1 / 40 Optimization Yuh-Jye Lee Data Science and Machine Intelligence Lab National Chiao Tung University March 28, 2017 1 / 40 The Key Idea of Newton s Method Let f : R n R be a twice differentiable function

More information

ECE Optimization for wireless networks Final. minimize f o (x) s.t. Ax = b,

ECE Optimization for wireless networks Final. minimize f o (x) s.t. Ax = b, ECE 788 - Optimization for wireless networks Final Please provide clear and complete answers. PART I: Questions - Q.. Discuss an iterative algorithm that converges to the solution of the problem minimize

More information

Part IB Optimisation

Part IB Optimisation Part IB Optimisation Theorems Based on lectures by F. A. Fischer Notes taken by Dexter Chua Easter 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after

More information

LP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra

LP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality

More information

Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006 Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

More information

Conic Linear Optimization and its Dual. yyye

Conic Linear Optimization and its Dual.   yyye Conic Linear Optimization and Appl. MS&E314 Lecture Note #04 1 Conic Linear Optimization and its Dual Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.

More information

Duality of LPs and Applications

Duality of LPs and Applications Lecture 6 Duality of LPs and Applications Last lecture we introduced duality of linear programs. We saw how to form duals, and proved both the weak and strong duality theorems. In this lecture we will

More information

minimize x subject to (x 2)(x 4) u,

minimize x subject to (x 2)(x 4) u, Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for

More information

Lecture 3 January 28

Lecture 3 January 28 EECS 28B / STAT 24B: Advanced Topics in Statistical LearningSpring 2009 Lecture 3 January 28 Lecturer: Pradeep Ravikumar Scribe: Timothy J. Wheeler Note: These lecture notes are still rough, and have only

More information