# 4. Algebra and Duality

Size: px
Start display at page:

Transcription

1 4-1 Algebra and Duality P. Parrilo and S. Lall, CDC Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone of valid inequalities Algebraic geometry The cone generated by a set of polynomials An algebraic approach to duality Example: feasibility Searching the cone Interpretation as formal proof Example: linear inequalities and Farkas lemma

2 4-2 Algebra and Duality P. Parrilo and S. Lall, CDC Example minimize x 1 x 2 subject to x 1 0 x 2 0 x x2 2 1 The objective is not convex. The Lagrange dual function is ) g(λ) = inf (x 1 x 2 λ 1 x 1 λ 2 x 2 + λ 3 (x x2 2 1) = x [ λ λ 1 λ 2 ] T [ 2λ λ ] 1 [ 0 λ 1 λ 2 ] if λ 3 > otherwise, except bdry

3 4-3 Algebra and Duality P. Parrilo and S. Lall, CDC Example, continued The dual problem is maximize g(λ) 0.5 subject to λ 1 0 λ 2 0 g(λ) λ λ 1 λ 3 By symmetry, if the maximum g(λ) is attained, then λ 1 = λ 2 at optimality The optimal g(λ ) = 1 2 at λ = (0, 0, 1 2 ) Here we see an example of a duality gap; the primal optimal is strictly greater than the dual optimal

4 4-4 Algebra and Duality P. Parrilo and S. Lall, CDC Example, continued It turns out that, using the Schur complement, the dual problem may be written as maximize subject to γ 2γ 2λ 3 λ 1 λ 2 λ 1 2λ 3 1 > 0 λ 2 1 2λ 3 λ 1 > 0 λ 2 > 0 In this workshop we ll see a systematic way to convert a dual problem to an SDP, whenever the objective and constraint functions are polynomials.

5 4-5 Algebra and Duality P. Parrilo and S. Lall, CDC The Dual is Not Intrinsic The dual problem, and its corresponding optimal value, are not properties of the primal feasible set and objective function alone. Instead, they depend on the particular equations and inequalities used To construct equivalent primal optimization problems with different duals: replace the objective f 0 (x) by h(f 0 (x)) where h is increasing introduce new variables and associated constraints, e.g. is replaced by minimize (x 1 x 2 ) 2 + (x 2 x 3 ) 2 minimize (x 1 x 2 ) 2 + (x 4 x 3 ) 2 add redundant constraints subject to x 2 = x 4

6 4-6 Algebra and Duality P. Parrilo and S. Lall, CDC Example Adding the redundant constraint x 1 x 2 0 to the previous example gives 0.6 minimize x 1 x 2 subject to x 1 0 x 2 0 x x2 2 1 x 1 x Clearly, this has the same primal feasible set and same optimal value as before

7 4-7 Algebra and Duality P. Parrilo and S. Lall, CDC Example Continued The Lagrange dual function is ) g(λ) = inf (x 1 x 2 λ 1 x 1 λ 2 x 2 + λ 3 (x x2 2 1) λ 4x 1 x 2 x λ = [ λ 1 λ 2 ] T [ ] 1 [ 2λ 3 1 λ 4 1 λ 4 2λ 3 λ 1 λ 2 ] if 2λ 3 > 1 λ 4 otherwise, except bdry Again, this problem may also be written as an SDP. The optimal value is g(λ ) = 0 at λ = (0, 0, 0, 1) Adding redundant constraints makes the dual bound tighter This always happens! inequalities. Such redundant constraints are called valid

8 4-8 Algebra and Duality P. Parrilo and S. Lall, CDC Constructing Valid Inequalities The function f : R n R is called a valid inequality if f(x) 0 for all feasible x Given a set of inequality constraints, we can generate others as follows. (i) If f 1 and f 2 define valid inequalities, then so does h(x) = f 1 (x)+f 2 (x) (ii) If f 1 and f 2 define valid inequalities, then so does h(x) = f 1 (x)f 2 (x) (iii) For any f, the function h(x) = f(x) 2 defines a valid inequality We view these as inference rules Now we can use algebra to generate valid inequalities.

9 4-9 Algebra and Duality P. Parrilo and S. Lall, CDC The Cone of Valid Inequalities The set of polynomial functions on R n with real coefficients is denoted R[x 1,..., x n ] Computationally, they are easy to parametrize. We will consider polynomial constraint functions. A set of polynomials P R[x 1,..., x n ] is called a cone if (i) f 1 P and f 2 P implies f 1 f 2 P (ii) f 1 P and f 2 P implies f 1 + f 2 P (iii) f R[x 1,..., x n ] implies f 2 P It is called a proper cone if 1 P By applying the above rules to the inequality constraint functions, we can generate a cone of valid inequalities

10 4-10 Algebra and Duality P. Parrilo and S. Lall, CDC Algebraic Geometry There is a correspondence between the geometric object (the feasible subset of R n ) and the algebraic object (the cone of valid inequalities) This is a dual relationship; we ll see more of this later. The dual problem is constructed from the cone. For equality constraints, there is another algebraic object; the ideal generated by the equality constraints. For optimization, we need to look both at the geometric objects (for the primal) and the algebraic objects (for the dual problem)

11 4-11 Algebra and Duality P. Parrilo and S. Lall, CDC Cones For S R n, the cone defined by S is { C(S) = f R[x 1,..., x n ] f(x) 0 for all x S } If P 1 and P 2 are cones, then so is P 1 P 2 A polynomial f R[x 1,..., x n ] is called a sum-of-squares (SOS) if f(x) = r s i (x) 2 i=1 for some polynomials s 1,..., s r and some r 0. The set of SOS polynomials Σ is a cone. Every cone contains Σ.

12 4-12 Algebra and Duality P. Parrilo and S. Lall, CDC Cones The set monoid{f 1,..., f m } R[x 1,..., x n ] is the set of all finite products of polynomials f i, together with 1. The smallest cone containing the polynomials f 1,..., f m is { r cone{f 1,..., f m } = i=1 s i g i s 0,..., s r Σ, g i monoid{f 1,..., f m } } cone{f 1,..., f m } is called the cone generated by f 1,..., f m

13 4-13 Algebra and Duality P. Parrilo and S. Lall, CDC Explicit Parametrization of the Cone If f 1,..., f m are valid inequalities, then so is every polynomial in cone{f 1,..., f m } The polynomial h is an element of cone{f 1,..., f m } if and only if h(x) = s 0 (x) + m i=1 s i (x)f i (x) + i j r ij (x)f i (x)f j (x) +... where the s i and r ij are sums-of-squares.

14 4-14 Algebra and Duality P. Parrilo and S. Lall, CDC An Algebraic Approach to Duality Suppose f 1,..., f m are polynomials, and consider the feasibility problem does there exist x R n such that f i (x) 0 for all i = 1,..., m Every polynomial in cone{f 1,..., f m } is non-negative on the feasible set. So if there is a polynomial q cone{f 1,..., f m } which satisfies q(x) ε < 0 for all x R n then the primal problem is infeasible.

15 4-15 Algebra and Duality P. Parrilo and S. Lall, CDC Example Let s look at the feasibility version of the previous problem. Given t R, does there exist x R 2 such that x 1 x 2 t x x2 2 1 x 1 0 x 2 0 Equivalently, is the set S nonempty, where { S = x R n f i (x) 0 for all i = 1,..., m } where f 1 (x) = t x 1 x 2 f 2 (x) = 1 x 2 1 x2 2 f 3 (x) = x 1 f 4 (x) = x 2

16 4-16 Algebra and Duality P. Parrilo and S. Lall, CDC Example Continued Let q(x) = f 1 (x) f 2(x). Then clearly q cone{f 1, f 2, f 3, f 4 } and q(x) = t x 1 x (1 x2 1 x2 2 ) = t (x 1 + x 2 ) 2 t So for any t < 1 2, the primal problem is infeasible. Corresponds to Lagrange multipliers (1, 0, 0, 1 2 ) for the thm. of alternatives. Alternatively, this is a proof by contradiction. If there exists x such that f i (x) 0 for i = 1,..., 4 then we must also have q(x) 0, since q cone{f 1,..., f 4 } But we proved that q is negative if t < 1 2

17 4-17 Algebra and Duality P. Parrilo and S. Lall, CDC Example Continued We can also do better by using other functions in the cone. Try q(x) = f 1 (x) + f 3 (x)f 4 (x) = t giving the stronger result that for any t < 0 the inequalities are infeasible. Again, this corresponds to Lagrange multipliers (1, 0, 0, 1) In both of these examples, we found q in the cone which was globally negative. We can view q as the Lagrangian function evaluated at a particular value of λ The Lagrange multiplier procedure is searching over a particular subset of functions in the cone; those which are generated by linear combinations of the original constraints. By searching over more functions in the cone we can do better

18 4-18 Algebra and Duality P. Parrilo and S. Lall, CDC Normalization In the above example, we have q(x) = t (x 1 + x 2 ) 2 We can also show that 1 cone{f 1,..., f 4 }, which gives a very simple proof of primal infeasibility. Because, for t < 1 2, we have 1 = a 0 q(x) + a 1 (x 1 + x 2 ) 2 and by construction q is in the cone, and (x 1 + x 2 ) 2 is a sum of squares. Here a 0 and a 1 are positive constants a 0 = 2 2t + 1 a 1 = 1 2t + 1

19 4-19 Algebra and Duality P. Parrilo and S. Lall, CDC An Algebraic Dual Problem Suppose f 1,..., f m are polynomials. The primal feasibility problem is does there exist x R n such that f i (x) 0 for all i = 1,..., m The dual feasibility problem is Is it true that 1 cone{f 1,..., f m } If the dual problem is feasible, then the primal problem is infeasible. In fact, a result called the Positivstellensatz implies that strong duality holds here.

20 4-20 Algebra and Duality P. Parrilo and S. Lall, CDC Interpretation: Searching the Cone Lagrange duality is searching over linear combinations with nonnegative coefficients λ 1 f λ m f m to find a globally negative function as a certificate The above algebraic procedure is searching over conic combinations s 0 (x) + m s i (x)f i (x) + i j r ij (x)f i (x)f j (x) +... i=1 where the s i and r ij are sums-of-squares

21 4-21 Algebra and Duality P. Parrilo and S. Lall, CDC Interpretation: Formal Proof We can also view this as a type of formal proof: View f 1,..., f m are predicates, with f i (x) 0 meaning that x satisfies f i. Then cone{f 1,..., f m } consists of predicates which are logical consequences of f 1,..., f m. If we find 1 in the cone, then we have a proof by contradiction. Our objective is to automatically search the cone for negative functions; i.e., proofs of infeasibility.

22 4-22 Algebra and Duality P. Parrilo and S. Lall, CDC Example: Linear Inequalities Does there exist x R n such that Ax 0 c T x 1 Write A in terms of its rows A = a T 1., a T m then we have inequality constraints defined by linear polynomials f i (x) = a T i x f m+1 (x) = 1 c T x for i = 1,..., m

23 4-23 Algebra and Duality P. Parrilo and S. Lall, CDC Example: Linear Inequalities We ll search over functions q cone{f 1,..., f m+1 } of the form q(x) = m i=1 λ i f i (x) + µf m+1 (x) Then the algebraic form of the dual is: does there exist λ i 0, µ 0 such that q(x) = 1 for all x if the dual is feasible, then the primal problem is infeasible

24 4-24 Algebra and Duality P. Parrilo and S. Lall, CDC Example: Linear Inequalities The above dual condition is λ T Ax + µ( 1 c T x) = 1 for all x which holds if and only if A T λ = µc and µ = 1. So we can state the duality result as follows. Farkas Lemma If there exists λ R m such that A T λ = c and λ 0 then there does not exist x R n such that Ax 0 and c T x 1

25 4-25 Algebra and Duality P. Parrilo and S. Lall, CDC Farkas Lemma Farkas Lemma states that the following are strong alternatives (i) there exists λ R m such that A T λ = c and λ 0 (ii) there exists x R n such that Ax 0 and c T x < 0 Since this is just Lagrangian duality, there is a geometric interpretation (i) c is in the convex cone { A T λ λ 0 } a 2 (ii) x defines the hyperplane { y R n y T x = 0 } c a 1 x which separates c from the cone

26 4-26 Algebra and Duality P. Parrilo and S. Lall, CDC Optimization Problems Let s return to optimization problems instead of feasibility problems. minimize f 0 (x) subject to f i (x) 0 for all i = 1,..., m The corresponding feasibility problem is t f 0 (x) 0 f i (x) 0 for all i = 1,..., m One simple dual is to find polynomials s i and r ij such that t f 0 (x) + m s i (x)f i (x) + i j r ij (x)f i (x)f j (x) +... i=1 is globally negative, where the s i and r ij are sums-of-squares

27 4-27 Algebra and Duality P. Parrilo and S. Lall, CDC Optimization Problems We can combine this with a maximization over t maximize t m subject to t f 0 (x) + s i (x)f i (x)+ i=1 m i=1 j=1 s i, r ij are sums-of-squares m r ij (x)f i (x)f j (x) 0 for all x The variables here are (coefficients of) the polynomials s i, r i We will see later how to approach this kind of problem using semidefinite programming

### Example: feasibility. Interpretation as formal proof. Example: linear inequalities and Farkas lemma

4-1 Algebra and Duality P. Parrilo and S. Lall 2006.06.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone of valid

### 6-1 The Positivstellensatz P. Parrilo and S. Lall, ECC

6-1 The Positivstellensatz P. Parrilo and S. Lall, ECC 2003 2003.09.02.10 6. The Positivstellensatz Basic semialgebraic sets Semialgebraic sets Tarski-Seidenberg and quantifier elimination Feasibility

### 4-1 The Algebraic-Geometric Dictionary P. Parrilo and S. Lall, ECC

4-1 The Algebraic-Geometric Dictionary P. Parrilo and S. Lall, ECC 2003 2003.09.02.02 4. The Algebraic-Geometric Dictionary Equality constraints Ideals and Varieties Feasibility problems and duality The

### Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers

Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex

### I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010

I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0

### LP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra

LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality

### Convex Optimization M2

Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization

### Convex Optimization. (EE227A: UC Berkeley) Lecture 28. Suvrit Sra. (Algebra + Optimization) 02 May, 2013

Convex Optimization (EE227A: UC Berkeley) Lecture 28 (Algebra + Optimization) 02 May, 2013 Suvrit Sra Admin Poster presentation on 10th May mandatory HW, Midterm, Quiz to be reweighted Project final report

### SEMIDEFINITE PROGRAM BASICS. Contents

SEMIDEFINITE PROGRAM BASICS BRIAN AXELROD Abstract. A introduction to the basics of Semidefinite programs. Contents 1. Definitions and Preliminaries 1 1.1. Linear Algebra 1 1.2. Convex Analysis (on R n

### Robust and Optimal Control, Spring 2015

Robust and Optimal Control, Spring 2015 Instructor: Prof. Masayuki Fujita (S5-303B) G. Sum of Squares (SOS) G.1 SOS Program: SOS/PSD and SDP G.2 Duality, valid ineqalities and Cone G.3 Feasibility/Optimization

### EE364a Review Session 5

EE364a Review Session 5 EE364a Review announcements: homeworks 1 and 2 graded homework 4 solutions (check solution to additional problem 1) scpd phone-in office hours: tuesdays 6-7pm (650-723-1156) 1 Complementary

### Approximate Farkas Lemmas in Convex Optimization

Approximate Farkas Lemmas in Convex Optimization Imre McMaster University Advanced Optimization Lab AdvOL Graduate Student Seminar October 25, 2004 1 Exact Farkas Lemma Motivation 2 3 Future plans The

### Lecture 3: Semidefinite Programming

Lecture 3: Semidefinite Programming Lecture Outline Part I: Semidefinite programming, examples, canonical form, and duality Part II: Strong Duality Failure Examples Part III: Conditions for strong duality

### Lecture Note 5: Semidefinite Programming for Stability Analysis

ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State

### Part IB Optimisation

Part IB Optimisation Theorems Based on lectures by F. A. Fischer Notes taken by Dexter Chua Easter 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after

### 9. Interpretations, Lifting, SOS and Moments

9-1 Interpretations, Lifting, SOS and Moments P. Parrilo and S. Lall, CDC 2003 2003.12.07.04 9. Interpretations, Lifting, SOS and Moments Polynomial nonnegativity Sum of squares (SOS) decomposition Eample

### Lecture 5. Theorems of Alternatives and Self-Dual Embedding

IE 8534 1 Lecture 5. Theorems of Alternatives and Self-Dual Embedding IE 8534 2 A system of linear equations may not have a solution. It is well known that either Ax = c has a solution, or A T y = 0, c

### Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

### Conic Linear Optimization and its Dual. yyye

Conic Linear Optimization and Appl. MS&E314 Lecture Note #04 1 Conic Linear Optimization and its Dual Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.

### Convex Optimization & Lagrange Duality

Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT

### Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs

Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following

### Agenda. 1 Duality for LP. 2 Theorem of alternatives. 3 Conic Duality. 4 Dual cones. 5 Geometric view of cone programs. 6 Conic duality theorem

Agenda 1 Duality for LP 2 Theorem of alternatives 3 Conic Duality 4 Dual cones 5 Geometric view of cone programs 6 Conic duality theorem 7 Examples Lower bounds on LPs By eliminating variables (if needed)

### EE 227A: Convex Optimization and Applications October 14, 2008

EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider

### Week 3 Linear programming duality

Week 3 Linear programming duality This week we cover the fascinating topic of linear programming duality. We will learn that every minimization program has associated a maximization program that has the

### Midterm Review. Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.

Midterm Review Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye (LY, Chapter 1-4, Appendices) 1 Separating hyperplane

### Lecture 5. The Dual Cone and Dual Problem

IE 8534 1 Lecture 5. The Dual Cone and Dual Problem IE 8534 2 For a convex cone K, its dual cone is defined as K = {y x, y 0, x K}. The inner-product can be replaced by x T y if the coordinates of the

### Lagrange duality. The Lagrangian. We consider an optimization program of the form

Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization

### 12. Interior-point methods

12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

### Semidefinite Programming

Semidefinite Programming Basics and SOS Fernando Mário de Oliveira Filho Campos do Jordão, 2 November 23 Available at: www.ime.usp.br/~fmario under talks Conic programming V is a real vector space h, i

### 5. Duality. Lagrangian

5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

### Note 3: LP Duality. If the primal problem (P) in the canonical form is min Z = n (1) then the dual problem (D) in the canonical form is max W = m (2)

Note 3: LP Duality If the primal problem (P) in the canonical form is min Z = n j=1 c j x j s.t. nj=1 a ij x j b i i = 1, 2,..., m (1) x j 0 j = 1, 2,..., n, then the dual problem (D) in the canonical

### Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

### Chapter 2: Preliminaries and elements of convex analysis

Chapter 2: Preliminaries and elements of convex analysis Edoardo Amaldi DEIB Politecnico di Milano edoardo.amaldi@polimi.it Website: http://home.deib.polimi.it/amaldi/opt-14-15.shtml Academic year 2014-15

Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.

### Lecture 14: Optimality Conditions for Conic Problems

EE 227A: Conve Optimization and Applications March 6, 2012 Lecture 14: Optimality Conditions for Conic Problems Lecturer: Laurent El Ghaoui Reading assignment: 5.5 of BV. 14.1 Optimality for Conic Problems

### LINEAR PROGRAMMING II

LINEAR PROGRAMMING II LP duality strong duality theorem bonus proof of LP duality applications Lecture slides by Kevin Wayne Last updated on 7/25/17 11:09 AM LINEAR PROGRAMMING II LP duality Strong duality

### Two hours. To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER. xx xxxx 2017 xx:xx xx.

Two hours To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER CONVEX OPTIMIZATION - SOLUTIONS xx xxxx 27 xx:xx xx.xx Answer THREE of the FOUR questions. If

### MIT Algebraic techniques and semidefinite optimization February 14, Lecture 3

MI 6.97 Algebraic techniques and semidefinite optimization February 4, 6 Lecture 3 Lecturer: Pablo A. Parrilo Scribe: Pablo A. Parrilo In this lecture, we will discuss one of the most important applications

### 14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity

### 4TE3/6TE3. Algorithms for. Continuous Optimization

4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality

### Duality Theory of Constrained Optimization

Duality Theory of Constrained Optimization Robert M. Freund April, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 The Practical Importance of Duality Duality is pervasive

### 15. Conic optimization

L. Vandenberghe EE236C (Spring 216) 15. Conic optimization conic linear program examples modeling duality 15-1 Generalized (conic) inequalities Conic inequality: a constraint x K where K is a convex cone

### Global Optimization of Polynomials

Semidefinite Programming Lecture 9 OR 637 Spring 2008 April 9, 2008 Scribe: Dennis Leventhal Global Optimization of Polynomials Recall we were considering the problem min z R n p(z) where p(z) is a degree

### Lecture 1. 1 Conic programming. MA 796S: Convex Optimization and Interior Point Methods October 8, Consider the conic program. min.

MA 796S: Convex Optimization and Interior Point Methods October 8, 2007 Lecture 1 Lecturer: Kartik Sivaramakrishnan Scribe: Kartik Sivaramakrishnan 1 Conic programming Consider the conic program min s.t.

### LECTURE 10 LECTURE OUTLINE

LECTURE 10 LECTURE OUTLINE Min Common/Max Crossing Th. III Nonlinear Farkas Lemma/Linear Constraints Linear Programming Duality Convex Programming Duality Optimality Conditions Reading: Sections 4.5, 5.1,5.2,

### Convex Optimization Boyd & Vandenberghe. 5. Duality

5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Linear and non-linear programming

Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)

### Semidefinite Programming Basics and Applications

Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent

### BBM402-Lecture 20: LP Duality

BBM402-Lecture 20: LP Duality Lecturer: Lale Özkahya Resources for the presentation: https://courses.engr.illinois.edu/cs473/fa2016/lectures.html An easy LP? which is compact form for max cx subject to

### Lagrangian Duality Theory

Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual

### 12. Interior-point methods

12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

### Convex Optimization and Modeling

Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

### Strong duality in Lasserre s hierarchy for polynomial optimization

Strong duality in Lasserre s hierarchy for polynomial optimization arxiv:1405.7334v1 [math.oc] 28 May 2014 Cédric Josz 1,2, Didier Henrion 3,4,5 Draft of January 24, 2018 Abstract A polynomial optimization

### Chapter 1. Preliminaries

Introduction This dissertation is a reading of chapter 4 in part I of the book : Integer and Combinatorial Optimization by George L. Nemhauser & Laurence A. Wolsey. The chapter elaborates links between

### Summer School: Semidefinite Optimization

Summer School: Semidefinite Optimization Christine Bachoc Université Bordeaux I, IMB Research Training Group Experimental and Constructive Algebra Haus Karrenberg, Sept. 3 - Sept. 7, 2012 Duality Theory

### Convex Optimization and Support Vector Machine

Convex Optimization and Support Vector Machine Problem 0. Consider a two-class classification problem. The training data is L n = {(x 1, t 1 ),..., (x n, t n )}, where each t i { 1, 1} and x i R p. We

### Assignment 1: From the Definition of Convexity to Helley Theorem

Assignment 1: From the Definition of Convexity to Helley Theorem Exercise 1 Mark in the following list the sets which are convex: 1. {x R 2 : x 1 + i 2 x 2 1, i = 1,..., 10} 2. {x R 2 : x 2 1 + 2ix 1x

### Duality. Geoff Gordon & Ryan Tibshirani Optimization /

Duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider

### 1 Review of last lecture and introduction

Semidefinite Programming Lecture 10 OR 637 Spring 2008 April 16, 2008 (Wednesday) Instructor: Michael Jeremy Todd Scribe: Yogeshwer (Yogi) Sharma 1 Review of last lecture and introduction Let us first

### Lecture 8. Strong Duality Results. September 22, 2008

Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation

### E5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming

E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program

### A ten page introduction to conic optimization

CHAPTER 1 A ten page introduction to conic optimization This background chapter gives an introduction to conic optimization. We do not give proofs, but focus on important (for this thesis) tools and concepts.

### Duality of LPs and Applications

Lecture 6 Duality of LPs and Applications Last lecture we introduced duality of linear programs. We saw how to form duals, and proved both the weak and strong duality theorems. In this lecture we will

### Lecture: Duality.

Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong

### Today: Linear Programming (con t.)

Today: Linear Programming (con t.) COSC 581, Algorithms April 10, 2014 Many of these slides are adapted from several online sources Reading Assignments Today s class: Chapter 29.4 Reading assignment for

### CO 250 Final Exam Guide

Spring 2017 CO 250 Final Exam Guide TABLE OF CONTENTS richardwu.ca CO 250 Final Exam Guide Introduction to Optimization Kanstantsin Pashkovich Spring 2017 University of Waterloo Last Revision: March 4,

### Lecture 18: Optimization Programming

Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming

### Linear and Combinatorial Optimization

Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality

### Enhanced Fritz John Optimality Conditions and Sensitivity Analysis

Enhanced Fritz John Optimality Conditions and Sensitivity Analysis Dimitri P. Bertsekas Laboratory for Information and Decision Systems Massachusetts Institute of Technology March 2016 1 / 27 Constrained

### Duality in Linear Programs. Lecturer: Ryan Tibshirani Convex Optimization /36-725

Duality in Linear Programs Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: proximal gradient descent Consider the problem x g(x) + h(x) with g, h convex, g differentiable, and

### Lecture Note 18: Duality

MATH 5330: Computational Methods of Linear Algebra 1 The Dual Problems Lecture Note 18: Duality Xianyi Zeng Department of Mathematical Sciences, UTEP The concept duality, just like accuracy and stability,

### LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE

LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization

### Semidefinite Programming

Semidefinite Programming Notes by Bernd Sturmfels for the lecture on June 26, 208, in the IMPRS Ringvorlesung Introduction to Nonlinear Algebra The transition from linear algebra to nonlinear algebra has

### Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 5

Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 5 Instructor: Farid Alizadeh Scribe: Anton Riabov 10/08/2001 1 Overview We continue studying the maximum eigenvalue SDP, and generalize

### Nonlinear Programming 3rd Edition. Theoretical Solutions Manual Chapter 6

Nonlinear Programming 3rd Edition Theoretical Solutions Manual Chapter 6 Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts 1 NOTE This manual contains

### Lecture 7: Semidefinite programming

CS 766/QIC 820 Theory of Quantum Information (Fall 2011) Lecture 7: Semidefinite programming This lecture is on semidefinite programming, which is a powerful technique from both an analytic and computational

### POLYNOMIAL OPTIMIZATION WITH SUMS-OF-SQUARES INTERPOLANTS

POLYNOMIAL OPTIMIZATION WITH SUMS-OF-SQUARES INTERPOLANTS Sercan Yıldız syildiz@samsi.info in collaboration with Dávid Papp (NCSU) OPT Transition Workshop May 02, 2017 OUTLINE Polynomial optimization and

### EE/AA 578, Univ of Washington, Fall Duality

7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

### Convex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version

Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com

### Introduction to Semidefinite Programming I: Basic properties a

Introduction to Semidefinite Programming I: Basic properties and variations on the Goemans-Williamson approximation algorithm for max-cut MFO seminar on Semidefinite Programming May 30, 2010 Semidefinite

### A new look at nonnegativity on closed sets

A new look at nonnegativity on closed sets LAAS-CNRS and Institute of Mathematics, Toulouse, France IPAM, UCLA September 2010 Positivstellensatze for semi-algebraic sets K R n from the knowledge of defining

### HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.

HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

### Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities

Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form

### UC Berkeley Department of Electrical Engineering and Computer Science. EECS 227A Nonlinear and Convex Optimization. Solutions 5 Fall 2009

UC Berkeley Department of Electrical Engineering and Computer Science EECS 227A Nonlinear and Convex Optimization Solutions 5 Fall 2009 Reading: Boyd and Vandenberghe, Chapter 5 Solution 5.1 Note that

### Problem 1 (Exercise 2.2, Monograph)

MS&E 314/CME 336 Assignment 2 Conic Linear Programming January 3, 215 Prof. Yinyu Ye 6 Pages ASSIGNMENT 2 SOLUTIONS Problem 1 (Exercise 2.2, Monograph) We prove the part ii) of Theorem 2.1 (Farkas Lemma

### CS-E4830 Kernel Methods in Machine Learning

CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This

### Copositive Programming and Combinatorial Optimization

Copositive Programming and Combinatorial Optimization Franz Rendl http://www.math.uni-klu.ac.at Alpen-Adria-Universität Klagenfurt Austria joint work with I.M. Bomze (Wien) and F. Jarre (Düsseldorf) IMA

### Lecture 6: Conic Optimization September 8

IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

### Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation

Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu

### CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008.

1 ECONOMICS 594: LECTURE NOTES CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS W. Erwin Diewert January 31, 2008. 1. Introduction Many economic problems have the following structure: (i) a linear function

### Lecture: Duality of LP, SOCP and SDP

1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:

### Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:

CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

### Lagrangian Duality and Convex Optimization

Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?

### CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS

CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS A Dissertation Submitted For The Award of the Degree of Master of Philosophy in Mathematics Neelam Patel School of Mathematics

### CSC Linear Programming and Combinatorial Optimization Lecture 10: Semidefinite Programming

CSC2411 - Linear Programming and Combinatorial Optimization Lecture 10: Semidefinite Programming Notes taken by Mike Jamieson March 28, 2005 Summary: In this lecture, we introduce semidefinite programming

### On duality gap in linear conic problems

On duality gap in linear conic problems C. Zălinescu Abstract In their paper Duality of linear conic problems A. Shapiro and A. Nemirovski considered two possible properties (A) and (B) for dual linear

### Chapter 1 Linear Programming. Paragraph 5 Duality

Chapter 1 Linear Programming Paragraph 5 Duality What we did so far We developed the 2-Phase Simplex Algorithm: Hop (reasonably) from basic solution (bs) to bs until you find a basic feasible solution