Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Convexification of Mixed-Integer Quadratically Constrained Quadratic Programs Laura Galli 1 Adam N. Letchford 2 Lancaster, April DEIS, University of Bologna, Italy 2 Department of Management Science, Lancaster University, United Kingdom

2 Outline Literature Review Motivation Relaxations of QCQPs Quadratic Convex Reformulation (QCR) Convexification of 0-1 QCQP Reformulation with equations only Reformulation with inequalities Convexification of MIQCQP General integer variables Continuous variables Conclusions

3 Motivation It is well known that Semidefinite Programming (SDP) can be used to form useful relaxations for a variety of N P-hard optimisation problems, such as: zero-one linear programs (0-1 LP) zero-one quadratic programs (0-1 QP) non-convex quadratically constrained quadratic programs (QCQP) general polynomial programs

4 Motivation (cont.) Recently, Billionnet et al. (2009, 2010) proposed to use SDP to reformulate 0-1 QPs, not just to relax them. The idea is to perturb the objective function in such a way that it is convex and the continuous relaxation of the 0-1 QP is tighter. We examine to what extent this approach can be applied to the (much) more general case of mixed-integer quadratically constrained quadratic programs (MIQCQP).

5 Semidefinite relaxation of QCQP We start with Shor s (1987) relaxation of non-convex QCQP (which is an N P-hard global optimisation problem). An instance takes the form: inf x T Q 0 x +c 0 x s.t. x T Q j x +c j x b j x R n, (j = 1,...,m) where the Q j are symmetric, square and rational matrices of order n, the c j are rational n-vectors, and the b j are rational scalars.

6 Semidefinite relaxation of QCQP (cont.) The basic SDP relaxation of the QCQP is derived as follows: define the n n symmetric matrix X = xx T replace the terms x T Q j x with Q j X define the augmented symmetric matrix ( )( ) T ( ) x T Y = = x x x X note that Y is positive semidefinite (psd).

7 Semidefinite relaxation of QCQP (cont.) The resulting SDP is: inf Q 0 X +c 0 x s.t. Q j X +c j x b j Y 0. (j = 1,...,m) (The last constraint imposes psd-ness on Y.)

8 Lagrangian relaxation of QCQP As noted by Fujie & Kojima (1997), Lémaréchal & Oustry (2001) and others, there is a connection between semidefinite relaxation and Lagrangian relaxation. Suppose we relax all of the constraint in Lagrangian fashion, and let λ denote the vector of Lagrangian multipliers. The Lagrangian objective becomes: ( ) m m m x (Q T 0 + λ j Q )x j + c 0 + λ j c j x λ j b j. j=1 j=1 j=1

9 Lagrangian relaxation of QCQP (cont.) Let the Lagrangian objective be denoted by f(x,λ). The relaxed problem is: inf { f(x,λ) : x R n }, which is an unconstrained quadratic minimisation problem. The Lagrangian dual is: sup inf x R nf(x,λ). λ R m +

10 Lagrangian relaxation of QCQP (cont.) Suppose that strong duality holds for the SDP. Then one can show that: the value of the Lagrangian dual equals the SDP bound, and the optimal Lagrangian multiplier vector λ is nothing but the optimal dual vector for the SDP.

11 QCQP summary We have: opt SDP bound dual SDP bound = Lagrangian bound and the second inequality is an equality under suitable constraint qualifications.

12 0-1 QP Now consider a 0-1 QP of the form: min x T Qx +c T x s.t. Ax = b Dx f x {0,1} n. As observed by many authors, the condition that x be binary is equivalent to requiring x i xi 2 = 0 (i = 1,...,n).

13 SDP relaxation of 0-1 QP This leads immediately to the following semidefinite relaxation: min Q X +c T x s.t. Ax = b Dx f x = diag(x) Y 0.

14 Convex reformulation of 0-1 QP Reformulating a 0-1 QP instance means perturbing the objective function in such a way that: the objective function is convex the cost of each feasible solution is unchanged the constraints remain unchanged, so that the feasible set is unchanged.

15 Minimum eigenvalue reformulation of 0-1 QP A simple reformulation technique was given by Hammer & Rubin (1970). They add 1 i n M(x2 i x i ) to the objective function, where M > 0 is large enough to ensure that the matrix Q +MI of quadratic coefficients is psd. In this way, they ensure that the reformulated instance is convex. (A suitable value for M is the minimum eigenvalue of Q, multiplied by 1.)

16 Quadratic Convex Reformulation (QCR) Billionnet et al. (2009) develop this idea further, in two ways: they perturb the objective function by addinggeneral terms of the form λ i (x i x 2 i ) and µ jk (a j x b j )x k. instead of merely ensuring that the reformulated instance is convex, they also ensure that its continuous relaxation is as tight as possible.

17 Quadratic Convex Reformulation (QCR) (cont.) Billionnet et al. (2009) show that the optimal multipliers λ and µ can be found by solving an SDP. They call this approach convex quadratic reformulation or QCR. Once QCR has been applied, one can feed the reformulated instance into any convex MIQP solver. (An extension of QCR to general MIQP was given recently by Billionnet et al., 2010.)

18 0-1 QCQP Our first goal is to extend the QCR method to 0-1 QCQP. We suppose the instance is written as: min x T Q 0 x +c 0 x s.t. x T Q j x +c j x = b j (j N 1) x T Q j x +c j x b j (j N 2) x {0,1} n. (We assume for simplicity of notation that any linear constraints, along with any additional valid inequalities, are subsumed in the above system.)

19 SDP relaxation of 0-1 QCQP The semidefinite relaxation is immediate: min Q 0 X +c 0 x s.t. Q j X +c j x = b j (j N 1) Q j X +c j x b j (j N 2) x = diag(x) Y 0. The Lagrangian relaxation is easy to write as well.

20 Convex reformulation of 0-1 QCQP By analogy with the original QCR method, we seek to transform a 0-1 QCQP instance into another 0-1 QCQP instance, that has the following four properties: the objective and constraint functions are convex the set of feasible solutions is identical to that of the original instance the cost of each feasible solution is the same as it was in the original instance...

21 Convex reformulation of 0-1 QCQP By analogy with the original QCR method, we seek to transform a 0-1 QCQP instance into another 0-1 QCQP instance, that has the following four properties: the objective and constraint functions are convex the set of feasible solutions is identical to that of the original instance the cost of each feasible solution is the same as it was in the original instance and the lower bound from the continuous relaxation is equal to the semidefinite/lagrangian bound for the original instance.

22 0-1 QCQP with Quadratic Equations For the case in which all constraints are equations (i.e., N 2 = ), we propose the following scheme: 1 Solve the SDP, obtaining optimal dual vectors λ R N 1 and ν R n ; 2 Perturb the objective function by adding terms of the form x T Diag(ν )x (ν ) T x and λ j (xt Q j x +c j x b j ) for j N 1 ; 3 Replace each quadratic equation with two quadratic inequalities; 4 Convexify the quadratic inequalities arbitrarily (e.g., using the minimum eigenvalue method).

23 0-1 QCQP with Quadratic Equations (cont.) Theorem When the above scheme is applied to a 0-1 QCQP instance with N 2 =, the reformulated instance is convex. Moreover, if the primal SDP satisfies the Slater condition, then the lower bound from the continuous relaxation of the reformulated instance will equal the SDP bound.

24 Handling Quadratic Inequalities: two difficulties The case of quadratic inequalities is more tricky because of two serious difficulties! The first can be shown considering the following simple instance: max x 1 + +x n s.t. x i x j 0 (1 i < j n) x 2 i x i = 0 (i = 1,...,n). The optimum is 1 and one can check that the SDP bound is also 1.

25 Handling Quadratic Inequalities: two difficulties (cont.) The quadratic inequalities are non-convex, so we must use the equations x 2 i x i = 0 to convexify them. It turns out that the best we can do is to replace the inequalities with: 1 2 (x2 i x i )+ 1 2 (x2 j x j )+x i x j 0 (1 i < j n). But the optimal solution to the continuous relaxation of the reformulated instance is then (1/2,...,1/2) T, yielding a very poor upper bound of n/2.

26 Handling Quadratic Inequalities: two difficulties (cont.) In other words, when quadratic inequalities are present: opt SDP bound = dual SDP bound = Lagrangian bound LP bound from reformulated instance and the last inequality is usually strict! An intuitive explanation for this phenomenon is that quadratic inequalities cannot be used to perturb the objective function.

27 Handling Quadratic Inequalities: two difficulties (cont.) The second difficulty is that finding an optimal reformulation is itself a rather complex optimisation problem. It is possible to formulate the problem of finding the best such reformulation as one huge SDP. Unfortunaley, the size and complexity of this SDP grows rapidly as the number of inequalities increases.

28 Handling Quadratic Inequalities: possible remedies If we are willing to convert the 0-1 QCQP into a mixed 0-1 QCQP, then we can do better. We can add continuous slack variables to convert every quadratic inequality into a quadratic equation. Once this is done, it is always possible to reformulate the mixed 0-1 QCQP instance in such a way that the bound from the continuous relaxation is equal to the Lagrangian bound. As before, if the primal SDP satisfies the Slater condition, this bound will be equal to the SDP bound.

29 Handling Quadratic Inequalities: possible remedies (cont.) The above scheme has one disadvantage: we have to add N 2 continuous variables. This is unattractive when N 2 is large. We have another variant in which only one continuous variable needs to be added. (The basic idea is to use the dual SDP solution to construct a surrogate quadratic inequality that contains all of the information we need to get a good bound, and then assign a slack variable only to the surrogate constraint.)

30 General-Integer Variables Next, we move on to the case of general-integer variables. A first observation is that integer variables present no problem for the semidefinite and Lagrangian relaxations. As for reformulation, note that we cannot use terms of the form x 2 i x i to perturb the objective or constraints coefficient of integer variables. This can prevent the existence of a convex reformulation.

31 General-Integer Variables (cont.) To get around this, we can use binary expansion. In fact, if such a variable is nonnegative and bounded from above by 2 p for some positive integer p, then it can be replaced by p binary variables. Then, if all variables are bounded in this way, we can reduce the problem to 0-1 QCQP. We show that this transformation cannot make the SDP bound any worse, and in fact in some cases it can make it better!

32 General-Integer Variables (cont.) Consider the following instance with just one integer variable: } min {x1 2 3x 1 : x 1 3, x 1 Z +. There are two optimal solutions of cost 2, obtained by setting x 1 to either 1 or 2. The basic SDP relaxation is: min{x 11 3x 1 : 0 x 1 3, Y 0}. The optimal solution to this relaxation is to set x 1 to 1.5 and X 11 to 2.25, giving a lower bound of 2.25.

33 General-Integer Variables (cont.) Now, applying binary expansion to this instance, we use the substitution x 1 = x x 2 1, where x 1 1 and x 2 1 are new binary variables. The SDP relaxation for the transformed instance is: min X 1, X 2,2 11 s.t. x 1 1 = X 1,1 11 x 1 2 2,2 = X 11 Ỹ X 1, x1 1 6 x 2 1 One can check with an SDP solver that all optimal solutions to this SDP have cost 2. So, the transformed SDP yields an improved lower bound.

34 General-Integer Variables (cont.) If, on the other hand, the general-integer variables are not bounded from above a priori, then we run into a serious problem: Theorem (Jeroslow 1973) Testing feasibility of an integer program with quadratic constraints is undecidable. So we cannot expect an efficient convexification procedure in general. (Of course, in practical applications, variables can usually be bounded quite easily.)

35 Continuous Variables Finally, we consider the case in which continuous variables are permitted. In the context of the semidefinite and Lagrangian approaches, continuous variables can be handled in exactly the same way as integer variables, and therefore they present no problems. When it comes to reformulation, on the other hand, continuous variables cause serious difficulties. (Indeed, a convex reformulation may not exist in general, and we cannot get around this issue by binarisation, as we do for integer variables.)

36 Continuous Variables (cont.) Here, three outcomes are possible: The instance can be convexified, and the bound from the continuous relaxation is as good as the SDP bound. The instance can be convexified, but the bound from the continuous relaxation is worse than the SDP bound. The instance cannot be convexified at all. We have some necessary and sufficient conditions for these three cases to hold.

37 Conclusions Basically, we have three possible situations: When there are no quadratic inequalities and no continuous variables, and all general-integer variables are bounded, we can always obtain a convex reformulation that gives a strong bound. When quadratic inequalities are present, this may not be possible, unless we are prepared to add slack variables. When continuous variables are present, a convex reformulation is not even guaranteed to exist.

### 14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity

Nonconvex Quadratic Programming: Return of the Boolean Quadric Polytope Kurt M. Anstreicher Dept. of Management Sciences University of Iowa Seminar, Chinese University of Hong Kong, October 2009 We consider

### Lecture 6: Conic Optimization September 8

IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

### Lecture 8. Strong Duality Results. September 22, 2008

Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation

### Cutting Planes for First Level RLT Relaxations of Mixed 0-1 Programs

Cutting Planes for First Level RLT Relaxations of Mixed 0-1 Programs 1 Cambridge, July 2013 1 Joint work with Franklin Djeumou Fomeni and Adam N. Letchford Outline 1. Introduction 2. Literature Review

### Lecture: Introduction to LP, SDP and SOCP

Lecture: Introduction to LP, SDP and SOCP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2015.html wenzw@pku.edu.cn Acknowledgement:

### Module 04 Optimization Problems KKT Conditions & Solvers

Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September

### Relaxations and Randomized Methods for Nonconvex QCQPs

Relaxations and Randomized Methods for Nonconvex QCQPs Alexandre d Aspremont, Stephen Boyd EE392o, Stanford University Autumn, 2003 Introduction While some special classes of nonconvex problems can be

### Lecture: Cone programming. Approximating the Lorentz cone.

Strong relaxations for discrete optimization problems 10/05/16 Lecture: Cone programming. Approximating the Lorentz cone. Lecturer: Yuri Faenza Scribes: Igor Malinović 1 Introduction Cone programming is

### Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:

CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

### 10 Numerical methods for constrained problems

10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside

### Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

### EE 227A: Convex Optimization and Applications October 14, 2008

EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider

### Tutorial on Convex Optimization for Engineers Part II

Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

### ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications

ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March

### Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

### HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.

HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

### Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming

School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio

### Lecture 1. 1 Conic programming. MA 796S: Convex Optimization and Interior Point Methods October 8, Consider the conic program. min.

MA 796S: Convex Optimization and Interior Point Methods October 8, 2007 Lecture 1 Lecturer: Kartik Sivaramakrishnan Scribe: Kartik Sivaramakrishnan 1 Conic programming Consider the conic program min s.t.

### Copositive Programming and Combinatorial Optimization

Copositive Programming and Combinatorial Optimization Franz Rendl http://www.math.uni-klu.ac.at Alpen-Adria-Universität Klagenfurt Austria joint work with I.M. Bomze (Wien) and F. Jarre (Düsseldorf) IMA

### Lecture #21. c T x Ax b. maximize subject to

COMPSCI 330: Design and Analysis of Algorithms 11/11/2014 Lecture #21 Lecturer: Debmalya Panigrahi Scribe: Samuel Haney 1 Overview In this lecture, we discuss linear programming. We first show that the

CHAPTER 2: QUADRATIC PROGRAMMING Overview Quadratic programming (QP) problems are characterized by objective functions that are quadratic in the design variables, and linear constraints. In this sense,

### Semidefinite Programming

Semidefinite Programming Basics and SOS Fernando Mário de Oliveira Filho Campos do Jordão, 2 November 23 Available at: www.ime.usp.br/~fmario under talks Conic programming V is a real vector space h, i

### Solving Dual Problems

Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem

### Cutting Planes for RLT Relaxations of Mixed 0-1 Polynomial Programs

Cutting Planes for RLT Relaxations of Mixed 0-1 Polynomial Programs Franklin Djeumou Fomeni Konstantinos Kaparis Adam N. Letchford To Appear in Mathematical Programming Abstract The Reformulation-Linearization

### MIT Algebraic techniques and semidefinite optimization February 14, Lecture 3

MI 6.97 Algebraic techniques and semidefinite optimization February 4, 6 Lecture 3 Lecturer: Pablo A. Parrilo Scribe: Pablo A. Parrilo In this lecture, we will discuss one of the most important applications

### LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE

LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization

### 16.1 L.P. Duality Applied to the Minimax Theorem

CS787: Advanced Algorithms Scribe: David Malec and Xiaoyong Chai Lecturer: Shuchi Chawla Topic: Minimax Theorem and Semi-Definite Programming Date: October 22 2007 In this lecture, we first conclude our

### Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725

Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =

### Chap 2. Optimality conditions

Chap 2. Optimality conditions Version: 29-09-2012 2.1 Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1

### Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization. Nick Gould (RAL)

Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization Nick Gould (RAL) x IR n f(x) subject to c(x) = Part C course on continuoue optimization CONSTRAINED MINIMIZATION x

### On the Method of Lagrange Multipliers

On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should

### 15. Conic optimization

L. Vandenberghe EE236C (Spring 216) 15. Conic optimization conic linear program examples modeling duality 15-1 Generalized (conic) inequalities Conic inequality: a constraint x K where K is a convex cone

### Cutting Planes for RLT Relaxations of Mixed 0-1 Polynomial Programs

Cutting Planes for RLT Relaxations of Mixed 0-1 Polynomial Programs Franklin Djeumou Fomeni Konstantinos Kaparis Adam N. Letchford December 2013 Abstract The Reformulation-Linearization Technique (RLT),

### Primal-Dual Interior-Point Methods

Primal-Dual Interior-Point Methods Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 Outline Today: Primal-dual interior-point method Special case: linear programming

### On Conic QPCCs, Conic QCQPs and Completely Positive Programs

Noname manuscript No. (will be inserted by the editor) On Conic QPCCs, Conic QCQPs and Completely Positive Programs Lijie Bai John E.Mitchell Jong-Shi Pang July 28, 2015 Received: date / Accepted: date

### Support Vector Machines

EE 17/7AT: Optimization Models in Engineering Section 11/1 - April 014 Support Vector Machines Lecturer: Arturo Fernandez Scribe: Arturo Fernandez 1 Support Vector Machines Revisited 1.1 Strictly) Separable

### The Strong Duality Theorem 1

1/39 The Strong Duality Theorem 1 Adrian Vetta 1 This presentation is based upon the book Linear Programming by Vasek Chvatal 2/39 Part I Weak Duality 3/39 Primal and Dual Recall we have a primal linear

### Optimization Theory. Lectures 4-6

Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x\$0} Existence of a maximum:

### Semidefinite Programming, Combinatorial Optimization and Real Algebraic Geometry

Semidefinite Programming, Combinatorial Optimization and Real Algebraic Geometry assoc. prof., Ph.D. 1 1 UNM - Faculty of information studies Edinburgh, 16. September 2014 Outline Introduction Definition

### Lecture: Convex Optimization Problems

1/36 Lecture: Convex Optimization Problems http://bicmr.pku.edu.cn/~wenzw/opt-2015-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/36 optimization

### Convex Relaxations of Non-Convex Mixed Integer Quadratically Constrained Programs: Projected Formulations

Anureet Saxena Pierre Bonami Jon Lee Convex Relaxations of Non-Convex Mixed Integer Quadratically Constrained Programs: Projected Formulations November 19, 008 Abstract A common way to produce a convex

### ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications

ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications Professor M. Chiang Electrical Engineering Department, Princeton University February

### You should be able to...

Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set

### Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

### Linear Programming Versus Convex Quadratic Programming for the Module Allocation Problem

Linear Programming Versus Convex Quadratic Programming for the Module Allocation Problem Sourour Elloumi 1 April 13, 2005 Abstract We consider the Module Allocation Problem with Non-Uniform communication

### Kernel Methods. Machine Learning A W VO

Kernel Methods Machine Learning A 708.063 07W VO Outline 1. Dual representation 2. The kernel concept 3. Properties of kernels 4. Examples of kernel machines Kernel PCA Support vector regression (Relevance

### The moment-lp and moment-sos approaches in optimization

The moment-lp and moment-sos approaches in optimization LAAS-CNRS and Institute of Mathematics, Toulouse, France Workshop Linear Matrix Inequalities, Semidefinite Programming and Quantum Information Theory

### Copositive Programming and Combinatorial Optimization

Copositive Programming and Combinatorial Optimization Franz Rendl http://www.math.uni-klu.ac.at Alpen-Adria-Universität Klagenfurt Austria joint work with M. Bomze (Wien) and F. Jarre (Düsseldorf) and

### Second Order Cone Programming Relaxation of Nonconvex Quadratic Optimization Problems

Second Order Cone Programming Relaxation of Nonconvex Quadratic Optimization Problems Sunyoung Kim Department of Mathematics, Ewha Women s University 11-1 Dahyun-dong, Sudaemoon-gu, Seoul 120-750 Korea

### - Well-characterized problems, min-max relations, approximate certificates. - LP problems in the standard form, primal and dual linear programs

LP-Duality ( Approximation Algorithms by V. Vazirani, Chapter 12) - Well-characterized problems, min-max relations, approximate certificates - LP problems in the standard form, primal and dual linear programs

### Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 5

Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 5 Instructor: Farid Alizadeh Scribe: Anton Riabov 10/08/2001 1 Overview We continue studying the maximum eigenvalue SDP, and generalize

### LP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra

LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality

### The moment-lp and moment-sos approaches

The moment-lp and moment-sos approaches LAAS-CNRS and Institute of Mathematics, Toulouse, France CIRM, November 2013 Semidefinite Programming Why polynomial optimization? LP- and SDP- CERTIFICATES of POSITIVITY

### Lecture Support Vector Machine (SVM) Classifiers

Introduction to Machine Learning Lecturer: Amir Globerson Lecture 6 Fall Semester Scribe: Yishay Mansour 6.1 Support Vector Machine (SVM) Classifiers Classification is one of the most important tasks in

### 4. Convex optimization problems

Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization

### U.C. Berkeley CS294: Beyond Worst-Case Analysis Handout 12 Luca Trevisan October 3, 2017

U.C. Berkeley CS94: Beyond Worst-Case Analysis Handout 1 Luca Trevisan October 3, 017 Scribed by Maxim Rabinovich Lecture 1 In which we begin to prove that the SDP relaxation exactly recovers communities

### FINDING LOW-RANK SOLUTIONS OF SPARSE LINEAR MATRIX INEQUALITIES USING CONVEX OPTIMIZATION

FINDING LOW-RANK SOLUTIONS OF SPARSE LINEAR MATRIX INEQUALITIES USING CONVEX OPTIMIZATION RAMTIN MADANI, GHAZAL FAZELNIA, SOMAYEH SOJOUDI AND JAVAD LAVAEI Abstract. This paper is concerned with the problem

### Constrained optimization: direct methods (cont.)

Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a

### Optimization for Machine Learning

Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html

### Structured Problems and Algorithms

Integer and quadratic optimization problems Dept. of Engg. and Comp. Sci., Univ. of Cal., Davis Aug. 13, 2010 Table of contents Outline 1 2 3 Benefits of Structured Problems Optimization problems may become

### Lecture 13: Constrained optimization

2010-12-03 Basic ideas A nonlinearly constrained problem must somehow be converted relaxed into a problem which we can solve (a linear/quadratic or unconstrained problem) We solve a sequence of such problems

### Convex Optimization and Modeling

Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

### A CONIC DANTZIG-WOLFE DECOMPOSITION APPROACH FOR LARGE SCALE SEMIDEFINITE PROGRAMMING

A CONIC DANTZIG-WOLFE DECOMPOSITION APPROACH FOR LARGE SCALE SEMIDEFINITE PROGRAMMING Kartik Krishnan Advanced Optimization Laboratory McMaster University Joint work with Gema Plaza Martinez and Tamás

### A Tutorial on Convex Optimization II: Duality and Interior Point Methods

A Tutorial on Convex Optimization II: Duality and Interior Point Methods Haitham Hindi Palo Alto Research Center (PARC), Palo Alto, California 94304 email: hhindi@parc.com Abstract In recent years, convex

### The Ongoing Development of CSDP

The Ongoing Development of CSDP Brian Borchers Department of Mathematics New Mexico Tech Socorro, NM 87801 borchers@nmt.edu Joseph Young Department of Mathematics New Mexico Tech (Now at Rice University)

### Convex Optimization Overview (cnt d)

Conve Optimization Overview (cnt d) Chuong B. Do November 29, 2009 During last week s section, we began our study of conve optimization, the study of mathematical optimization problems of the form, minimize

### Introduction to Semidefinite Programming (SDP)

Introduction to Semidefinite Programming (SDP) Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. 1 1 Introduction Semidefinite programming (SDP ) is the most exciting development

### 3E4: Modelling Choice. Introduction to nonlinear programming. Announcements

3E4: Modelling Choice Lecture 7 Introduction to nonlinear programming 1 Announcements Solutions to Lecture 4-6 Homework will be available from http://www.eng.cam.ac.uk/~dr241/3e4 Looking ahead to Lecture

### SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING

Nf SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING f(x R m g HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 5, DR RAPHAEL

### 13. Convex programming

CS/ISyE/ECE 524 Introduction to Optimization Spring 2015 16 13. Convex programming Convex sets and functions Convex programs Hierarchy of complexity Example: geometric programming Laurent Lessard (www.laurentlessard.com)

### A Review of Linear Programming

A Review of Linear Programming Instructor: Farid Alizadeh IEOR 4600y Spring 2001 February 14, 2001 1 Overview In this note we review the basic properties of linear programming including the primal simplex

### arxiv:math/ v1 [math.co] 23 May 2000

Some Fundamental Properties of Successive Convex Relaxation Methods on LCP and Related Problems arxiv:math/0005229v1 [math.co] 23 May 2000 Masakazu Kojima Department of Mathematical and Computing Sciences

### Conic Linear Programming. Yinyu Ye

Conic Linear Programming Yinyu Ye December 2004, revised October 2017 i ii Preface This monograph is developed for MS&E 314, Conic Linear Programming, which I am teaching at Stanford. Information, lecture

### Sum of Squares Relaxations for Polynomial Semi-definite Programming

Sum of Squares Relaxations for Polynomial Semi-definite Programming C.W.J. Hol, C.W. Scherer Delft University of Technology, Delft Center of Systems and Control (DCSC) Mekelweg 2, 2628CD Delft, The Netherlands

Convex Quadratic Relaxations of Nonconvex Quadratically Constrained Quadratic Progams John E. Mitchell, Jong-Shi Pang, and Bin Yu Original: June 10, 2011 Abstract Nonconvex quadratic constraints can be

### Bilevel Integer Linear Programming

Bilevel Integer Linear Programming TED RALPHS SCOTT DENEGRE ISE Department COR@L Lab Lehigh University ted@lehigh.edu MOPTA 2009, Lehigh University, 19 August 2009 Thanks: Work supported in part by the

### Optimization in Wireless Communication

Zhi-Quan (Tom) Luo Department of Electrical and Computer Engineering University of Minnesota 200 Union Street SE Minneapolis, MN 55455 2007 NSF Workshop Challenges Optimization problems from wireless applications

### Sparse Optimization Lecture: Basic Sparse Optimization Models

Sparse Optimization Lecture: Basic Sparse Optimization Models Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know basic l 1, l 2,1, and nuclear-norm

### CE 191: Civil and Environmental Engineering Systems Analysis. LEC 05 : Optimality Conditions

CE 191: Civil and Environmental Engineering Systems Analysis LEC : Optimality Conditions Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 214 Prof. Moura

### Support Vector Machines for Classification and Regression. 1 Linearly Separable Data: Hard Margin SVMs

E0 270 Machine Learning Lecture 5 (Jan 22, 203) Support Vector Machines for Classification and Regression Lecturer: Shivani Agarwal Disclaimer: These notes are a brief summary of the topics covered in

### Duality Theory, Optimality Conditions

5.1 Duality Theory, Optimality Conditions Katta G. Murty, IOE 510, LP, U. Of Michigan, Ann Arbor We only consider single objective LPs here. Concept of duality not defined for multiobjective LPs. Every

### Solving linear and non-linear SDP by PENNON

Solving linear and non-linear SDP by PENNON Michal Kočvara School of Mathematics, The University of Birmingham University of Warwick, February 2010 Outline Why nonlinear SDP? PENNON the new generation

### The Fixed Charge Transportation Problem: A Strong Formulation Based On Lagrangian Decomposition and Column Generation

The Fixed Charge Transportation Problem: A Strong Formulation Based On Lagrangian Decomposition and Column Generation Yixin Zhao, Torbjörn Larsson and Department of Mathematics, Linköping University, Sweden

### Solution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark

Solution Methods Richard Lusby Department of Management Engineering Technical University of Denmark Lecture Overview (jg Unconstrained Several Variables Quadratic Programming Separable Programming SUMT

### Region of attraction approximations for polynomial dynamical systems

Region of attraction approximations for polynomial dynamical systems Milan Korda EPFL Lausanne Didier Henrion LAAS-CNRS Toulouse & CTU Prague Colin N. Jones EPFL Lausanne Region of Attraction (ROA) ẋ(t)

### DUALITY, OPTIMALITY CONDITIONS AND PERTURBATION ANALYSIS

1 DUALITY, OPTIMALITY CONDITIONS AND PERTURBATION ANALYSIS Alexander Shapiro 1 School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332-0205, USA, E-mail: ashapiro@isye.gatech.edu

### Lecture 5. x 1,x 2,x 3 0 (1)

Computational Intractability Revised 2011/6/6 Lecture 5 Professor: David Avis Scribe:Ma Jiangbo, Atsuki Nagao 1 Duality The purpose of this lecture is to introduce duality, which is an important concept

### 4. Convex optimization problems (part 1: general)

EE/AA 578, Univ of Washington, Fall 2016 4. Convex optimization problems (part 1: general) optimization problem in standard form convex optimization problems quasiconvex optimization 4 1 Optimization problem

### 1 The independent set problem

ORF 523 Lecture 11 Spring 2016, Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Tuesday, March 29, 2016 When in doubt on the accuracy of these notes, please cross chec with the instructor

### Lecture 7: Weak Duality

EE 227A: Conve Optimization and Applications February 7, 2012 Lecture 7: Weak Duality Lecturer: Laurent El Ghaoui 7.1 Lagrange Dual problem 7.1.1 Primal problem In this section, we consider a possibly

### Robust Sensitivity Analysis of the Optimal Value of Linear Programming

Robust Sensitivity Analysis of the Optimal Value of Linear Programming Guanglin Xu Samuel Burer arxiv:1509.04682v2 [math.oc] 7 Nov 2015 September 14, 2015 Revised: November 4, 2015 Abstract We propose

### Primal-Dual Interior-Point Methods. Ryan Tibshirani Convex Optimization /36-725

Primal-Dual Interior-Point Methods Ryan Tibshirani Convex Optimization 10-725/36-725 Given the problem Last time: barrier method min x subject to f(x) h i (x) 0, i = 1,... m Ax = b where f, h i, i = 1,...

### The fundamental theorem of linear programming

The fundamental theorem of linear programming Michael Tehranchi June 8, 2017 This note supplements the lecture notes of Optimisation The statement of the fundamental theorem of linear programming and the

### CO350 Linear Programming Chapter 6: The Simplex Method

CO350 Linear Programming Chapter 6: The Simplex Method 8th June 2005 Chapter 6: The Simplex Method 1 Minimization Problem ( 6.5) We can solve minimization problems by transforming it into a maximization

### Introduction to Linear Programming

Nanjing University October 27, 2011 What is LP The Linear Programming Problem Definition Decision variables Objective Function x j, j = 1, 2,..., n ζ = n c i x i i=1 We will primarily discuss maxizming

### E5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization

E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained

### What can be expressed via Conic Quadratic and Semidefinite Programming?

What can be expressed via Conic Quadratic and Semidefinite Programming? A. Nemirovski Faculty of Industrial Engineering and Management Technion Israel Institute of Technology Abstract Tremendous recent