# EE 227A: Convex Optimization and Applications October 14, 2008

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV Direct approach Primal problem Consider the SDP in standard form: p : C, : A i, = b i, i =1,..., m, 0, (13.1) where C, A i are given symmetric matrices, A, B = Tr AB denotes the scalar product between two symmetric matrices, and b R m is given Dual problem At first glance, the problem (13.1) is not amenable to the duality theory developed so far, since the constraint 0 is not a scalar one. Minimum eigenvalue representation. We develop a dual based on a representation of the problem via the minimum eigenvalue, as p C, : A i, = b i, i =1,..., m, λ min () 0, (13.2) where we have used the minimum eigenvalue function of a symmetric matrix A, given by λ min (A) := min Y Y, A : Y 0, Tr Y = 1 (13.3) to represent the positive semi-definiteness condition 0 in the SDP. The proof of the above representation of the minimum eigenvalue can be obtained by first showing that we can without loss of generality assume that A is diagonal, and noticing that we can then restrict Y to be diagonal as well. Note that the above representation proves that λ min is concave, so problem (13.2) is indeed convex as written. 13-1

2 Lagrangian and dual function. The Lagrangian for the maximization problem (13.2) is L(, λ, ν) = C, + ν i (b i A i, )+λ λ min () = ν T b + C ν i A i, + λ λ min (), where nu R m and λ 0 are the dual variables. The corresponding dual function g(λ, ν) : L(, λ, ν). involves the following subproblem, in which Z = C m ν ia i and λ 0 are given: G(λ, Z) : Z, + λλ min(). (13.4) For fixed λ 0, the function G(,λ) is the conjugate of the convex function λλ min. We have ( ) G(λ, Z) Z, + λ min Y, where, : Tr Y =1 min Z + λy, [eqn. (13.3)], Tr Y =1 Z + Y, [replace Y by λy ] min, Tr Y =λ min max, Tr Y =λ = min, Tr Y =λ = G(λ, Z), Z + Y, [minimax inequality] { 0 if Z + Y =0 + otherwise { 0 if Tr Z = λ 0, Z 0, G(λ, Z) := + otherwise. We now show that G(λ, Z) =G(λ, Z). To prove this, first note that G(λ, Z) 0 (since = 0 is feasible). This shows that G(λ, Z) itself is 0 if (λ, Z) is in the domain of G. Conversely, if Tr Z + λ 0, choosing = ɛti with ɛ = sign(tr Z + λ) and t + implies G(λ, Z) =+. Likewise, if Tr Z + λ = 0 and λ< 0, we choose = ti with t +, with the same result. Finally, if Tr Z = λ 0 but λ max (Z) > 0, choose = tuu T, with u a unit-norm eigenvector corresponding to the largest eigenvalue of Z, and t +. Here, we have Z, + λλ min () =t(λ max (Z)+λ) +, where we have exploited the fact that λ max (Z)+λ>λ

3 Dual problem. Coming back to the Lagrangian, we need to apply our result to Z = C m ν ia i. The dual function is { 0 if Z := C m g(λ, ν) = ν ia i 0, Tr Z = λ 0, + otherwise. We obtain the dual problem λ,ν,z νt b : Z = C ν i A i 0, Tr Z = λ 0, or, after elimination of λ, and noticing that Z 0 implies Tr Z 0: ν ν T b : The dual problem is also an SDP, in standard inequality form Conic approach Conic Lagrangian The same dual can be obtained with the conic Lagrangian L(, ν, Y ) := C, + ν i A i C (13.5) ν i (b i A i, )+ Y,, where now we associate a matrix dual variable Y to the constraint 0. Let us check that the Lagrangian above works, in the sense that we can represent the constrained maximization problem (13.1) as an unconstrained, maximin problem: p min L(, ν, Y ). We need to check that, for an arbitrary matrix Z, we have min Y, = { 0 if 0 otherwise. (13.6) This is an immediate consequence of the following: min Y, = min t 0 min Y, = min, Tr Y =t t 0 tλ min(), where we have exploited the representation of the minimum eigenvalue given in (13.3). The geometric interpretation is that the cone of positive-semidefinite matrices has a 90 o angle at the origin. 13-3

4 Dual problem The minimax inequality then implies The corresponding dual function is p d := min max ν, L(, ν, Y ). g(y, ν) { ν L(, ν, Y )= T b if C m ν ia i + Y =0 otherwise. The dual problem then writes ν, g(y, ν) = min ν, νt b : C ν i A i = Y 0. After elimination of the variable Y, we find the same problem as before, namely (13.5) Weak and strong duality Weak duality For the maximization problem (13.2), weak duality states that p d. Note that the fact that weak duality inequality ν T b C, holds for any primal-dual feasible pair (, ν), is a direct consequence of (13.6) Strong duality From Slater s theorem, strong duality will hold if the primal problem is strictly feasible, that is, if there exist 0 such that A i, = b i, i =1,..., m. Using the same approach as above, one can show that the dual of problem (13.5) is precisely the primal problem (13.2). Hence, if the dual problem is strictly feasible, then strong duality also holds.recall that we say that a problem is attained if its optimal set is not empty. It turns out that if both problems are strictly feasible, then both problems are attained. A strong duality theorem. The following theorem summarizes our results. Theorem 1. Consider the SDP p : C, : A i, = b i, i =1,..., m,

5 and its dual The following holds: ν ν T b : ν i A i C. Duality is symmetric, in the sense that the dual of the dual is the primal. Weak duality always holds: p d, so that, for any primal-dual feasible pair (, ν), we have ν T b C,. If the primal (resp. dual) problem is bounded above (resp. below), and strictly feasible, then p = d and the dual (resp. primal) is attained. If both problems are strictly feasible, then p = d and both problems are attained Examples An SDP where strong duality fails Contrarily to linear optimization problems, SDPs can fail to have a zero duality gap, even when they are feasible. Consider the example: x p = min x 2 : 0 x 1 x 2 0. x 0 x 2 0 Any primal feasible x satisfies x 2 = 0. Indeed, positive-semidefiniteness of the lower-right 2 2 block in the LMI of the above problem writes, using Schur complements, as x 1 0, x Hence, we have p = 0. The dual is d Y S 3 Y 11 : Y 0, Y 22 =0, 1 Y 11 2Y 23 =0. Any dual feasible Y satisfies Y 23 = 0 (since Y 22 = 0), thus Y 11 = 1 =d An eigenvalue problem For a matrix A S n +, we consider the SDP p The associated Lagrangian, using the conic approach, is A, : Tr =1, 0. (13.7) L(, Y, ν) = A, + ν(1 Tr )+ Y,, 13-5

6 with the matrix dual variable Y 0, while ν R is free. The dual function is g(y, ν) We obtain the dual problem p ν,y L(, Y, ν) = { ν if νi = Y + A + otherwise. ν : Y + A = λi, Y 0. Eliminating Y leads to {ν : νi A} = λ max (A). ν Both the primal and dual problems are strictly feasible, so p = d, and both values are attained. This proves the representation (13.7) for the largest eigenvalue of A SDP relaxations for a non-convex quadratic problem In lectures 7 and 11, we have seen two kinds of relaxation for the non-convex problem p : x x T Wx : x 2 i 1, i =1,..., n, where the symmetric matrix W S n is given. One relaxation is based on a standard relaxation of the constraints, and leads to (see lecture 11) p d lag := min Tr D : D W, D diagonal. (13.8) D Another relaxation (lecture 7) involved expressing the problem as an SDP with rank constraints on the a matrix = xx T : d rank : W, : 0, ii =1, i =1,..., m. Let us examine the dual of the first relaxation (13.8). We note that the problem is strictly feasible, so strong duality holds. Using the conic approach, we have d lag := min D Y = d rank. max min D Y, W : Tr D + Y, W D Tr D + Y, W D 0, Y ii =1, i =1,..., m This shows that both Lagrange and rank relaxations give the same value, and are dual of each other. In general, for arbitrary non-convex quadratic problems, the rank relaxation can be shown to be always better than the Lagrange relaxation, as the former is the (conic) dual to the latter. If either is strictly feasible, then they have the same optimal value. 13-6

### Lecture 7: Weak Duality

EE 227A: Conve Optimization and Applications February 7, 2012 Lecture 7: Weak Duality Lecturer: Laurent El Ghaoui 7.1 Lagrange Dual problem 7.1.1 Primal problem In this section, we consider a possibly

### 14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity

### Lecture 6: Conic Optimization September 8

IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

### Lecture 1. 1 Conic programming. MA 796S: Convex Optimization and Interior Point Methods October 8, Consider the conic program. min.

MA 796S: Convex Optimization and Interior Point Methods October 8, 2007 Lecture 1 Lecturer: Kartik Sivaramakrishnan Scribe: Kartik Sivaramakrishnan 1 Conic programming Consider the conic program min s.t.

### Optimization for Machine Learning

Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html

### ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications

ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March

### Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

### Semidefinite Programming

Semidefinite Programming Basics and SOS Fernando Mário de Oliveira Filho Campos do Jordão, 2 November 23 Available at: www.ime.usp.br/~fmario under talks Conic programming V is a real vector space h, i

### Relaxations and Randomized Methods for Nonconvex QCQPs

Relaxations and Randomized Methods for Nonconvex QCQPs Alexandre d Aspremont, Stephen Boyd EE392o, Stanford University Autumn, 2003 Introduction While some special classes of nonconvex problems can be

### 16.1 L.P. Duality Applied to the Minimax Theorem

CS787: Advanced Algorithms Scribe: David Malec and Xiaoyong Chai Lecturer: Shuchi Chawla Topic: Minimax Theorem and Semi-Definite Programming Date: October 22 2007 In this lecture, we first conclude our

### c 2005 Society for Industrial and Applied Mathematics

SIAM J. MATRIX ANAL. APPL. Vol. 27, No. 2, pp. 532 546 c 2005 Society for Industrial and Applied Mathematics LEAST-SQUARES COVARIANCE MATRIX ADJUSTMENT STEPHEN BOYD AND LIN XIAO Abstract. We consider the

### On the Method of Lagrange Multipliers

On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should

### 15. Conic optimization

L. Vandenberghe EE236C (Spring 216) 15. Conic optimization conic linear program examples modeling duality 15-1 Generalized (conic) inequalities Conic inequality: a constraint x K where K is a convex cone

### MIT Algebraic techniques and semidefinite optimization February 14, Lecture 3

MI 6.97 Algebraic techniques and semidefinite optimization February 4, 6 Lecture 3 Lecturer: Pablo A. Parrilo Scribe: Pablo A. Parrilo In this lecture, we will discuss one of the most important applications

### Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 5

Semidefinite and Second Order Cone Programming Seminar Fall 2001 Lecture 5 Instructor: Farid Alizadeh Scribe: Anton Riabov 10/08/2001 1 Overview We continue studying the maximum eigenvalue SDP, and generalize

### Module 04 Optimization Problems KKT Conditions & Solvers

Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September

### Convex Optimization and Modeling

Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

### Advances in Convex Optimization: Theory, Algorithms, and Applications

Advances in Convex Optimization: Theory, Algorithms, and Applications Stephen Boyd Electrical Engineering Department Stanford University (joint work with Lieven Vandenberghe, UCLA) ISIT 02 ISIT 02 Lausanne

### Rank-one LMIs and Lyapunov's Inequality. Gjerrit Meinsma 4. Abstract. We describe a new proof of the well-known Lyapunov's matrix inequality about

Rank-one LMIs and Lyapunov's Inequality Didier Henrion 1;; Gjerrit Meinsma Abstract We describe a new proof of the well-known Lyapunov's matrix inequality about the location of the eigenvalues of a matrix

### Lecture 15 Newton Method and Self-Concordance. October 23, 2008

Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications

### Lecture: Cone programming. Approximating the Lorentz cone.

Strong relaxations for discrete optimization problems 10/05/16 Lecture: Cone programming. Approximating the Lorentz cone. Lecturer: Yuri Faenza Scribes: Igor Malinović 1 Introduction Cone programming is

### Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark

Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian

### LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE

LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization

### FINDING LOW-RANK SOLUTIONS OF SPARSE LINEAR MATRIX INEQUALITIES USING CONVEX OPTIMIZATION

FINDING LOW-RANK SOLUTIONS OF SPARSE LINEAR MATRIX INEQUALITIES USING CONVEX OPTIMIZATION RAMTIN MADANI, GHAZAL FAZELNIA, SOMAYEH SOJOUDI AND JAVAD LAVAEI Abstract. This paper is concerned with the problem

### HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.

HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

### Tutorial on Convex Optimization for Engineers Part II

Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

### Lecture 17: Primal-dual interior-point methods part II

10-725/36-725: Convex Optimization Spring 2015 Lecture 17: Primal-dual interior-point methods part II Lecturer: Javier Pena Scribes: Pinchao Zhang, Wei Ma Note: LaTeX template courtesy of UC Berkeley EECS

### The Nearest Doubly Stochastic Matrix to a Real Matrix with the same First Moment

he Nearest Doubly Stochastic Matrix to a Real Matrix with the same First Moment William Glunt 1, homas L. Hayden 2 and Robert Reams 2 1 Department of Mathematics and Computer Science, Austin Peay State

### An Introduction to Linear Matrix Inequalities. Raktim Bhattacharya Aerospace Engineering, Texas A&M University

An Introduction to Linear Matrix Inequalities Raktim Bhattacharya Aerospace Engineering, Texas A&M University Linear Matrix Inequalities What are they? Inequalities involving matrix variables Matrix variables

### Semidefinite Programming, Combinatorial Optimization and Real Algebraic Geometry

Semidefinite Programming, Combinatorial Optimization and Real Algebraic Geometry assoc. prof., Ph.D. 1 1 UNM - Faculty of information studies Edinburgh, 16. September 2014 Outline Introduction Definition

### Convex Optimization of Graph Laplacian Eigenvalues

Convex Optimization of Graph Laplacian Eigenvalues Stephen Boyd Abstract. We consider the problem of choosing the edge weights of an undirected graph so as to maximize or minimize some function of the

### Game Theory. Greg Plaxton Theory in Programming Practice, Spring 2004 Department of Computer Science University of Texas at Austin

Game Theory Greg Plaxton Theory in Programming Practice, Spring 2004 Department of Computer Science University of Texas at Austin Bimatrix Games We are given two real m n matrices A = (a ij ), B = (b ij

### Optimization Theory. A Concise Introduction. Jiongmin Yong

October 11, 017 16:5 ws-book9x6 Book Title Optimization Theory 017-08-Lecture Notes page 1 1 Optimization Theory A Concise Introduction Jiongmin Yong Optimization Theory 017-08-Lecture Notes page Optimization

### A Tutorial on Convex Optimization II: Duality and Interior Point Methods

A Tutorial on Convex Optimization II: Duality and Interior Point Methods Haitham Hindi Palo Alto Research Center (PARC), Palo Alto, California 94304 email: hhindi@parc.com Abstract In recent years, convex

### Lecture: Convex Optimization Problems

1/36 Lecture: Convex Optimization Problems http://bicmr.pku.edu.cn/~wenzw/opt-2015-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/36 optimization

### A CONIC DANTZIG-WOLFE DECOMPOSITION APPROACH FOR LARGE SCALE SEMIDEFINITE PROGRAMMING

A CONIC DANTZIG-WOLFE DECOMPOSITION APPROACH FOR LARGE SCALE SEMIDEFINITE PROGRAMMING Kartik Krishnan Advanced Optimization Laboratory McMaster University Joint work with Gema Plaza Martinez and Tamás

### CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008.

1 ECONOMICS 594: LECTURE NOTES CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS W. Erwin Diewert January 31, 2008. 1. Introduction Many economic problems have the following structure: (i) a linear function

### ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications

ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications Professor M. Chiang Electrical Engineering Department, Princeton University February

### 4. Convex optimization problems

Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization

### EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 2

EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 2 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 5, 2012 Andre Tkacenko

### Optimization Theory. Lectures 4-6

Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x\$0} Existence of a maximum:

### Solving Dual Problems

Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem

Robust Convex Quadratically Constrained Quadratic Programming with Mixed-Integer Uncertainty Can Gokalp, Areesh Mittal, and Grani A. Hanasusanto Graduate Program in Operations Research and Industrial Engineering,

Nonconvex Quadratic Programming: Return of the Boolean Quadric Polytope Kurt M. Anstreicher Dept. of Management Sciences University of Iowa Seminar, Chinese University of Hong Kong, October 2009 We consider

### A new graph parameter related to bounded rank positive semidefinite matrix completions

Mathematical Programming manuscript No. (will be inserted by the editor) Monique Laurent Antonios Varvitsiotis A new graph parameter related to bounded rank positive semidefinite matrix completions the

### SWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm. Convex Optimization. Computing and Software McMaster University

SWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm Convex Optimization Computing and Software McMaster University General NLO problem (NLO : Non Linear Optimization) (N LO) min

### Symmetric and Asymmetric Duality

journal of mathematical analysis and applications 220, 125 131 (1998) article no. AY975824 Symmetric and Asymmetric Duality Massimo Pappalardo Department of Mathematics, Via Buonarroti 2, 56127, Pisa,

### Constrained optimization: direct methods (cont.)

Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a

### The Q Method for Symmetric Cone Programmin

The Q Method for Symmetric Cone Programming The Q Method for Symmetric Cone Programmin Farid Alizadeh and Yu Xia alizadeh@rutcor.rutgers.edu, xiay@optlab.mcma Large Scale Nonlinear and Semidefinite Progra

### Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method

Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach

### Convex Optimization & Parsimony of L p-balls representation

Convex Optimization & Parsimony of L p -balls representation LAAS-CNRS and Institute of Mathematics, Toulouse, France IMA, January 2016 Motivation Unit balls associated with nonnegative homogeneous polynomials

### Support Vector Machines for Classification and Regression. 1 Linearly Separable Data: Hard Margin SVMs

E0 270 Machine Learning Lecture 5 (Jan 22, 203) Support Vector Machines for Classification and Regression Lecturer: Shivani Agarwal Disclaimer: These notes are a brief summary of the topics covered in

### INDEFINITE TRUST REGION SUBPROBLEMS AND NONSYMMETRIC EIGENVALUE PERTURBATIONS. Ronald J. Stern. Concordia University

INDEFINITE TRUST REGION SUBPROBLEMS AND NONSYMMETRIC EIGENVALUE PERTURBATIONS Ronald J. Stern Concordia University Department of Mathematics and Statistics Montreal, Quebec H4B 1R6, Canada and Henry Wolkowicz

### Determinant maximization with linear. S. Boyd, L. Vandenberghe, S.-P. Wu. Information Systems Laboratory. Stanford University

Determinant maximization with linear matrix inequality constraints S. Boyd, L. Vandenberghe, S.-P. Wu Information Systems Laboratory Stanford University SCCM Seminar 5 February 1996 1 MAXDET problem denition

### Convex Optimization Theory. Athena Scientific, Supplementary Chapter 6 on Convex Optimization Algorithms

Convex Optimization Theory Athena Scientific, 2009 by Dimitri P. Bertsekas Massachusetts Institute of Technology Supplementary Chapter 6 on Convex Optimization Algorithms This chapter aims to supplement

### Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

### MA 575 Linear Models: Cedric E. Ginestet, Boston University Regularization: Ridge Regression and Lasso Week 14, Lecture 2

MA 575 Linear Models: Cedric E. Ginestet, Boston University Regularization: Ridge Regression and Lasso Week 14, Lecture 2 1 Ridge Regression Ridge regression and the Lasso are two forms of regularized

### Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

### 1 The independent set problem

ORF 523 Lecture 11 Spring 2016, Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Tuesday, March 29, 2016 When in doubt on the accuracy of these notes, please cross chec with the instructor

### SPECTRAHEDRA. Bernd Sturmfels UC Berkeley

SPECTRAHEDRA Bernd Sturmfels UC Berkeley GAeL Lecture I on Convex Algebraic Geometry Coimbra, Portugal, Monday, June 7, 2010 Positive Semidefinite Matrices For a real symmetric n n-matrix A the following

### Robust Fisher Discriminant Analysis

Robust Fisher Discriminant Analysis Seung-Jean Kim Alessandro Magnani Stephen P. Boyd Information Systems Laboratory Electrical Engineering Department, Stanford University Stanford, CA 94305-9510 sjkim@stanford.edu

### CONVEX ALGEBRAIC GEOMETRY. Bernd Sturmfels UC Berkeley

CONVEX ALGEBRAIC GEOMETRY Bernd Sturmfels UC Berkeley Multifocal Ellipses Given m points (u 1, v 1 ),...,(u m, v m )intheplaner 2,and aradiusd > 0, their m-ellipse is the convex algebraic curve { (x, y)

### Introduction to Semidefinite Programming (SDP)

Introduction to Semidefinite Programming (SDP) Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. 1 1 Introduction Semidefinite programming (SDP ) is the most exciting development

### Support Vector Machines

EE 17/7AT: Optimization Models in Engineering Section 11/1 - April 014 Support Vector Machines Lecturer: Arturo Fernandez Scribe: Arturo Fernandez 1 Support Vector Machines Revisited 1.1 Strictly) Separable

### A Constraint Generation Approach to Learning Stable Linear Dynamical Systems

A Constraint Generation Approach to Learning Stable Linear Dynamical Systems Sajid M. Siddiqi Byron Boots Geoffrey J. Gordon Carnegie Mellon University NIPS 2007 poster W22 steam Application: Dynamic Textures

### Assignment 1: From the Definition of Convexity to Helley Theorem

Assignment 1: From the Definition of Convexity to Helley Theorem Exercise 1 Mark in the following list the sets which are convex: 1. {x R 2 : x 1 + i 2 x 2 1, i = 1,..., 10} 2. {x R 2 : x 2 1 + 2ix 1x

### U.C. Berkeley CS294: Beyond Worst-Case Analysis Handout 12 Luca Trevisan October 3, 2017

U.C. Berkeley CS94: Beyond Worst-Case Analysis Handout 1 Luca Trevisan October 3, 017 Scribed by Maxim Rabinovich Lecture 1 In which we begin to prove that the SDP relaxation exactly recovers communities

### A direct formulation for sparse PCA using semidefinite programming

A direct formulation for sparse PCA using semidefinite programming Alexandre d Aspremont alexandre.daspremont@m4x.org Department of Electrical Engineering and Computer Science Laurent El Ghaoui elghaoui@eecs.berkeley.edu

### Fast ADMM for Sum of Squares Programs Using Partial Orthogonality

Fast ADMM for Sum of Squares Programs Using Partial Orthogonality Antonis Papachristodoulou Department of Engineering Science University of Oxford www.eng.ox.ac.uk/control/sysos antonis@eng.ox.ac.uk with

### Second Order Cone Programming Relaxation of Nonconvex Quadratic Optimization Problems

Second Order Cone Programming Relaxation of Nonconvex Quadratic Optimization Problems Sunyoung Kim Department of Mathematics, Ewha Women s University 11-1 Dahyun-dong, Sudaemoon-gu, Seoul 120-750 Korea

### Problem 1 (Exercise 2.2, Monograph)

MS&E 314/CME 336 Assignment 2 Conic Linear Programming January 3, 215 Prof. Yinyu Ye 6 Pages ASSIGNMENT 2 SOLUTIONS Problem 1 (Exercise 2.2, Monograph) We prove the part ii) of Theorem 2.1 (Farkas Lemma

### ROBUST ANALYSIS WITH LINEAR MATRIX INEQUALITIES AND POLYNOMIAL MATRICES. Didier HENRION henrion

GRADUATE COURSE ON POLYNOMIAL METHODS FOR ROBUST CONTROL PART IV.1 ROBUST ANALYSIS WITH LINEAR MATRIX INEQUALITIES AND POLYNOMIAL MATRICES Didier HENRION www.laas.fr/ henrion henrion@laas.fr Airbus assembly

### Matrix Mathematics. Theory, Facts, and Formulas with Application to Linear Systems Theory. Dennis S. Bernstein

Matrix Mathematics Theory, Facts, and Formulas with Application to Linear Systems Theory Dennis S. Bernstein PRINCETON UNIVERSITY PRESS PRINCETON AND OXFORD Contents Special Symbols xv Conventions, Notation,

### Symmetric Matrices and Eigendecomposition

Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions

### Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization. Nick Gould (RAL)

Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization Nick Gould (RAL) x IR n f(x) subject to c(x) = Part C course on continuoue optimization CONSTRAINED MINIMIZATION x

### LP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra

LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality

### The Q Method for Second-Order Cone Programming

The Q Method for Second-Order Cone Programming Yu Xia Farid Alizadeh July 5, 005 Key words. Second-order cone programming, infeasible interior point method, the Q method Abstract We develop the Q method

### Conic Linear Programming. Yinyu Ye

Conic Linear Programming Yinyu Ye December 2004, revised October 2017 i ii Preface This monograph is developed for MS&E 314, Conic Linear Programming, which I am teaching at Stanford. Information, lecture

### A geometric proof of the spectral theorem for real symmetric matrices

0 0 0 A geometric proof of the spectral theorem for real symmetric matrices Robert Sachs Department of Mathematical Sciences George Mason University Fairfax, Virginia 22030 rsachs@gmu.edu January 6, 2011

### EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 4

EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 4 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 12, 2012 Andre Tkacenko

### Lecture 9: Implicit function theorem, constrained extrema and Lagrange multipliers

Lecture 9: Implicit function theorem, constrained extrema and Lagrange multipliers Rafikul Alam Department of Mathematics IIT Guwahati What does the Implicit function theorem say? Let F : R 2 R be C 1.

### Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

### Newton s Method. Ryan Tibshirani Convex Optimization /36-725

Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x

### On Conic QPCCs, Conic QCQPs and Completely Positive Programs

Noname manuscript No. (will be inserted by the editor) On Conic QPCCs, Conic QCQPs and Completely Positive Programs Lijie Bai John E.Mitchell Jong-Shi Pang July 28, 2015 Received: date / Accepted: date

### 1 Kernel methods & optimization

Machine Learning Class Notes 9-26-13 Prof. David Sontag 1 Kernel methods & optimization One eample of a kernel that is frequently used in practice and which allows for highly non-linear discriminant functions

### Ph.D. Katarína Bellová Page 1 Mathematics 2 (10-PHY-BIPMA2) EXAM - Solutions, 20 July 2017, 10:00 12:00 All answers to be justified.

PhD Katarína Bellová Page 1 Mathematics 2 (10-PHY-BIPMA2 EXAM - Solutions, 20 July 2017, 10:00 12:00 All answers to be justified Problem 1 [ points]: For which parameters λ R does the following system

### Agenda. Applications of semidefinite programming. 1 Control and system theory. 2 Combinatorial and nonconvex optimization

Agenda Applications of semidefinite programming 1 Control and system theory 2 Combinatorial and nonconvex optimization 3 Spectral estimation & super-resolution Control and system theory SDP in wide use

### 1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016

AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 8 February 22nd 1 Overview In the previous lecture we saw characterizations of optimality in linear optimization, and we reviewed the

### Lecture Support Vector Machine (SVM) Classifiers

Introduction to Machine Learning Lecturer: Amir Globerson Lecture 6 Fall Semester Scribe: Yishay Mansour 6.1 Support Vector Machine (SVM) Classifiers Classification is one of the most important tasks in

### Lecture 13: Constrained optimization

2010-12-03 Basic ideas A nonlinearly constrained problem must somehow be converted relaxed into a problem which we can solve (a linear/quadratic or unconstrained problem) We solve a sequence of such problems

### Eigenvalues and Eigenvectors

Eigenvalues and Eigenvectors week -2 Fall 26 Eigenvalues and eigenvectors The most simple linear transformation from R n to R n may be the transformation of the form: T (x,,, x n ) (λ x, λ 2,, λ n x n

### WORST-CASE VALUE-AT-RISK AND ROBUST PORTFOLIO OPTIMIZATION: A CONIC PROGRAMMING APPROACH

WORST-CASE VALUE-AT-RISK AND ROBUST PORTFOLIO OPTIMIZATION: A CONIC PROGRAMMING APPROACH LAURENT EL GHAOUI Department of Electrical Engineering and Computer Sciences, University of California, Berkeley,

### Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

### Computing regularization paths for learning multiple kernels

Computing regularization paths for learning multiple kernels Francis Bach Romain Thibaux Michael Jordan Computer Science, UC Berkeley December, 24 Code available at www.cs.berkeley.edu/~fbach Computing

### Convex Optimization Overview (cnt d)

Conve Optimization Overview (cnt d) Chuong B. Do November 29, 2009 During last week s section, we began our study of conve optimization, the study of mathematical optimization problems of the form, minimize

### Numerical optimization

Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal

### Copositive Programming and Combinatorial Optimization

Copositive Programming and Combinatorial Optimization Franz Rendl http://www.math.uni-klu.ac.at Alpen-Adria-Universität Klagenfurt Austria joint work with M. Bomze (Wien) and F. Jarre (Düsseldorf) and

### CONVEX OPTIMIZATION, DUALITY, AND THEIR APPLICATION TO SUPPORT VECTOR MACHINES. Contents 1. Introduction 1 2. Convex Sets

CONVEX OPTIMIZATION, DUALITY, AND THEIR APPLICATION TO SUPPORT VECTOR MACHINES DANIEL HENDRYCKS Abstract. This paper develops the fundamentals of convex optimization and applies them to Support Vector