# Chap 2. Optimality conditions

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Chap 2. Optimality conditions Version: Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1 a point x D α = with f (x) 0. Then: In a neigborhood of x the solution set D α = is a C 1 -manifold of dimension n 1 and at x we have f (x) D = α i.e., f (x) is perpendicular to D = α. f (x) points into the direction where f (x) has increasing values. CO, Chapter 2 p 1/34

2 Example: the humpback function min x 2 1 (4 2.1x x 4 1 ) + x 1x 2 + x 2 2 ( 4 + 4x 2 2 ). Two global minima: ( ) and ( ), and four strict local minima. CO, Chapter 2 p 2/34

3 Fermat s theorem, Necessary condition Th. 2.5 (Fermat) Let f : R n R be in C 1 (C 2 ). If the point x R n is a local minimizer of the function f then f (x) = 0 (and 2 f (x) is psd). Ex. The conditions f (x) = 0, 2 f (x) psd, are necessary but not sufficient minimality conditions. Take f (x) = x 3 or ±x 4 as examples. Sufficient conditions for local minimizer Cor. 2.8 Let f : R n R be in C 2. If f (x) = 0 and 2 f (x) is positive definite then the point x is a strict local minimizer of the function f. CO, Chapter 2 p 3/34

4 Global minimizer of convex functions For convex functions the situation is easier L2.4 (Ex2.1) Any (strict) local minimizer of a convex function f is a (strict) global minimizer. T.2.6 Let f : R n R be convex and in C 1. The point x R n is a global minimizer of f if and only if f (x) = 0. Ex. (convex quadratic function) With positive definite Q, c R n consider the quadratic function f (x) := 1 2 x T Qx + c T x. Now, f is convex and therefore x is a global minimizer iff f (x) = 0 or Qx + c = 0 The global minimizer is therefore given by x = Q 1 c. CO, Chapter 2 p 4/34

5 2.2 Optimality conditions for constrained (convex) optimization We consider the convex optimization problem: where J = {1,, m}, (CO) min f (x) s.t. g j (x) 0, j J x C, C R n is a convex (closed) set; f, g 1,, g m are convex functions on C (or on an open set containing C). The set of feasible points (feasible set) is: F = {x C g j (x) 0, j = 1,, m}. Ex. Show that F is a convex set. CO, Chapter 2 p 5/34

6 Consider first P 0 : min{ f (x) : x C}, with C, a relatively open convex set and f is convex, differentiable on C. Th.2.9 The point x C is a (global) minimizer of P 0 if and only if f (x) T s = 0 for all s L, where L denotes the linear subspace defined by aff(c) = x + L for any x C. (Here aff(c) is the affine hull of C.) Remark: Essentially, P 0 is an unconstrained problem CO, Chapter 2 p 6/34

7 Example 1 Consider the relative interior of the simplex S = {x R 3 3 i=1 x i = 1, x 0} in R 3 : C = {x R 3 : 3 x i = 1, x > 0}, i=1 and define the (convex!) function f : C R by f (x) = (x 1 1) 2 + (x 2 1) 2 + (x 3 1) 2. By Exercise 1.6: aff(c) = (1, 0, 0) + L, where L is the linear subspace L := {s R 3 : s 1 + s 2 + s 3 = 0} = α β α, β R. Now x C is a minimizer of min x C f (x), if and only if f (x) T s = 0 for all s L. CO, Chapter 2 p 7/34

8 Since f (x) = 2 x 1 1 x 2 1 x 3 1 this holds if and only if x 1 1 = x 2 1 and x 1 1 = x 3 1, which gives x 1 = x 2 = x 3. Since x 1 + x 2 + x 3 = 1, it follows that f has a unique minimizer, namely x = (1/3; 1/3; 1/3). CO, Chapter 2 p 8/34

9 Feasible Directions The next characterization of minimizers is based on the concept of feasible directions Def.2.10 The vector s R n is a feasible direction at a point x F if there is a λ 0 > 0 such that x + λs F for all 0 λ λ 0. The set of feasible directions at x F is denoted by FD(x). Note that 0 FD(x). Recall. K R n is a cone if: x K, λ > 0 λx K. L.2.13 For any x F the set FD(x) is a convex cone. Th.2.14 (Let f C 1.) The feasible point x F is a (global) minimizer of (CO) if and only if δf (x, s) = f (x) T s 0, s FD(x). CO, Chapter 2 p 9/34

10 Example 1. (continued): Reconsider the simplex in R 3 : C = {x R 3 : and f : C R given by 3 x i = 1, x 0}, i=1 f (x) = (x 1 1) 2 + (x 2 1) 2 + (x 3 1) 2. At the point x = (1/3, 1/3, 1/3), the cone of feasible directions is simply FD(x) = {s R 3 : s 1 + s 2 + s 3 = 0}. (why?) Therefore, x = (1/3, 1/3, 1/3) is an optimal solution of min x C f (x), because x 1 1 f (x) = 2 x 2 1 = 4 1 1, x and hence f (x) T s = 0 for all s FD(x). CO, Chapter 2 p 10/34

11 Optimality conditions based on the Karush-Kuhn-Tucker relations: The conditions in Theorem 2.9/2.14 are not easy to check (in practice). We are thus interested in easier minimality conditions. Definition 2.15 With respect to (CO), an index j J is called active in x F if g j (x) = 0. The set of all active indices, also called active index set in x, is denoted by J x. CO, Chapter 2 p 11/34

12 Let us assume: C = R n, f, g j C 1 (in the rest of this subsection) Def. The feasible point x F satisfies the Karush-Kuhn-Tucker (KKT) conditions if there exist multipliers y j, j J such that: y j 0, j J and f (x) = j J y j g j (x), y j g j (x) = 0, j J. The KKT-conditions can equivalently be given as f (x) = j J x y j g j (x), y j 0, j J x. Th.2.16 (Sufficient condition) If x F satisfies the KKT conditions then x is a minimizer of (CO) (with C = R n ). CO, Chapter 2 p 12/34

13 Ex. The KKT conditions are not necessary for minimality. Take as example: min{x : x 2 0}. The minimum occurs at x = 0 (which is the only feasible point!). But there is no y such that 1 = y 0, y 0 = 0, y 0. (The problem here is that the feasible set F = {0} does not have interior points.) CO, Chapter 2 p 13/34

14 Some examples: (Application of the KKT-conditions) Ex. (minimizing a linear function over a sphere) Let a, c R n, c 0. Consider the problem { n } n c i x i (x i a i ) 2 1. min x R n i=1 The objective function is linear (hence convex), and there is one constraint function g 1 (x) (m = 1), which is convex. The KKT conditions give c 1. c n i=1 x 1 a 1 = 2λ. x n a n, λ ( n (x i a i ) 2 1 ) = 0, λ 0. i=1 CO, Chapter 2 p 14/34

15 c 0 implies λ > 0 and n i=1 (x i a i ) 2 = 1. We conclude that x a = αc, for α = 1/(2λ) > 0. Since x a = 1, we have α c = 1. So, the minimizing vector is x = a αc = a c c. Hence, the minimal value of the objective function is: c T x = c T a ct c c = ct a c 2 c = ct a c. CO, Chapter 2 p 15/34

16 Ex. (Exercise 2.7) Find a cylindrical can with height h, radius r and volume at least V such that the total surface area is minimal. Denoting the surface by p(h, r), we want to solve min p(h, r) := 2π(r 2 + rh) s.t. πr 2 h V, (r, h > 0) This is not a convex optimization problem! (why?) Setting r = e x 1 and h = e x 2 the problem becomes: { ) ) } min 2π (e 2x 1 + e x 1+x 2 π (e 2x 1+x 2 V which is equivalent to ) min{2π (e 2x 1 + e x 1+x 2 ln V π 2x 1 x 2 0, x 1 R, x 2 R}. This is a convex optimization problem! (why?) CO, Chapter 2 p 16/34

17 Ex. Exercise 2.7 (ctd) Let ) f (x) = 2π (e 2x 1 + e x 1+x 2 and g(x) = ln V π 2x 1 x 2. Now the problem becomes: { min f (x) g(x) 0, x R 2}. KKT conditions for this problem: we are looking for some x F such that f (x) = y (g(x)) for some scalar y 0, such that yg(x) = 0. The KKT conditions are: f (x) = y g(x), yg(x) = 0, y 0. CO, Chapter 2 p 17/34

18 Ex. Exercise 2.7 (ctd.) Calculate the gradients to get: ( ) [ 2e 2x 1 + e 2π x 1+x 2 e x 1+x 2 ] = y [ 2 1 ] = y [ 2 1 and yg(x) = 0, y 0. From ( ) we derive: y > 0 and therefore 2e 2x 1 + e x 1+x 2 = 2y e x 1+x 2 y = 2. This implies 2e x 1 = e x 2, whence 2r = h. Conclusion: The surface is minimal if 2r = h. Now use g(x) = 0, to get r and h in terms of V. Remark Later (in Chapter 6 Nonlinear Optimization ) we will be able to solve the problem without the transformation trick. ], CO, Chapter 2 p 18/34

19 Question to be answered now: Under which condition is the KKT-relation necessary for minimality? The Slater condition Def.2.18 x 0 C 0 is called a Slater point of (CO) if g j (x 0 ) < 0, j where g j is nonlinear, g j (x 0 ) 0, j where g j is (affine) linear. We say that (CO) satisfies the Slater condition (constraint qualification) if a Slater point exists. CO, Chapter 2 p 19/34

20 Example. (Slater regularity:) Consider the optimization problem min f (x) s.t. x x x 1 x 2 2 x 2 1 C = R 2. The points (1, 1) and ( 3 2, 3 4 ) are Slater points, (2, 0) not. CO, Chapter 2 p 20/34

21 Definition Some constraints g j (x) 0 may take the value zero at all feasible points. Each such constraint can be replaced by the equality constraint g j (x) = 0, without changing the feasible region. These constraints are called singular while the others are called regular. Under the Slater condition, we define the index sets J s of singular- and J r of regular constraints: J s := {j J g j (x) = 0, x F}, J r := J \ J s = {j J g j (x) < 0 for some x F}. Remark: Note, that if (CO) satisfies the Slater condition, then all singular constraints must be linear. CO, Chapter 2 p 21/34

22 Ideal Slater points Def.2.20 A Slater point x C 0 is called an Ideal Slater point of (CO) if g j (x ) < 0 for all j J r, g j (x ) = 0 for all j J s. L.2.21 If (CO) has a Slater point then it also has an ideal Slater point x F. Rem: An ideal Slater point is in the relative interior of F (Exercise 2.10). CO, Chapter 2 p 22/34

23 Example. (Slater regularity, continued:) Reconsider the program min f (x) s.t. x x x 1 x 2 2 x 2 1 C = R 2. The point (1, 1) is a Slater point, but not an ideal Slater point. The point ( 3 2, 3 4 ) is an ideal Slater point. CO, Chapter 2 p 23/34

24 2.2.3 Convex Farkas Lemma Remark The so-called convex Farkas Lemma is the basis for the duality theory in convex optimization We begin with an auxiliary result. Th.2.23 (see also [FKS,Th10.1) Let = U R n be convex and w / U. Then there exists a separating hyperplain H = {x a T x = α}, 0 a R n, α R such that a T w α a T x x U and α > a T u 0 for some u 0 U. Proof: See pdf-file extraproofs for a proof. CO, Chapter 2 p 24/34

25 Observation: A number a is a lower bound for the optimal value of (CO) if and only if the system (1) has no solution. f (x) < a g j (x) 0, x C j = 1,..., m Lemma.2.25 (Farkas) Let (CO) satisfy the Slater condition. Then the inequality system (1) has no solution if and only there exists a vector y = (y 1,, y m ) such that (2) y 0, f (x) + m y j g j (x) a, x C. j=1 The systems (1) and (2) are called alternative systems, because exactly one of them has a solution. CO, Chapter 2 p 25/34

26 Proof of the (convex) Farkas lemma We need to show that precisely one of the systems (1) or (2) has a solution. First we show (easy part) that not both systems can have a solution: Otherwise for some x C and y 0, the relation a f (x) + m y j g j (x) f (x) < a, j=1 would hold, a contradiction. Then we show that at least one of the two systems has a solution. To do so, we show that if (1) has no solution. then (2) has a solution. CO, Chapter 2 p 26/34

27 Proof: Define the set U R m+1 as follows. U = {u = (u 0 ;... ; u m ) x C such that f (x) < a + u 0 g j (x) u j if j J r g j (x) = u j if j J s Since (1) is infeasible, 0 / U. One easily checks that U is a nonempty convex set (using that the singular functions are linear). So there exists a hyperplane that separates 0 from U, i.e., there exists a nonzero vector y = (y 0, y 1,, y m ) such that y T u 0 = y T 0 for all u U y T u > 0 for some u U. ( ) The rest of the proof is divided into four parts. CO, Chapter 2 p 27/34

28 I. Prove that y 0 0 and y j 0 for all j J r. II. Establish that holds y 0 (f (x) a) + m y j g j (x) 0, x C. j=1 III. Prove that y 0 must be positive. IV. Show by induction that we can assume y j > 0, j J s. CO, Chapter 2 p 28/34

29 Example. (Illustration of the Farkas lemma) Let us consider the convex optimization problem (CO) min 1 + x s.t. x x R. Then (CO) is Slater regular. The optimal value is 0 (for x = 1). Hence, the system 1 + x < 0, x 2 1 0, x R has no solution. By the Farkas lemma there must exist a real number y 0 such that ( ) 1 + x + y x 2 1 0, x R. Indeed, taking y = 1 2 we get x y(x 2 1) = 1 2 x 2 + x = 1 2 (x + 1)2 0. CO, Chapter 2 p 29/34

30 Exercise 2.11 ( Linear Farkas Lemma ) Let A R m n and b R m. Exactly one of the following alternative systems (I) or (II) is solvable: (I) A T x 0, x 0, b T x < 0, or (II) Ay b, y 0. Proof: Apply Farkas lemma to the special case: f (x) = b T x, ( g j (x) = (A T x) j = a j) T x, j = 1,, m where a j denotes column j of A, and C is the nonnegative orthant (nonnegative vectors) of R m. Make use of: z T x = i z i x i 0 x 0 z 0 ( ) CO, Chapter 2 p 30/34

31 Karush-Kuhn-Tucker theory (Duality theory) Remark: This (Lagrange) Duality Theory is based on the concept of saddle points. Def. We introduce the Lagrangean function of (CO): L(x, y) := f (x) + m y j g j (x), y 0 j=1 Note that L(x, y) is convex in x and linear in y. Note: For any y 0: inf {f (x) + m x C j=1 y j g j (x)} inf {f (x) + x C g j (x) 0, j { 0 }} { m j=1 y j g j (x)} inf {f (x) g j(x) 0, j} x C CO, Chapter 2 p 31/34

32 So: For any y 0, inf {f (x) + m x C j=1 y j g j (x)} inf x C {f (x) g j(x) 0, j} (CO) Note further that for any fixed x: m { f (x) if gj (x) 0, j sup{f (x) + y j g j (x)} = y 0 otherwise j=1 By taking (above) the sup over y 0 we find: Weak Duality: sup y 0 inf L(x, y) inf x C sup x C y 0 L(x, y) ( ) The righthand side is equivalent with (CO); the lefthand side is called the (Lagrangean) dual problem of (CO) (See also Chapter 3) CO, Chapter 2 p 32/34

33 Saddlepoint theory Def A vector pair (x, y), with x C and y 0 is called a saddle point of the Lagrange function L(x, y) if L(x, y) L(x, y) L(x, y), x C, y 0. Lemma 2.28: A saddlepoint (x, y) of L(x, y) satisfies the strong duality relation sup y 0 inf L(x, y) = L(x, y) = inf x C sup x C y 0 L(x, y). Theorem 2.30 Let the problem (CO) satisfy the Slater condition. Then the vector x is a minimizer of (CO) if and only if there is a vector y such that (x, y) is a saddle point of the Lagrange function L. CO, Chapter 2 p 33/34

34 We are now able to prove the converse of Th Cor.2.33 Let C = R n and let f, g j C 1 be convex functions. Suppose (CO) satisfies the Slater condition. Then x is a minimizer of (CO) if and only if with some y 0 the pair (x, y) satisfies the KKT condition. Ex. (without Slater condition) Show that the pair (x, y) (x F, y 0) satisfies the KKT condition if and only if (x, y) (y 0) is a saddle point of L. CO, Chapter 2 p 34/34

### The Karush-Kuhn-Tucker (KKT) conditions

The Karush-Kuhn-Tucker (KKT) conditions In this section, we will give a set of sufficient (and at most times necessary) conditions for a x to be the solution of a given convex optimization problem. These

### Date: July 5, Contents

2 Lagrange Multipliers Date: July 5, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 14 2.3. Informative Lagrange Multipliers...........

### SWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm. Convex Optimization. Computing and Software McMaster University

SWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm Convex Optimization Computing and Software McMaster University General NLO problem (NLO : Non Linear Optimization) (N LO) min

### Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

### Lecture 6: Conic Optimization September 8

IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

### Optimization for Machine Learning

Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html

### Convex Optimization and Modeling

Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

### Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

### Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

### Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

### 14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity

### LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE

LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization

### SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING

Nf SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING f(x R m g HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 5, DR RAPHAEL

### Nonlinear Programming 3rd Edition. Theoretical Solutions Manual Chapter 6

Nonlinear Programming 3rd Edition Theoretical Solutions Manual Chapter 6 Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts 1 NOTE This manual contains

### Symmetric and Asymmetric Duality

journal of mathematical analysis and applications 220, 125 131 (1998) article no. AY975824 Symmetric and Asymmetric Duality Massimo Pappalardo Department of Mathematics, Via Buonarroti 2, 56127, Pisa,

### Lecture 2: Convex Sets and Functions

Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are

### Assignment 1: From the Definition of Convexity to Helley Theorem

Assignment 1: From the Definition of Convexity to Helley Theorem Exercise 1 Mark in the following list the sets which are convex: 1. {x R 2 : x 1 + i 2 x 2 1, i = 1,..., 10} 2. {x R 2 : x 2 1 + 2ix 1x

### Applied Lagrange Duality for Constrained Optimization

Applied Lagrange Duality for Constrained Optimization February 12, 2002 Overview The Practical Importance of Duality ffl Review of Convexity ffl A Separating Hyperplane Theorem ffl Definition of the Dual

### 8 Barrier Methods for Constrained Optimization

IOE 519: NL, Winter 2012 c Marina A. Epelman 55 8 Barrier Methods for Constrained Optimization In this subsection, we will restrict our attention to instances of constrained problem () that have inequality

### Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

### Lecture 15 Newton Method and Self-Concordance. October 23, 2008

Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications

### HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.

HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

### Optimization Theory. Lectures 4-6

Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x\$0} Existence of a maximum:

### Jørgen Tind, Department of Statistics and Operations Research, University of Copenhagen, Universitetsparken 5, 2100 Copenhagen O, Denmark.

DUALITY THEORY Jørgen Tind, Department of Statistics and Operations Research, University of Copenhagen, Universitetsparken 5, 2100 Copenhagen O, Denmark. Keywords: Duality, Saddle point, Complementary

### STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II - KUHN TUCKER THEORY

STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II - KUHN TUCKER THEORY UNIVERSITY OF MARYLAND: ECON 600 1. Some Eamples 1 A general problem that arises countless times in economics takes the form: (Verbally):

### On the Method of Lagrange Multipliers

On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should

### Approximate Farkas Lemmas in Convex Optimization

Approximate Farkas Lemmas in Convex Optimization Imre McMaster University Advanced Optimization Lab AdvOL Graduate Student Seminar October 25, 2004 1 Exact Farkas Lemma Motivation 2 3 Future plans The

### Tutorial on Convex Optimization for Engineers Part II

Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

### Module 04 Optimization Problems KKT Conditions & Solvers

Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September

### The Relation Between Pseudonormality and Quasiregularity in Constrained Optimization 1

October 2003 The Relation Between Pseudonormality and Quasiregularity in Constrained Optimization 1 by Asuman E. Ozdaglar and Dimitri P. Bertsekas 2 Abstract We consider optimization problems with equality,

### Tangent spaces, normals and extrema

Chapter 3 Tangent spaces, normals and extrema If S is a surface in 3-space, with a point a S where S looks smooth, i.e., without any fold or cusp or self-crossing, we can intuitively define the tangent

### LP Duality: outline. Duality theory for Linear Programming. alternatives. optimization I Idea: polyhedra

LP Duality: outline I Motivation and definition of a dual LP I Weak duality I Separating hyperplane theorem and theorems of the alternatives I Strong duality and complementary slackness I Using duality

### MAT-INF4110/MAT-INF9110 Mathematical optimization

MAT-INF4110/MAT-INF9110 Mathematical optimization Geir Dahl August 20, 2013 Convexity Part IV Chapter 4 Representation of convex sets different representations of convex sets, boundary polyhedra and polytopes:

### Convex Functions. Pontus Giselsson

Convex Functions Pontus Giselsson 1 Today s lecture lower semicontinuity, closure, convex hull convexity preserving operations precomposition with affine mapping infimal convolution image function supremum

### Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method

Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach

### Input: System of inequalities or equalities over the reals R. Output: Value for variables that minimizes cost function

Linear programming Input: System of inequalities or equalities over the reals R A linear cost function Output: Value for variables that minimizes cost function Example: Minimize 6x+4y Subject to 3x + 2y

### Finite Dimensional Optimization Part I: The KKT Theorem 1

John Nachbar Washington University March 26, 2018 1 Introduction Finite Dimensional Optimization Part I: The KKT Theorem 1 These notes characterize maxima and minima in terms of first derivatives. I focus

### Conic Linear Programming. Yinyu Ye

Conic Linear Programming Yinyu Ye December 2004, revised October 2017 i ii Preface This monograph is developed for MS&E 314, Conic Linear Programming, which I am teaching at Stanford. Information, lecture

### CONVEX OPTIMIZATION, DUALITY, AND THEIR APPLICATION TO SUPPORT VECTOR MACHINES. Contents 1. Introduction 1 2. Convex Sets

CONVEX OPTIMIZATION, DUALITY, AND THEIR APPLICATION TO SUPPORT VECTOR MACHINES DANIEL HENDRYCKS Abstract. This paper develops the fundamentals of convex optimization and applies them to Support Vector

### Lagrange multipliers. Technique for turning constrained optimization problems into unconstrained ones. Useful in general

Lagrange multipliers Technique for turning constrained optimization problems into unconstrained ones for intuition or for algorithm Useful in general but in particular, leads to a famous ML method: the

### Convex Optimization Theory

Convex Optimization Theory A SUMMARY BY DIMITRI P. BERTSEKAS We provide a summary of theoretical concepts and results relating to convex analysis, convex optimization, and duality theory. In particular,

### Linear Programming: Simplex

Linear Programming: Simplex Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Linear Programming: Simplex IMA, August 2016

### REGULAR LAGRANGE MULTIPLIERS FOR CONTROL PROBLEMS WITH MIXED POINTWISE CONTROL-STATE CONSTRAINTS

REGULAR LAGRANGE MULTIPLIERS FOR CONTROL PROBLEMS WITH MIXED POINTWISE CONTROL-STATE CONSTRAINTS fredi tröltzsch 1 Abstract. A class of quadratic optimization problems in Hilbert spaces is considered,

### 4. Convex optimization problems (part 1: general)

EE/AA 578, Univ of Washington, Fall 2016 4. Convex optimization problems (part 1: general) optimization problem in standard form convex optimization problems quasiconvex optimization 4 1 Optimization problem

### Computational Optimization. Constrained Optimization Part 2

Computational Optimization Constrained Optimization Part Optimality Conditions Unconstrained Case X* is global min Conve f X* is local min SOSC f ( *) = SONC Easiest Problem Linear equality constraints

### Elements of Convex Optimization Theory

Elements of Convex Optimization Theory Costis Skiadas August 2015 This is a revised and extended version of Appendix A of Skiadas (2009), providing a self-contained overview of elements of convex optimization

### Centre d Economie de la Sorbonne UMR 8174

Centre d Economie de la Sorbonne UMR 8174 On alternative theorems and necessary conditions for efficiency Do Van LUU Manh Hung NGUYEN 2006.19 Maison des Sciences Économiques, 106-112 boulevard de L'Hôpital,

### 10 Numerical methods for constrained problems

10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside

### Concave programming. Concave programming is another special case of the general constrained optimization. subject to g(x) 0

1 Introduction Concave programming Concave programming is another special case of the general constrained optimization problem max f(x) subject to g(x) 0 in which the objective function f is concave and

### AN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING

AN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING XIAO WANG AND HONGCHAO ZHANG Abstract. In this paper, we propose an Augmented Lagrangian Affine Scaling (ALAS) algorithm for general

### Lagrangian Duality. Richard Lusby. Department of Management Engineering Technical University of Denmark

Lagrangian Duality Richard Lusby Department of Management Engineering Technical University of Denmark Today s Topics (jg Lagrange Multipliers Lagrangian Relaxation Lagrangian Duality R Lusby (42111) Lagrangian

### Convex Optimization Overview (cnt d)

Conve Optimization Overview (cnt d) Chuong B. Do November 29, 2009 During last week s section, we began our study of conve optimization, the study of mathematical optimization problems of the form, minimize

### Solving Dual Problems

Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem

### LECTURE SLIDES ON BASED ON CLASS LECTURES AT THE CAMBRIDGE, MASS FALL 2007 BY DIMITRI P. BERTSEKAS.

LECTURE SLIDES ON CONVEX ANALYSIS AND OPTIMIZATION BASED ON 6.253 CLASS LECTURES AT THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY CAMBRIDGE, MASS FALL 2007 BY DIMITRI P. BERTSEKAS http://web.mit.edu/dimitrib/www/home.html

### The proximal mapping

The proximal mapping http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes Outline 2/37 1 closed function 2 Conjugate function

### First Order Conditions for Ideal Minimization of Matrix-Valued Problems

Journal of Convex Analysis Volume 10 (2003), No. 1, 1 19 First Order Conditions for Ideal Minimization of Matrix-Valued Problems L. M. Graña Drummond Programa de Engenharia de Sistemas de Computação, COPPE

### Linear Matrix Inequalities in Control

Linear Matrix nequalities in Control Carsten Scherer and Siep Weiland Department of Mathematics University of Stuttgart Germany Department of Electrical Engineering Eindhoven University of Technology The

### Locally convex spaces, the hyperplane separation theorem, and the Krein-Milman theorem

56 Chapter 7 Locally convex spaces, the hyperplane separation theorem, and the Krein-Milman theorem Recall that C(X) is not a normed linear space when X is not compact. On the other hand we could use semi

### AN INTRODUCTION TO CONVEXITY

AN INTRODUCTION TO CONVEXITY GEIR DAHL NOVEMBER 2010 University of Oslo, Centre of Mathematics for Applications, P.O.Box 1053, Blindern, 0316 Oslo, Norway (geird@math.uio.no) Contents 1 The basic concepts

### Duality Theory, Optimality Conditions

5.1 Duality Theory, Optimality Conditions Katta G. Murty, IOE 510, LP, U. Of Michigan, Ann Arbor We only consider single objective LPs here. Concept of duality not defined for multiobjective LPs. Every

### Duality and optimality in multiobjective optimization

Duality and optimality in multiobjective optimization Von der Fakultät für Mathematik der Technischen Universität Chemnitz genehmigte D i s s e r t a t i o n zur Erlangung des akademischen Grades Doctor

### Convex Sets with Applications to Economics

Convex Sets with Applications to Economics Debasis Mishra March 10, 2010 1 Convex Sets A set C R n is called convex if for all x, y C, we have λx+(1 λ)y C for all λ [0, 1]. The definition says that for

### Lecture 1. 1 Conic programming. MA 796S: Convex Optimization and Interior Point Methods October 8, Consider the conic program. min.

MA 796S: Convex Optimization and Interior Point Methods October 8, 2007 Lecture 1 Lecturer: Kartik Sivaramakrishnan Scribe: Kartik Sivaramakrishnan 1 Conic programming Consider the conic program min s.t.

### c 1999 Society for Industrial and Applied Mathematics

SIAM J. OPTIM. Vol. 9, No. 3, pp. 729 754 c 1999 Society for Industrial and Applied Mathematics A POTENTIAL REDUCTION NEWTON METHOD FOR CONSTRAINED EQUATIONS RENATO D. C. MONTEIRO AND JONG-SHI PANG Abstract.

### Convex Sets. Prof. Dan A. Simovici UMB

Convex Sets Prof. Dan A. Simovici UMB 1 / 57 Outline 1 Closures, Interiors, Borders of Sets in R n 2 Segments and Convex Sets 3 Properties of the Class of Convex Sets 4 Closure and Interior Points of Convex

### Support Vector Machines and Kernel Methods

2018 CS420 Machine Learning, Lecture 3 Hangout from Prof. Andrew Ng. http://cs229.stanford.edu/notes/cs229-notes3.pdf Support Vector Machines and Kernel Methods Weinan Zhang Shanghai Jiao Tong University

### Problem 1 (Exercise 2.2, Monograph)

MS&E 314/CME 336 Assignment 2 Conic Linear Programming January 3, 215 Prof. Yinyu Ye 6 Pages ASSIGNMENT 2 SOLUTIONS Problem 1 (Exercise 2.2, Monograph) We prove the part ii) of Theorem 2.1 (Farkas Lemma

### Solution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark

Solution Methods Richard Lusby Department of Management Engineering Technical University of Denmark Lecture Overview (jg Unconstrained Several Variables Quadratic Programming Separable Programming SUMT

### Lecture: Convex Optimization Problems

1/36 Lecture: Convex Optimization Problems http://bicmr.pku.edu.cn/~wenzw/opt-2015-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/36 optimization

### Chapter 1: Linear Programming

Chapter 1: Linear Programming Math 368 c Copyright 2013 R Clark Robinson May 22, 2013 Chapter 1: Linear Programming 1 Max and Min For f : D R n R, f (D) = {f (x) : x D } is set of attainable values of

### Zero sum games Proving the vn theorem. Zero sum games. Roberto Lucchetti. Politecnico di Milano

Politecnico di Milano General form Definition A two player zero sum game in strategic form is the triplet (X, Y, f : X Y R) f (x, y) is what Pl1 gets from Pl2, when they play x, y respectively. Thus g

### Support Vector Machines

EE 17/7AT: Optimization Models in Engineering Section 11/1 - April 014 Support Vector Machines Lecturer: Arturo Fernandez Scribe: Arturo Fernandez 1 Support Vector Machines Revisited 1.1 Strictly) Separable

### Closing the Duality Gap in Linear Vector Optimization

Journal of Convex Analysis Volume 11 (2004), No. 1, 163 178 Received July 4, 2003 Closing the Duality Gap in Linear Vector Optimization Andreas H. Hamel Martin-Luther-University Halle-Wittenberg, Department

### Support Vector Machines

Wien, June, 2010 Paul Hofmarcher, Stefan Theussl, WU Wien Hofmarcher/Theussl SVM 1/21 Linear Separable Separating Hyperplanes Non-Linear Separable Soft-Margin Hyperplanes Hofmarcher/Theussl SVM 2/21 (SVM)

### MIT Algebraic techniques and semidefinite optimization February 14, Lecture 3

MI 6.97 Algebraic techniques and semidefinite optimization February 4, 6 Lecture 3 Lecturer: Pablo A. Parrilo Scribe: Pablo A. Parrilo In this lecture, we will discuss one of the most important applications

### FRIEDRICH-ALEXANDER-UNIVERSITÄT ERLANGEN-NÜRNBERG INSTITUT FÜR ANGEWANDTE MATHEMATIK

FRIEDRICH-ALEXANDER-UNIVERSITÄT ERLANGEN-NÜRNBERG INSTITUT FÜR ANGEWANDTE MATHEMATIK Optimality conditions for vector optimization problems with variable ordering structures by G. Eichfelder & T.X.D. Ha

### Introduction to Semidefinite Programming (SDP)

Introduction to Semidefinite Programming (SDP) Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. 1 1 Introduction Semidefinite programming (SDP ) is the most exciting development

### On the relations between different duals assigned to composed optimization problems

manuscript No. will be inserted by the editor) On the relations between different duals assigned to composed optimization problems Gert Wanka 1, Radu Ioan Boţ 2, Emese Vargyas 3 1 Faculty of Mathematics,

### Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization

Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization Roger Behling a, Clovis Gonzaga b and Gabriel Haeser c March 21, 2013 a Department

### Gaps in Support Vector Optimization

Gaps in Support Vector Optimization Nikolas List 1 (student author), Don Hush 2, Clint Scovel 2, Ingo Steinwart 2 1 Lehrstuhl Mathematik und Informatik, Ruhr-University Bochum, Germany nlist@lmi.rub.de

### Linear programming: theory, algorithms and applications

Linear programming: theory, algorithms and applications illes@math.bme.hu Department of Differential Equations Budapest 2014/2015 Fall Vector spaces A nonempty set L equipped with addition and multiplication

### Optimization Tutorial 1. Basic Gradient Descent

E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.

### EC /11. Math for Microeconomics September Course, Part II Lecture Notes. Course Outline

LONDON SCHOOL OF ECONOMICS Professor Leonardo Felli Department of Economics S.478; x7525 EC400 20010/11 Math for Microeconomics September Course, Part II Lecture Notes Course Outline Lecture 1: Tools for

### Asymptotic at infinity. g(x) g(x) Tangential at finite point. slope y. slope y

Conjugate Transformations Since sup x2c [yx g(x)] = inf [g(x) yx] it is geometrically clear that y 2Diff there is a number t such that the graph of the linear equation : t + yx "supports" g : C, in that

### AN ABADIE-TYPE CONSTRAINT QUALIFICATION FOR MATHEMATICAL PROGRAMS WITH EQUILIBRIUM CONSTRAINTS. Michael L. Flegel and Christian Kanzow

AN ABADIE-TYPE CONSTRAINT QUALIFICATION FOR MATHEMATICAL PROGRAMS WITH EQUILIBRIUM CONSTRAINTS Michael L. Flegel and Christian Kanzow University of Würzburg Institute of Applied Mathematics and Statistics

### Mixed-integer nonlinear programming, Conic programming, Duality, Cutting

A STRONG DUAL FOR CONIC MIXED-INTEGER PROGRAMS DIEGO A. MORÁN R., SANTANU S. DEY, AND JUAN PABLO VIELMA Abstract. Mixed-integer conic programming is a generalization of mixed-integer linear programming.

### Convexity in R n. The following lemma will be needed in a while. Lemma 1 Let x E, u R n. If τ I(x, u), τ 0, define. f(x + τu) f(x). τ.

Convexity in R n Let E be a convex subset of R n. A function f : E (, ] is convex iff f(tx + (1 t)y) (1 t)f(x) + tf(y) x, y E, t [0, 1]. A similar definition holds in any vector space. A topology is needed

### General Convexity, General Concavity, Fixed Points, Geometry, and Min-Max Points

Mathematica Moravica Vol. 12-1 2008), 63 109 General Convexity, General Concavity, Fixed Points, Geometry, and Min-Max Points Milan R. Tasković Abstract. This paper continues the study of general convexity

### Calculus of Variations and Partial Differential Equations. Diogo Aguiar Gomes

Calculus of Variations and Partial Differential Equations Diogo Aguiar Gomes Contents. Introduction 5 1. Finite dimensional optimization problems 9 1. Unconstrained minimization in R n 1 2. Convexity

### Mingbin Feng, John E. Mitchell, Jong-Shi Pang, Xin Shen, Andreas Wächter

Complementarity Formulations of l 0 -norm Optimization Problems 1 Mingbin Feng, John E. Mitchell, Jong-Shi Pang, Xin Shen, Andreas Wächter Abstract: In a number of application areas, it is desirable to

### Optimality Conditions

Chapter 2 Optimality Conditions 2.1 Global and Local Minima for Unconstrained Problems When a minimization problem does not have any constraints, the problem is to find the minimum of the objective function.

### Duality for almost convex optimization problems via the perturbation approach

Duality for almost convex optimization problems via the perturbation approach Radu Ioan Boţ Gábor Kassay Gert Wanka Abstract. We deal with duality for almost convex finite dimensional optimization problems

### Math 273a: Optimization Basic concepts

Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize

### Mathematical Foundations -1- Constrained Optimization. Constrained Optimization. An intuitive approach 2. First Order Conditions (FOC) 7

Mathematical Foundations -- Constrained Optimization Constrained Optimization An intuitive approach First Order Conditions (FOC) 7 Constraint qualifications 9 Formal statement of the FOC for a maximum

### Support Vector Machine (SVM) and Kernel Methods

Support Vector Machine (SVM) and Kernel Methods CE-717: Machine Learning Sharif University of Technology Fall 2015 Soleymani Outline Margin concept Hard-Margin SVM Soft-Margin SVM Dual Problems of Hard-Margin

### Lecture: Cone programming. Approximating the Lorentz cone.

Strong relaxations for discrete optimization problems 10/05/16 Lecture: Cone programming. Approximating the Lorentz cone. Lecturer: Yuri Faenza Scribes: Igor Malinović 1 Introduction Cone programming is

### Chapter 3. Some Applications. 3.1 The Cone of Positive Semidefinite Matrices

Chapter 3 Some Applications Having developed the basic theory of cone programming, it is time to apply it to our actual subject, namely that of semidefinite programming. Indeed, any semidefinite program

### c 2012 by Anne-Sophie Gaby Marthe Puydupin.

c 2012 by Anne-Sophie Gaby Marthe Puydupin. INVERSE OPTIMIZATION OF DISCRETE-TIME SYSTEMS APPLIED TO HUMAN LOCOMOTION BY ANNE-SOPHIE GABY MARTHE PUYDUPIN THESIS Submitted in partial fulfillment of the