Nonlinear Programming Models
|
|
- Brian Haynes
- 5 years ago
- Views:
Transcription
1 Nonlinear Programming Models Fabio Schoen Nonlinear Programming Models p.
2 Introduction Nonlinear Programming Models p.
3 NLP problems minf(x) x S R n Standard form: min f(x) h i (x) = 0 i = 1,m g j (x) 0 j = 1,k Here S = {x R n : h i (x) = 0 i,g j (x) 0 j} Nonlinear Programming Models p.
4 Local and global optima A global minimum or global optimum is any x S such that x S f(x) f(x ) A point x is a local optimum if ε > 0 such that x S B( x,ε) f(x) f( x) where B( x,ε) = {x R n : x x ε} is a ball in R n. Any global optimum is also a local optimum, but the opposite is generally false. Nonlinear Programming Models p.
5 Convex Functions A set S R n is convex if x,y S λx + (1 λ)y S for all choices of λ [0, 1]. Let Ω R n : non empty convex set. A function f : Ω R is convex iff for all x,y Ω,λ [0, 1] f(λx + (1 λ)y) λf(x) + (1 λ)f(y) Nonlinear Programming Models p.
6 Convex Functions x y Nonlinear Programming Models p.
7 Properties of convex functions Every convex function is continuous in the interior of Ω. It might be discontinuous, but only on the frontier. If f is continuously differentiable then it is convex iff for all y Ω f(y) f(x) + (y x) T f(x) Nonlinear Programming Models p.
8 Convex functions x y Nonlinear Programming Models p.
9 If f is twice continuously differentiable f it is convex iff its Hessian matrix is positive semi-definite: [ ] 2 2 f f(x) := x i x j then 2 f(x) 0 iff v T 2 f(x)v 0 v R n or, equivalently, all eigenvalues of 2 f(x) are non negative. Nonlinear Programming Models p.
10 Example: an affine function is convex (and concave) For a quadratic function (Q: symmetric matrix): we have f(x) = 1 2 xt Qx + b T x + c f(x) = Qx + b 2 f(x) = Q f is convex iff Q 0 Nonlinear Programming Models p. 1
11 Convex Optimization Problems minf(x) x S is a convex optimization problem iff S is a convex set and f is convex on S. For a problem in standard form min f(x) h i (x) = 0 i = 1,m g j (x) 0 j = 1,k if f is convex, h i (x) are affine functions, g j (x) are convex functions, then the problem is convex. Nonlinear Programming Models p. 1
12 Maximization Slight abuse in notation: a problem maxf(x) x S is called convex iff S is a convex set and f is a concave function (not to be confused with minimization of a concave function, (or maximization of a convex function) which are NOT a convex optimization problem) Nonlinear Programming Models p. 1
13 Convex and non convex optimization Convex optimization is easy, non convex optimization is usually very hard. Fundamental property of convex optimization problems: every local optimum is also a global optimum (will give a proof later) Minimizing a positive semidefinite quadratic function on a polyhedron is easy (polynomially solvable); if even a single eigenvalue of the hessian is negative the problem becomes NP hard Nonlinear Programming Models p. 1
14 Convex functions: examples Many (of course not all... ) functions are convex! affine functions a T x + b quadratic functions 1 2 xt Qx + b T x + c with Q = Q T, Q 0 any norm is a convex function x log x (however log x is concave) f is convex if and only if x 0,d R n, its restriction to any line: φ(α) = f(x 0 + αd), is a convex function a linear non negative combination of convex functions is convex g(x,y) convex in x for all y g(x,y)dy convex Nonlinear Programming Models p. 1
15 more examples... max i {a T i x + b} is convex f,g: convex max{f(x),g(x)} is convex f a convex functions for any a A (a possibly uncountable set) sup a A f a (x) is convex f convex f(ax + b) let S R n be any set f(x) = sup s S x s is convex Trace(A T X) = i,j A ijx ij is convex (it is linear!) log det X 1 is convex over the set of matrices X R n n : X 0 λ max (X) (the largest eigenvalue of a matrix X) Nonlinear Programming Models p. 1
16 Data Approximation Nonlinear Programming Models p. 1
17 Table of contents norm approximation maximum likelihood robust estimation Nonlinear Programming Models p. 1
18 Norm approximation Problem: min x Ax b where A, b: parameters. Usually the system is over-determined, i.e. b Range(A). For example, this happens when A R m n with m > n and A has full rank. r := Ax b: residual. Nonlinear Programming Models p. 1
19 Examples r = r T r: least squares (or regression ) r = r T Pr with P 0: weighted least squares r = max i r i : minimax, or l or di Tchebichev approximation r = i r i : absolute or l 1 approximation Possible (convex) additional constraints: maximum deviation from an initial estimate: x x est ǫ simple bounds l i x i u i ordering: x 1 x 2 x n Nonlinear Programming Models p. 1
20 Example: l 1 norm Matrix A R norm 1 residuals Nonlinear Programming Models p. 2
21 l norm norm residuals Nonlinear Programming Models p. 2
22 l 2 norm norm 2 residuals Nonlinear Programming Models p. 2
23 Variants min i h(y i a T i x) where h: convex function: { z 2 z 1 h linear quadratic h(z) = 2 z 1 z > 1 { 0 z 1 dead zone : h(z) = z 1 z > 1 { log(1 z 2 ) z < 1 logarithmic barrier: h(z) = z 1 Nonlinear Programming Models p. 2
24 comparison norm 1(x) norm 2(x) linquad(x) deadzone(x) logbarrier(x) Nonlinear Programming Models p. 2
25 Maximum likelihood Given a sample X 1,X 2,...,X k and a parametric family of probability density functions L( ; θ), the maximum likelihood estimate of θ given the sample is ˆθ = arg max θ L(X 1,...,X k ;θ) Example: linear measures with and additive i.i.d. (independent identically dsitributed) noise: X i = a T i θ + ε i (1) where ε i iid random variables with density p( ): L(X 1...,X k ;θ) = k i=1 p(x i a T i θ) Nonlinear Programming Models p. 2
26 Max likelihood estimate - MLE (taking the logarithm, which does not change optimum points): ˆθ = arg max θ log(p(x i a T i θ)) If p is log concave this problem is convex. Examples: i ε N(0,σ), i.e. p(z) = (2πσ) 1/2 exp( z 2 /2σ 2 ) MLE is the l 2 estimate: θ = arg min Aθ X 2 ; p(z) = (1/(2a)) exp( z /a) l 1 estimate: ˆθ = arg min θ Aθ X 1 Nonlinear Programming Models p. 2
27 p(z) = (1/a) exp( z/a)1 {z 0} (negative exponential) the estimate can be found solving the LP problem: min 1 T (X Aθ) Aθ X p uniform on [ a,a] the MLE is any θ such that Aθ X a Nonlinear Programming Models p. 2
28 Ellipsoids An ellipsoid is a subset of R n of the form E = {x R n : (x x 0 ) T P 1 (x x 0 ) 1} where x 0 R n is the center of the ellipsoid and P is a symmetric positive-definite matrix. Alternative representations: where A 0, or E = {x R n : Ax b 2 1} E = {x R n : x = x 0 + Au u 2 1} where A is square and non singular (affine transformation of the unit ball) Nonlinear Programming Models p. 2
29 Robust Least Squares Least Squares: ˆx = arg min i (at i x b i) 2 Hp: a i not known, but it is known that a i E i = {ā i + P i u : u 1} where P i = P T i 0. Definition: worst case residuals: max a i E i (a T i x b i) 2 i A robust estimate of x is the solution of ˆx r = arg minmax (a T i x b x i) 2 a i E i i Nonlinear Programming Models p. 2
30 RLS It holds: α + β T y α + β y then, choosing y = β/ β if α 0 and y = β/ β, otherwise if α < 0, then y = 1 and then: α + β T y = α + β T β/ β sign(α) = α + β max (a T i x b i ) a i E i = max ā T i x b i + u T P i x u 1 = ā T i x b i + P i x Nonlinear Programming Models p. 3
31 ... Thus the Robust Least Squares problem reduces to min ( ) 1/2 ( ā T i x b i + P i x ) 2 i (a convex optimization problem). Transformation: min x,t t 2 ā T i x b i + P i x t i i i.e. Nonlinear Programming Models p. 3
32 ... min x,t t 2 ā T i x b i + P i x t i ā T i x + b i + P i x t i (Second Order Cone Problem). A norm cone is a convex set C = {(x,t) R n+1 : x t} Nonlinear Programming Models p. 3
33 Geometrical Problems Nonlinear Programming Models p. 3
34 Geometrical Problems projections and distances polyhedral intersection extremal volume ellipsoids classification problems Nonlinear Programming Models p. 3
35 Projection on a set Given a set C the projection of x on C is defined as: P C (x) = arg min z C z x Nonlinear Programming Models p. 3
36 Projection on a convex set If C = {x : Ax = b,f i (x) 0,i = 1,m} where f i : convex C is a convex set and the problem P C (x) = arg min x z Az = b f i (z) 0 i = 1,m is convex. Nonlinear Programming Models p. 3
37 Distance between convex sets dist(c (1),C (2) ) = min x y x C (1),y C (2) Nonlinear Programming Models p. 3
38 Distance between convex sets If C (j) = {x : A (j) x = b (j),f (j) i 0} then the minimum distance can be found through a convex model: min x (1) x (2) A (1) x (1) = b (1) A (2) x (2) = b (2) f (1) i x (1) 0 f (2) i x (2) 0 Nonlinear Programming Models p. 3
39 Polyhedral intersection 1: polyhedra described by means of linear inequalities: P 1 = {x : Ax b}, P 2 = {x : Cx d} Nonlinear Programming Models p. 3
40 Polyhedral intersection P 1 P2 =? It is a linear feasibility problem: Ax b,cx d P 1 P 2? Just check sup{c T k x : Ax b} d k k (solution of a finite number of LP s) Nonlinear Programming Models p. 4
41 Polyhedral intersection (2) 2: polyhedra (polytopes) described through vertices: P 1 = conv{v 1,...,v k }, P 2 = conv{w 1,...,w h } P 1 P2 =? Need to find λ 1,λ k,µ 1,µ h 0: λ i = 1 µ j = 1 i λ i v i i j = µ j w j j P 1 P 2? i = 1,...,k check whether µ j 0: µ j = 1 j µj w j = v i Nonlinear Programming Models p. 4
42 Minimal ellipsoid containing k points Given v 1,...,v k R n find an ellipsoid E = {x : Ax b 1} with minimal volume containing the k given points. Nonlinear Programming Models p. 4
43 A = A T 0. Volume of E is proportional to deta 1 convex optimization problem (in the unknowns: A, b): min log deta 1 A = A T A 0 Av i b 1 i = 1,k Nonlinear Programming Models p. 4
44 Max. ellipsoid contained in a polyhedron Given P = {x : Ax b} find an ellipsoid: E = {By + d : y 1} contained in P with maximum volume. Nonlinear Programming Models p. 4
45 Max. ellipsoid contained in a polyhedron E P a T i (By + d) b i y : y 1 sup {a T i By + a T i d} b i i y 1 Ba i + a T i d b i max B,d log detb B = B T 0 Ba i + a T i d b i i = 1,... Nonlinear Programming Models p. 4
46 Difficult variants These problems are hard: find a maximal volume ellipsoid contained in a polyhedron given by its vertices Nonlinear Programming Models p. 4
47 find a minimal volume ellipsoid containing a polyhedron described as a system of linear inequalities. Nonlinear Programming Models p. 4
48 It is already a difficult problem to show whether a given ellipsoid E contains a polyhedron P = {Ax b}. This problem is still difficult even when the ellipsoid is a sphere: this problem is equivalent to norm maximization in a polyhedron it is an NP hard concave optimization problem. Nonlinear Programming Models p. 4
49 Linear classification (separation) Nonlinear Programming Models p. 4
50 Given two point sets X 1,...,X k,y 1,...,Y h find an hyperplane a T x = t such that: (LP feasibility problem). a T X i 1 i = 1,k a T Y j 1 j = 1,h Nonlinear Programming Models p. 5
51 Robust separation Nonlinear Programming Models p. 5
52 Robust separation Find a maximal separation: max a: a 1 equivalent to the convex problem: ( ) min a T X i maxa T Y j i j maxt 1 t 2 a T X i t 1 i a T Y j t 2 j a 1 Nonlinear Programming Models p. 5
Nonlinear Programming Models p. 1
Nonlinear Programming Models Fabio Schoen Introduction 2008 http://gol.dsi.unifi.it/users/schoen Nonlinear Programming Models p. Nonlinear Programming Models p. 2 NLP problems Local and global optima min
More information9. Geometric problems
9. Geometric problems EE/AA 578, Univ of Washington, Fall 2016 projection on a set extremal volume ellipsoids centering classification 9 1 Projection on convex set projection of point x on set C defined
More information8. Geometric problems
8. Geometric problems Convex Optimization Boyd & Vandenberghe extremal volume ellipsoids centering classification placement and facility location 8 1 Minimum volume ellipsoid around a set Löwner-John ellipsoid
More information8. Geometric problems
8. Geometric problems Convex Optimization Boyd & Vandenberghe extremal volume ellipsoids centering classification placement and facility location 8 Minimum volume ellipsoid around a set Löwner-John ellipsoid
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 17 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 29, 2012 Andre Tkacenko
More informationGeometric problems. Chapter Projection on a set. The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as
Chapter 8 Geometric problems 8.1 Projection on a set The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as dist(x 0,C) = inf{ x 0 x x C}. The infimum here is always achieved.
More informationConvex Optimization: Applications
Convex Optimization: Applications Lecturer: Pradeep Ravikumar Co-instructor: Aarti Singh Convex Optimization 1-75/36-75 Based on material from Boyd, Vandenberghe Norm Approximation minimize Ax b (A R m
More informationMATHEMATICAL ECONOMICS: OPTIMIZATION. Contents
MATHEMATICAL ECONOMICS: OPTIMIZATION JOÃO LOPES DIAS Contents 1. Introduction 2 1.1. Preliminaries 2 1.2. Optimal points and values 2 1.3. The optimization problems 3 1.4. Existence of optimal points 4
More informationConvex Optimization and Modeling
Convex Optimization and Modeling Convex Optimization Fourth lecture, 05.05.2010 Jun.-Prof. Matthias Hein Reminder from last time Convex functions: first-order condition: f(y) f(x) + f x,y x, second-order
More informationLecture 2: Convex functions
Lecture 2: Convex functions f : R n R is convex if dom f is convex and for all x, y dom f, θ [0, 1] f is concave if f is convex f(θx + (1 θ)y) θf(x) + (1 θ)f(y) x x convex concave neither x examples (on
More informationLecture: Convex Optimization Problems
1/36 Lecture: Convex Optimization Problems http://bicmr.pku.edu.cn/~wenzw/opt-2015-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/36 optimization
More informationLinear and non-linear programming
Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)
More informationIntroduction to Nonlinear Stochastic Programming
School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS
More informationLMI Methods in Optimal and Robust Control
LMI Methods in Optimal and Robust Control Matthew M. Peet Arizona State University Lecture 02: Optimization (Convex and Otherwise) What is Optimization? An Optimization Problem has 3 parts. x F f(x) :
More informationUNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems
UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction
More informationLMI MODELLING 4. CONVEX LMI MODELLING. Didier HENRION. LAAS-CNRS Toulouse, FR Czech Tech Univ Prague, CZ. Universidad de Valladolid, SP March 2009
LMI MODELLING 4. CONVEX LMI MODELLING Didier HENRION LAAS-CNRS Toulouse, FR Czech Tech Univ Prague, CZ Universidad de Valladolid, SP March 2009 Minors A minor of a matrix F is the determinant of a submatrix
More informationA Brief Review on Convex Optimization
A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review
More informationSemidefinite Programming Basics and Applications
Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent
More information4. Convex optimization problems
Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization
More informationLecture 4: Convex Functions, Part I February 1
IE 521: Convex Optimization Instructor: Niao He Lecture 4: Convex Functions, Part I February 1 Spring 2017, UIUC Scribe: Shuanglong Wang Courtesy warning: These notes do not necessarily cover everything
More informationConvex Optimization M2
Convex Optimization M2 Lecture 8 A. d Aspremont. Convex Optimization M2. 1/57 Applications A. d Aspremont. Convex Optimization M2. 2/57 Outline Geometrical problems Approximation problems Combinatorial
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions
More informationLecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016
Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 1 Entropy Since this course is about entropy maximization,
More informationSemidefinite Programming
Chapter 2 Semidefinite Programming 2.0.1 Semi-definite programming (SDP) Given C M n, A i M n, i = 1, 2,..., m, and b R m, the semi-definite programming problem is to find a matrix X M n for the optimization
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 18
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 18 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 31, 2012 Andre Tkacenko
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions
More informationSymmetric Matrices and Eigendecomposition
Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions
More informationMath 273a: Optimization Basic concepts
Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize
More informationA function(al) f is convex if dom f is a convex set, and. f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) f(x) = x 3
Convex functions The domain dom f of a functional f : R N R is the subset of R N where f is well-defined. A function(al) f is convex if dom f is a convex set, and f(θx + (1 θ)y) θf(x) + (1 θ)f(y) for all
More informationHW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.
HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard
More informationCSCI : Optimization and Control of Networks. Review on Convex Optimization
CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one
More informationNonlinear Programming Algorithms Handout
Nonlinear Programming Algorithms Handout Michael C. Ferris Computer Sciences Department University of Wisconsin Madison, Wisconsin 5376 September 9 1 Eigenvalues The eigenvalues of a matrix A C n n are
More information1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016
AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 8 February 22nd 1 Overview In the previous lecture we saw characterizations of optimality in linear optimization, and we reviewed the
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationConvex optimization problems. Optimization problem in standard form
Convex optimization problems optimization problem in standard form convex optimization problems linear optimization quadratic optimization geometric programming quasiconvex optimization generalized inequality
More informationCS 143 Linear Algebra Review
CS 143 Linear Algebra Review Stefan Roth September 29, 2003 Introductory Remarks This review does not aim at mathematical rigor very much, but instead at ease of understanding and conciseness. Please see
More informationMinimizing Cubic and Homogeneous Polynomials over Integers in the Plane
Minimizing Cubic and Homogeneous Polynomials over Integers in the Plane Alberto Del Pia Department of Industrial and Systems Engineering & Wisconsin Institutes for Discovery, University of Wisconsin-Madison
More informationConvex optimization. Javier Peña Carnegie Mellon University. Universidad de los Andes Bogotá, Colombia September 2014
Convex optimization Javier Peña Carnegie Mellon University Universidad de los Andes Bogotá, Colombia September 2014 1 / 41 Convex optimization Problem of the form where Q R n convex set: min x f(x) x Q,
More informationChapter 4: Unconstrained nonlinear optimization
Chapter 4: Unconstrained nonlinear optimization Edoardo Amaldi DEIB Politecnico di Milano edoardo.amaldi@polimi.it Website: http://home.deib.polimi.it/amaldi/opt-15-16.shtml Academic year 2015-16 Edoardo
More informationminimize x x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x 2 u 2, 5x 1 +76x 2 1,
4 Duality 4.1 Numerical perturbation analysis example. Consider the quadratic program with variables x 1, x 2, and parameters u 1, u 2. minimize x 2 1 +2x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x
More informationCOURSE ON LMI PART I.2 GEOMETRY OF LMI SETS. Didier HENRION henrion
COURSE ON LMI PART I.2 GEOMETRY OF LMI SETS Didier HENRION www.laas.fr/ henrion October 2006 Geometry of LMI sets Given symmetric matrices F i we want to characterize the shape in R n of the LMI set F
More information4. Convex optimization problems
Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization
More informationConvex Optimization in Classification Problems
New Trends in Optimization and Computational Algorithms December 9 13, 2001 Convex Optimization in Classification Problems Laurent El Ghaoui Department of EECS, UC Berkeley elghaoui@eecs.berkeley.edu 1
More informationMathematical Preliminaries
Chapter 33 Mathematical Preliminaries In this appendix, we provide essential definitions and key results which are used at various points in the book. We also provide a list of sources where more details
More informationConvex Functions and Optimization
Chapter 5 Convex Functions and Optimization 5.1 Convex Functions Our next topic is that of convex functions. Again, we will concentrate on the context of a map f : R n R although the situation can be generalized
More information15. Conic optimization
L. Vandenberghe EE236C (Spring 216) 15. Conic optimization conic linear program examples modeling duality 15-1 Generalized (conic) inequalities Conic inequality: a constraint x K where K is a convex cone
More informationLecture 2: Convex Sets and Functions
Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationLecture 6 - Convex Sets
Lecture 6 - Convex Sets Definition A set C R n is called convex if for any x, y C and λ [0, 1], the point λx + (1 λ)y belongs to C. The above definition is equivalent to saying that for any x, y C, the
More informationConic Linear Programming. Yinyu Ye
Conic Linear Programming Yinyu Ye December 2004, revised January 2015 i ii Preface This monograph is developed for MS&E 314, Conic Linear Programming, which I am teaching at Stanford. Information, lecture
More informationAssignment 1: From the Definition of Convexity to Helley Theorem
Assignment 1: From the Definition of Convexity to Helley Theorem Exercise 1 Mark in the following list the sets which are convex: 1. {x R 2 : x 1 + i 2 x 2 1, i = 1,..., 10} 2. {x R 2 : x 2 1 + 2ix 1x
More information3. Linear Programming and Polyhedral Combinatorics
Massachusetts Institute of Technology 18.433: Combinatorial Optimization Michel X. Goemans February 28th, 2013 3. Linear Programming and Polyhedral Combinatorics Summary of what was seen in the introductory
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More informationAppendix PRELIMINARIES 1. THEOREMS OF ALTERNATIVES FOR SYSTEMS OF LINEAR CONSTRAINTS
Appendix PRELIMINARIES 1. THEOREMS OF ALTERNATIVES FOR SYSTEMS OF LINEAR CONSTRAINTS Here we consider systems of linear constraints, consisting of equations or inequalities or both. A feasible solution
More informationLecture: Examples of LP, SOCP and SDP
1/34 Lecture: Examples of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:
More informationLecture 1 Introduction
L. Vandenberghe EE236A (Fall 2013-14) Lecture 1 Introduction course overview linear optimization examples history approximate syllabus basic definitions linear optimization in vector and matrix notation
More informationComputational Optimization. Mathematical Programming Fundamentals 1/25 (revised)
Computational Optimization Mathematical Programming Fundamentals 1/5 (revised) If you don t know where you are going, you probably won t get there. -from some book I read in eight grade If you do get there,
More informationMaximum likelihood estimation
Maximum likelihood estimation Guillaume Obozinski Ecole des Ponts - ParisTech Master MVA Maximum likelihood estimation 1/26 Outline 1 Statistical concepts 2 A short review of convex analysis and optimization
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationOptimization WS 13/14:, by Y. Goldstein/K. Reinert, 9. Dezember 2013, 16: Linear programming. Optimization Problems
Optimization WS 13/14:, by Y. Goldstein/K. Reinert, 9. Dezember 2013, 16:38 2001 Linear programming Optimization Problems General optimization problem max{z(x) f j (x) 0,x D} or min{z(x) f j (x) 0,x D}
More information3. Linear Programming and Polyhedral Combinatorics
Massachusetts Institute of Technology 18.453: Combinatorial Optimization Michel X. Goemans April 5, 2017 3. Linear Programming and Polyhedral Combinatorics Summary of what was seen in the introductory
More informationThe Q-parametrization (Youla) Lecture 13: Synthesis by Convex Optimization. Lecture 13: Synthesis by Convex Optimization. Example: Spring-mass System
The Q-parametrization (Youla) Lecture 3: Synthesis by Convex Optimization controlled variables z Plant distubances w Example: Spring-mass system measurements y Controller control inputs u Idea for lecture
More informationConic Linear Programming. Yinyu Ye
Conic Linear Programming Yinyu Ye December 2004, revised October 2017 i ii Preface This monograph is developed for MS&E 314, Conic Linear Programming, which I am teaching at Stanford. Information, lecture
More informationThe general programming problem is the nonlinear programming problem where a given function is maximized subject to a set of inequality constraints.
1 Optimization Mathematical programming refers to the basic mathematical problem of finding a maximum to a function, f, subject to some constraints. 1 In other words, the objective is to find a point,
More informationEC /11. Math for Microeconomics September Course, Part II Problem Set 1 with Solutions. a11 a 12. x 2
LONDON SCHOOL OF ECONOMICS Professor Leonardo Felli Department of Economics S.478; x7525 EC400 2010/11 Math for Microeconomics September Course, Part II Problem Set 1 with Solutions 1. Show that the general
More informationLecture 5. 1 Goermans-Williamson Algorithm for the maxcut problem
Math 280 Geometric and Algebraic Ideas in Optimization April 26, 2010 Lecture 5 Lecturer: Jesús A De Loera Scribe: Huy-Dung Han, Fabio Lapiccirella 1 Goermans-Williamson Algorithm for the maxcut problem
More informationLinear Regression and Its Applications
Linear Regression and Its Applications Predrag Radivojac October 13, 2014 Given a data set D = {(x i, y i )} n the objective is to learn the relationship between features and the target. We usually start
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationLinear Algebra Review: Linear Independence. IE418 Integer Programming. Linear Algebra Review: Subspaces. Linear Algebra Review: Affine Independence
Linear Algebra Review: Linear Independence IE418: Integer Programming Department of Industrial and Systems Engineering Lehigh University 21st March 2005 A finite collection of vectors x 1,..., x k R n
More informationLecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima
B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationLecture Note 5: Semidefinite Programming for Stability Analysis
ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State
More informationc 2000 Society for Industrial and Applied Mathematics
SIAM J. OPIM. Vol. 10, No. 3, pp. 750 778 c 2000 Society for Industrial and Applied Mathematics CONES OF MARICES AND SUCCESSIVE CONVEX RELAXAIONS OF NONCONVEX SES MASAKAZU KOJIMA AND LEVEN UNÇEL Abstract.
More informationApplications of Linear Programming
Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal
More information10725/36725 Optimization Homework 2 Solutions
10725/36725 Optimization Homework 2 Solutions 1 Convexity (Kevin) 1.1 Sets Let A R n be a closed set with non-empty interior that has a supporting hyperplane at every point on its boundary. (a) Show that
More informationIntroduction and Math Preliminaries
Introduction and Math Preliminaries Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Appendices A, B, and C, Chapter
More informationNotes, March 4, 2013, R. Dudley Maximum likelihood estimation: actual or supposed
18.466 Notes, March 4, 2013, R. Dudley Maximum likelihood estimation: actual or supposed 1. MLEs in exponential families Let f(x,θ) for x X and θ Θ be a likelihood function, that is, for present purposes,
More informationScientific Computing: Optimization
Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture
More informationFRTN10 Multivariable Control, Lecture 13. Course outline. The Q-parametrization (Youla) Example: Spring-mass System
FRTN Multivariable Control, Lecture 3 Anders Robertsson Automatic Control LTH, Lund University Course outline The Q-parametrization (Youla) L-L5 Purpose, models and loop-shaping by hand L6-L8 Limitations
More informationFundamentals of Unconstrained Optimization
dalmau@cimat.mx Centro de Investigación en Matemáticas CIMAT A.C. Mexico Enero 2016 Outline Introduction 1 Introduction 2 3 4 Optimization Problem min f (x) x Ω where f (x) is a real-valued function The
More informationJensen s inequality for multivariate medians
Jensen s inequality for multivariate medians Milan Merkle University of Belgrade, Serbia emerkle@etf.rs Given a probability measure µ on Borel sigma-field of R d, and a function f : R d R, the main issue
More informationLecture Unconstrained optimization. In this lecture we will study the unconstrained problem. minimize f(x), (2.1)
Lecture 2 In this lecture we will study the unconstrained problem minimize f(x), (2.1) where x R n. Optimality conditions aim to identify properties that potential minimizers need to satisfy in relation
More informationInterior Point Methods: Second-Order Cone Programming and Semidefinite Programming
School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods: Second-Order Cone Programming and Semidefinite Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio
More informationStrong Formulations for Convex Functions over Non-Convex Domains
Strong Formulations for Convex Functions over Non-Convex Domains Daniel Bienstock and Alexander Michalka University JMM 2013 Introduction Generic Problem: min Q(x), s.t. x F, Introduction Generic Problem:
More informationWhat can be expressed via Conic Quadratic and Semidefinite Programming?
What can be expressed via Conic Quadratic and Semidefinite Programming? A. Nemirovski Faculty of Industrial Engineering and Management Technion Israel Institute of Technology Abstract Tremendous recent
More informationIE 521 Convex Optimization
Lecture 1: 16th January 2019 Outline 1 / 20 Which set is different from others? Figure: Four sets 2 / 20 Which set is different from others? Figure: Four sets 3 / 20 Interior, Closure, Boundary Definition.
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationComputational Optimization. Convexity and Unconstrained Optimization 1/29/08 and 2/1(revised)
Computational Optimization Convexity and Unconstrained Optimization 1/9/08 and /1(revised) Convex Sets A set S is convex if the line segment joining any two points in the set is also in the set, i.e.,
More informationConvex envelopes, cardinality constrained optimization and LASSO. An application in supervised learning: support vector machines (SVMs)
ORF 523 Lecture 8 Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Any typos should be emailed to a a a@princeton.edu. 1 Outline Convexity-preserving operations Convex envelopes, cardinality
More informationDO NOT OPEN THIS QUESTION BOOKLET UNTIL YOU ARE TOLD TO DO SO
QUESTION BOOKLET EECS 227A Fall 2009 Midterm Tuesday, Ocotober 20, 11:10-12:30pm DO NOT OPEN THIS QUESTION BOOKLET UNTIL YOU ARE TOLD TO DO SO You have 80 minutes to complete the midterm. The midterm consists
More informationSolution to EE 617 Mid-Term Exam, Fall November 2, 2017
Solution to EE 67 Mid-erm Exam, Fall 207 November 2, 207 EE 67 Solution to Mid-erm Exam - Page 2 of 2 November 2, 207 (4 points) Convex sets (a) (2 points) Consider the set { } a R k p(0) =, p(t) for t
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationLecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima
B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained
More informationThe proximal mapping
The proximal mapping http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes Outline 2/37 1 closed function 2 Conjugate function
More informationModule 04 Optimization Problems KKT Conditions & Solvers
Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September
More informationCSCI 1951-G Optimization Methods in Finance Part 10: Conic Optimization
CSCI 1951-G Optimization Methods in Finance Part 10: Conic Optimization April 6, 2018 1 / 34 This material is covered in the textbook, Chapters 9 and 10. Some of the materials are taken from it. Some of
More information7. Statistical estimation
7. Statistical estimation Convex Optimization Boyd & Vandenberghe maximum likelihood estimation optimal detector design experiment design 7 1 Parametric distribution estimation distribution estimation
More informationELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications
ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March
More informationAdditional Homework Problems
Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.
More informationAlgorithmic Convex Geometry
Algorithmic Convex Geometry August 2011 2 Contents 1 Overview 5 1.1 Learning by random sampling.................. 5 2 The Brunn-Minkowski Inequality 7 2.1 The inequality.......................... 8 2.1.1
More information