Lecture 2: Convex Sets and Functions
|
|
- Lucas Bradley
- 6 years ago
- Views:
Transcription
1 Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall / 22
2 Optimization Problems Optimization problems are generally expressed as min f(x) subject to x X, where X is a subset of R n. The function f : X R is called the objective function of the optimization problem, and the condition x X is called the constraint(s) of the problem. The set X is called the feasible set, and a vector x is said to be feasible if x X. The goal is to find a vector x X that minimizes the objective function. When the objective function or the feasible set is nonlinear, the optimization problem is called a nonlinear optimization problem or nonlinear program. If both the objective function and feasible set are linear, it is called a linear optimization problem or linear program. Lecture 2 Network Optimization, Fall / 22
3 Convex Optimization When the objective function and the feasible set are both convex (We will soon study the convexity of sets and functions), the optimization problem is called a convex optimization problem or convex program, and it can be solved efficiently (under some conditions). Examples Linear regression: min w Xw y, where (X, y) is the measurement setup and observed data pair, and x is the vector of (linear) model parameters. Classification (logistic regression or SVM): min w n log(1 + exp( y i x T i w)) or w 2 + C n ξ s.t. ξ 1 y i x T i w, ξ 0. Lecture 2 Network Optimization, Fall / 22
4 Examples Examples (contd.) Maximum likelihood estimation: k-means: max θ min µ 1,...,µ k n log p θ (x i ) k j=1 i C j x i µ j 2 Lecture 2 Network Optimization, Fall / 22
5 Course Outline Convexity of sets and functions Unconstrained optimization Optimality conditions and algorithms Constrained optimization Optimality conditions and algorithms Lagrange multiplier theory Lagrange multiplier algorithms Duality and convex programming Dual methods First order methods for large scale problems Robust optimization Lecture 2 Network Optimization, Fall / 22
6 Convex Sets A convex combination of points x 1,..., x m R n is defined as α 1 x α m x m where α α m = 1 and α i 0, i. Note that convex combinations of two points are the line segment connecting the two points. Definition of Convex Set A subset C of R n is said to be convex if αx + (1 α)y C, x, y C, α [0, 1]. For a convex set C and any two points in C, the line segment connecting the two points is contained in C. Examples on the board Lecture 2 Network Optimization, Fall / 22
7 Illustration of Convex Sets αx + (1 α)y, 0 α 1 x y x y x x y y Figure : Examples of convex/nonconvex sets ([Bertsekas, p.688]) Lecture 2 Network Optimization, Fall / 22
8 Convex Functions Definition of Convex Function Let C be a convex subset of R n. A function f : C R is said to be convex if f(αx + (1 α)y) αf(x) + (1 α)f(y), x, y C, α [0, 1]. The function f is called concave if f is convex. The function f is called strictly convex if the above inequality is strict for all x, y C with x y, and all α (0, 1). Examples Convex functions: x 2, e x Concave functions: log(x), x Neither convex nor concave: x 3 Lecture 2 Network Optimization, Fall / 22
9 Illustration of Convex Functions f(y) "#*% f(x) "#$% αf(x)!"#$%&'&#(&)&!%"#*% + (1 α)f(y) f ( αx "#!$&'&#(&)&!%*% + (1 α)y ) x$ αx!$&'&#(&)&!%* + (1 α)y y* C+ Figure : Illustration of definition of convex function ([Bertsekas, p.689]) Lecture 2 Network Optimization, Fall / 22
10 Convex Sets and Functions Proposition (a) For any collection {C i i I} of convex sets, the set intersection i I C i is convex. (b) The vector sum of two convex sets C 1 and C 2 is convex. (c) The image of a convex set under a linear transformation is convex. (d) For a convex set C and convex function f : C R, the level sets {x C f(x) α} and {x C f(x) < α} are convex for all scalars α. Exercises Is the union of convex sets convex? Lecture 2 Network Optimization, Fall / 22
11 Convex Functions Jensen s Inequality For a convex set C and a convex function f : C R, (a) ( m ) m f α i x i α i f(x i ), x 1,..., x m, α 1,..., α m 0 and m α i = 1. (b) ( f C ) xw(x)dx f(x)w(x)dx, C where w : C R + such that w(x)dx = 1. C Exercises Apply Jensen s inequality to prove that (x 1 x n ) 1/n x1+ +xn n nonnegative numbers x 1,..., x n. (Hint. Use the convexity of e x ) for Lecture 2 Network Optimization, Fall / 22
12 Jensen s Inequality Part (b) can be viewed as an extension of part (a) in the sense that the nonnegative weight function w corresponds to the nonnegative weight α and it integrates to 1, which is an analogy to the condition that the weights α i add up to 1. Jensen s inequality is one of the most used inequalities in applied mathematics and probability theory. For a random variable X and a convex function f, Part (b) leads to f(e[x]) E[f(X)] where E denotes the expectation. For example, we have E[X 2 ] E[X] 2 which is clear from the definition of variance. Part (a) can be proved by mathematical induction (assume WLOG α i > 0, i) ( m ) m 1 α f α i x i α mf(x n) + (1 α m)f i x i (why?) 1 α m m 1 α i α mf(x n) + (1 α m) f (x i ) (by induction hypothesis) 1 α m m = α i f(x i ) Lecture 2 Network Optimization, Fall / 22
13 Convex Functions Properties of Convex Function (a) A linear function is convex. (b) Any vector norm is convex. (c) The weighted sum of convex functions, with positive weights, is convex. (d) If I is an index set, C is a convex subset of R n, and f i : C R is convex for each i I, then the function h : C (, ] defined by is also convex. h(x) = sup f i (x) i I Exercises Let f(x) = 2x + 3 and g(x) = 2x + 1. Draw the function h(x) = max{f(x), g(x)} and check the convexity. Lecture 2 Network Optimization, Fall / 22
14 Differentiable Convex Functions Characterization of Differentiable Convex Functions Let C be a convex subset of R n and let f : R n R be differentiable over R n. (a) f is convex over C if and only if f(z) f(x) + (z x) T f(x), x, z C. (b) f is strictly convex over C if and only if the above inequality is strict whenever x z. Proof: ( ) Assume the inequality holds. Choose any x, y C and α [0, 1], and let z = αx + (1 α)y. Using the inequality twice, we obtain f(x) f(z) + (x z) T f(z), f(y) f(z) + (y z) T f(z). Multiply the first inequality by α, the second by (1 α), and add them to obtain αf(x) + (1 α)f(y) f(z) + (αx + (1 α)y z) T f(z) = z, which proves that f is convex. Conversely, assume that f is convex, let x and z be any vectors in C with x z, and for α (0, 1), consider the function f(x + α(z x)) f(x) g(α) =, α (0, 1]. α Lecture 2 Network Optimization, Fall / 22
15 Differentiable Convex Functions Proof (contd.): Consider any α 1, α 2, with 0 < α 1 < α 2 < 1, and let ᾱ = α 1 α 2, z = x + α 2 (z x). We have f(x + ᾱ( z x)) f(x) f(x + ᾱ( z x)) = ᾱf( z) + (1 ᾱ)f(x) ( z) f(x) ᾱ Note that the above inequality is strict if f is strictly convex. It follows from the above inequality that f(x + α 1 (z x)) f(x) α 1 f(x + α 2(z x)) f(x) α 2 g(α 1 ) g(α 2 ), with strict inequality if f is convex. This shows that g is an monotonically increasing function with α and thus, (z x) T f(x) = lim α 0 g(α) g(1) = f(z) f(x), which is the desired inequality. Exercises Check the above characterization by drawing convex function x 2 and its outer linear approximation. Lecture 2 Network Optimization, Fall / 22
16 Second Order Characterization Conditions for Convexity Let C be a convex subset of R n and let f : R n R be twice continuously differentiable over R n. (a) If 2 f(x) is positive semidefinite for all x C, then f is convex over C. (b) If 2 f(x) is positive definite for all x C, then f is strictly convex over C. (c) If C is open and f is convex over C, then 2 f(x) is positive semidefinite for all x C. (d) If f(x) = x T Qx, where Q is a symmetric matrix, then f is convex if and only if Q is positive semidefinite. Furthermore, f is strictly convex if and only if Q is positive definite. Proof of (a): For all x, y C, we have f(y) = f(x) + (y x) T f(x) (y x)t 2 f(x + α(y x))(y x) for some α [0, 1]. By the positive semidefiniteness of 2 f, we have f(y) = f(x) + (y x) T f(x), which shows that f is convex over C. Lecture 2 Network Optimization, Fall / 22
17 Second Order Characterization Exercises Find an example of a strictly convex function f such that the Hessian 2 f is not positive definite. This shows that the converse of part (b) is not true in general. Check whether the following functions are (strictly) convex (a) f(x 1, x 2) = x x 2 2 over R 2 (b) f(x) = log(x) over R ++ (c) f(x, y, z) = e x+y+z (d) f(x, y, z) = e x2 +y+z log(x + y) + 3 z2 Lecture 2 Network Optimization, Fall / 22
18 Strong Convexity We now consider a strengthened form of convexity, which is the key property in proving the linear convergence of first order method for solving the convex optimization problem. A continuously differentiable function f : C R, where C is a convex set, is said to be strongly convex if for some α > 0, ( f(x) f(y)) T (x y) α x y 2, x, y C. Strong Convexity Let C be an open convex subset of R n, and let f : C R be a function that is continuously differentiable over C. If f is strongly convex, then f is strictly convex. Furthermore, if f is twice continuously differentiable over C, then f satisfies the strong convexity condition if and only if the matrix 2 f(x) αi, where I is the identity, is positive definite for every x C. Lecture 2 Network Optimization, Fall / 22
19 Exercises Assume that is l 2 -norm in this slide. Consider a twice continuously differentiable function f defined over an open convex set C. Prove that for some positive constant α the following are equivalent: (a) f(x) α 2 x 2 is convex. (b) f(cx + (1 c)y) cf(x) + (1 c)f(y) c(1 c)α 2 x y 2, c [0, 1]. Prove that f is strongly convex if either (a) or (b) holds. Lecture 2 Network Optimization, Fall / 22
20 Convex and Affine Hull Let X be a subset of R n. Recall that a convex combination of elements of X is a vector of the form m α ix i, where x 1,..., x m X and α 1,..., α m are nonnegative and add up to 1. The convex hull of X, denoted conv(x), is the set of all convex combinations of elements of X. In particular, if X contains a finite number of elements x 1,...x m, then { m } m conv(x) = α i x i α i 0, i = 1,..., m, α i = 1. Caratheodory s Theorem Let X be a subset of R n. Every element of conv(x) can be represented as a convex combination of no more than n + 1 elements of X. Lecture 2 Network Optimization, Fall / 22
21 Local and Global Minima Let X R n and let f : X R be a function. A vector x X is called a local minimum of f if there exists some ɛ > 0 such that f(x) f(y) for every y X satisfying x y ɛ, where is some vector norm. A vector x is called a global minimum if f(x) f(y), y X. Under convexity assumptions, local minimum is equivalent to global minimum. Local Min = Global Min under Convexity If C is a convex subset of R n and f : C R is a convex function, then a local minimum of f is also a global minimum. If in addition f is strictly convex, then there exists at most one global minimum of f. Lecture 2 Network Optimization, Fall / 22
22 Projection Theorem Projection Theorem Let C be a closed convex set and let be the Euclidean norm. (a) For every x R n, there exists a unique vector z C that minimizes z x over all z C. This vector is called the projection of x on C, and is denoted by [x] +, i.e., [x] + = arg min z x. z C (b) Given some x R n, a vector z C is equal to [x] + if and only if (y z) T (x z) 0, y C. (c) The mapping f : R n C defined by f(x) = [x] + is continuous and nonexpansive, i.e., [x] + [y] + x y, x, y R n. Lecture 2 Network Optimization, Fall / 22
Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationLECTURE 3 LECTURE OUTLINE
LECTURE 3 LECTURE OUTLINE Differentiable Conve Functions Conve and A ne Hulls Caratheodory s Theorem Reading: Sections 1.1, 1.2 All figures are courtesy of Athena Scientific, and are used with permission.
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More information14 Lecture 14 Local Extrema of Function
14 Lecture 14 Local Extrema of Function 14.1 Taylor s Formula with Lagrangian Remainder Term Theorem 14.1. Let n N {0} and f : (a,b) R. We assume that there exists f (n+1) (x) for all x (a,b). Then for
More informationStatic Problem Set 2 Solutions
Static Problem Set Solutions Jonathan Kreamer July, 0 Question (i) Let g, h be two concave functions. Is f = g + h a concave function? Prove it. Yes. Proof: Consider any two points x, x and α [0, ]. Let
More informationLinear and non-linear programming
Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationSubgradient. Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes. definition. subgradient calculus
1/41 Subgradient Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes definition subgradient calculus duality and optimality conditions directional derivative Basic inequality
More informationLecture Note 5: Semidefinite Programming for Stability Analysis
ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State
More informationConvex optimization COMS 4771
Convex optimization COMS 4771 1. Recap: learning via optimization Soft-margin SVMs Soft-margin SVM optimization problem defined by training data: w R d λ 2 w 2 2 + 1 n n [ ] 1 y ix T i w. + 1 / 15 Soft-margin
More informationN. L. P. NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP. Optimization. Models of following form:
0.1 N. L. P. Katta G. Murty, IOE 611 Lecture slides Introductory Lecture NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP does not include everything
More informationConvex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationConvex Analysis and Optimization Chapter 2 Solutions
Convex Analysis and Optimization Chapter 2 Solutions Dimitri P. Bertsekas with Angelia Nedić and Asuman E. Ozdaglar Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationIntroduction to Convex Analysis Microeconomics II - Tutoring Class
Introduction to Convex Analysis Microeconomics II - Tutoring Class Professor: V. Filipe Martins-da-Rocha TA: Cinthia Konichi April 2010 1 Basic Concepts and Results This is a first glance on basic convex
More informationMATHEMATICAL ECONOMICS: OPTIMIZATION. Contents
MATHEMATICAL ECONOMICS: OPTIMIZATION JOÃO LOPES DIAS Contents 1. Introduction 2 1.1. Preliminaries 2 1.2. Optimal points and values 2 1.3. The optimization problems 3 1.4. Existence of optimal points 4
More informationOptimization and Optimal Control in Banach Spaces
Optimization and Optimal Control in Banach Spaces Bernhard Schmitzer October 19, 2017 1 Convex non-smooth optimization with proximal operators Remark 1.1 (Motivation). Convex optimization: easier to solve,
More informationSpeech Recognition Lecture 7: Maximum Entropy Models. Mehryar Mohri Courant Institute and Google Research
Speech Recognition Lecture 7: Maximum Entropy Models Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.com This Lecture Information theory basics Maximum entropy models Duality theorem
More informationHandout 2: Elements of Convex Analysis
ENGG 5501: Foundations of Optimization 2018 19 First Term Handout 2: Elements of Convex Analysis Instructor: Anthony Man Cho So September 10, 2018 As briefly mentioned in Handout 1, the notion of convexity
More informationOptimization Theory. Lectures 4-6
Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x$0} Existence of a maximum:
More informationLECTURE SLIDES ON BASED ON CLASS LECTURES AT THE CAMBRIDGE, MASS FALL 2007 BY DIMITRI P. BERTSEKAS.
LECTURE SLIDES ON CONVEX ANALYSIS AND OPTIMIZATION BASED ON 6.253 CLASS LECTURES AT THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY CAMBRIDGE, MASS FALL 2007 BY DIMITRI P. BERTSEKAS http://web.mit.edu/dimitrib/www/home.html
More informationCS-E4830 Kernel Methods in Machine Learning
CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More information1/37. Convexity theory. Victor Kitov
1/37 Convexity theory Victor Kitov 2/37 Table of Contents 1 2 Strictly convex functions 3 Concave & strictly concave functions 4 Kullback-Leibler divergence 3/37 Convex sets Denition 1 Set X is convex
More informationModule 04 Optimization Problems KKT Conditions & Solvers
Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September
More informationLecture 1: January 12
10-725/36-725: Convex Optimization Fall 2015 Lecturer: Ryan Tibshirani Lecture 1: January 12 Scribes: Seo-Jin Bang, Prabhat KC, Josue Orellana 1.1 Review We begin by going through some examples and key
More informationLECTURE SLIDES ON CONVEX OPTIMIZATION AND DUALITY THEORY TATA INSTITUTE FOR FUNDAMENTAL RESEARCH MUMBAI, INDIA JANUARY 2009 PART I
LECTURE SLIDES ON CONVEX OPTIMIZATION AND DUALITY THEORY TATA INSTITUTE FOR FUNDAMENTAL RESEARCH MUMBAI, INDIA JANUARY 2009 PART I BY DIMITRI P. BERTSEKAS M.I.T. http://web.mit.edu/dimitrib/www/home.html
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 4. Subgradient
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 4 Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 2 4.1. Subgradients definition subgradient calculus duality and optimality conditions Shiqian
More informationHamiltonian Mechanics
Chapter 3 Hamiltonian Mechanics 3.1 Convex functions As background to discuss Hamiltonian mechanics we discuss convexity and convex functions. We will also give some applications to thermodynamics. We
More informationHW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.
HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard
More informationStructural and Multidisciplinary Optimization. P. Duysinx and P. Tossings
Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be
More informationAssignment 1: From the Definition of Convexity to Helley Theorem
Assignment 1: From the Definition of Convexity to Helley Theorem Exercise 1 Mark in the following list the sets which are convex: 1. {x R 2 : x 1 + i 2 x 2 1, i = 1,..., 10} 2. {x R 2 : x 2 1 + 2ix 1x
More informationMicroeconomics I. September, c Leopold Sögner
Microeconomics I c Leopold Sögner Department of Economics and Finance Institute for Advanced Studies Stumpergasse 56 1060 Wien Tel: +43-1-59991 182 soegner@ihs.ac.at http://www.ihs.ac.at/ soegner September,
More information8. Geometric problems
8. Geometric problems Convex Optimization Boyd & Vandenberghe extremal volume ellipsoids centering classification placement and facility location 8 Minimum volume ellipsoid around a set Löwner-John ellipsoid
More informationConvex Functions and Optimization
Chapter 5 Convex Functions and Optimization 5.1 Convex Functions Our next topic is that of convex functions. Again, we will concentrate on the context of a map f : R n R although the situation can be generalized
More information13. Convex programming
CS/ISyE/ECE 524 Introduction to Optimization Spring 2015 16 13. Convex programming Convex sets and functions Convex programs Hierarchy of complexity Example: geometric programming Laurent Lessard (www.laurentlessard.com)
More informationLecture 8 Plus properties, merit functions and gap functions. September 28, 2008
Lecture 8 Plus properties, merit functions and gap functions September 28, 2008 Outline Plus-properties and F-uniqueness Equation reformulations of VI/CPs Merit functions Gap merit functions FP-I book:
More informationMVE165/MMG631 Linear and integer optimization with applications Lecture 13 Overview of nonlinear programming. Ann-Brith Strömberg
MVE165/MMG631 Overview of nonlinear programming Ann-Brith Strömberg 2015 05 21 Areas of applications, examples (Ch. 9.1) Structural optimization Design of aircraft, ships, bridges, etc Decide on the material
More informationAppendix PRELIMINARIES 1. THEOREMS OF ALTERNATIVES FOR SYSTEMS OF LINEAR CONSTRAINTS
Appendix PRELIMINARIES 1. THEOREMS OF ALTERNATIVES FOR SYSTEMS OF LINEAR CONSTRAINTS Here we consider systems of linear constraints, consisting of equations or inequalities or both. A feasible solution
More informationLECTURE SLIDES ON CONVEX ANALYSIS AND OPTIMIZATION BASED ON LECTURES GIVEN AT THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY CAMBRIDGE, MASS
LECTURE SLIDES ON CONVEX ANALYSIS AND OPTIMIZATION BASED ON LECTURES GIVEN AT THE MASSACHUSETTS INSTITUTE OF TECHNOLOGY CAMBRIDGE, MASS BY DIMITRI P. BERTSEKAS http://web.mit.edu/dimitrib/www/home.html
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationOptimality Conditions for Nonsmooth Convex Optimization
Optimality Conditions for Nonsmooth Convex Optimization Sangkyun Lee Oct 22, 2014 Let us consider a convex function f : R n R, where R is the extended real field, R := R {, + }, which is proper (f never
More informationMath 273a: Optimization Subgradient Methods
Math 273a: Optimization Subgradient Methods Instructor: Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com Nonsmooth convex function Recall: For ˉx R n, f(ˉx) := {g R
More informationConvex Optimization Theory
Convex Optimization Theory A SUMMARY BY DIMITRI P. BERTSEKAS We provide a summary of theoretical concepts and results relating to convex analysis, convex optimization, and duality theory. In particular,
More informationDefinition of convex function in a vector space
Convex functions Nonlinear optimization Instituto Superior Técnico and Carnegie Mellon University PhD course João Xavier TAs: Brian Swenson, Shanghang Zhang, Lucas Balthazar Convex functions Nonconvex
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationON GAP FUNCTIONS OF VARIATIONAL INEQUALITY IN A BANACH SPACE. Sangho Kum and Gue Myung Lee. 1. Introduction
J. Korean Math. Soc. 38 (2001), No. 3, pp. 683 695 ON GAP FUNCTIONS OF VARIATIONAL INEQUALITY IN A BANACH SPACE Sangho Kum and Gue Myung Lee Abstract. In this paper we are concerned with theoretical properties
More informationConvex Optimization Theory. Chapter 4 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 4 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationConvex Optimization. (EE227A: UC Berkeley) Lecture 4. Suvrit Sra. (Conjugates, subdifferentials) 31 Jan, 2013
Convex Optimization (EE227A: UC Berkeley) Lecture 4 (Conjugates, subdifferentials) 31 Jan, 2013 Suvrit Sra Organizational HW1 due: 14th Feb 2013 in class. Please L A TEX your solutions (contact TA if this
More informationUNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems
UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction
More information4. Convex Sets and (Quasi-)Concave Functions
4. Convex Sets and (Quasi-)Concave Functions Daisuke Oyama Mathematics II April 17, 2017 Convex Sets Definition 4.1 A R N is convex if (1 α)x + αx A whenever x, x A and α [0, 1]. A R N is strictly convex
More informationChapter 2: Preliminaries and elements of convex analysis
Chapter 2: Preliminaries and elements of convex analysis Edoardo Amaldi DEIB Politecnico di Milano edoardo.amaldi@polimi.it Website: http://home.deib.polimi.it/amaldi/opt-14-15.shtml Academic year 2014-15
More informationConvex Analysis and Economic Theory AY Elementary properties of convex functions
Division of the Humanities and Social Sciences Ec 181 KC Border Convex Analysis and Economic Theory AY 2018 2019 Topic 6: Convex functions I 6.1 Elementary properties of convex functions We may occasionally
More informationLecture 3: Functions of Symmetric Matrices
Lecture 3: Functions of Symmetric Matrices Yilin Mo July 2, 2015 1 Recap 1 Bayes Estimator: (a Initialization: (b Correction: f(x 0 Y 1 = f(x 0 f(x k Y k = αf(y k x k f(x k Y k 1, where ( 1 α = f(y k x
More informationMath 273a: Optimization Subgradients of convex functions
Math 273a: Optimization Subgradients of convex functions Made by: Damek Davis Edited by Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com 1 / 42 Subgradients Assumptions
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More information8. Geometric problems
8. Geometric problems Convex Optimization Boyd & Vandenberghe extremal volume ellipsoids centering classification placement and facility location 8 1 Minimum volume ellipsoid around a set Löwner-John ellipsoid
More information1 Strict local optimality in unconstrained optimization
ORF 53 Lecture 14 Spring 016, Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Thursday, April 14, 016 When in doubt on the accuracy of these notes, please cross check with the instructor s
More informationMath 273a: Optimization Subgradients of convex functions
Math 273a: Optimization Subgradients of convex functions Made by: Damek Davis Edited by Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com 1 / 20 Subgradients Assumptions
More informationThe general programming problem is the nonlinear programming problem where a given function is maximized subject to a set of inequality constraints.
1 Optimization Mathematical programming refers to the basic mathematical problem of finding a maximum to a function, f, subject to some constraints. 1 In other words, the objective is to find a point,
More informationConvex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Convex Functions Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Definition convex function Examples
More informationLecture 3: Basics of set-constrained and unconstrained optimization
Lecture 3: Basics of set-constrained and unconstrained optimization (Chap 6 from textbook) Xiaoqun Zhang Shanghai Jiao Tong University Last updated: October 9, 2018 Optimization basics Outline Optimization
More informationEconomics 101A (Lecture 3) Stefano DellaVigna
Economics 101A (Lecture 3) Stefano DellaVigna January 24, 2017 Outline 1. Implicit Function Theorem 2. Envelope Theorem 3. Convexity and concavity 4. Constrained Maximization 1 Implicit function theorem
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions
More informationNewton s Method. Javier Peña Convex Optimization /36-725
Newton s Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationLecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More information1 Directional Derivatives and Differentiability
Wednesday, January 18, 2012 1 Directional Derivatives and Differentiability Let E R N, let f : E R and let x 0 E. Given a direction v R N, let L be the line through x 0 in the direction v, that is, L :=
More informationMathematical Economics: Lecture 16
Mathematical Economics: Lecture 16 Yu Ren WISE, Xiamen University November 26, 2012 Outline 1 Chapter 21: Concave and Quasiconcave Functions New Section Chapter 21: Concave and Quasiconcave Functions Concave
More informationChap 2. Optimality conditions
Chap 2. Optimality conditions Version: 29-09-2012 2.1 Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1
More informationA Brief Review on Convex Optimization
A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationMATH2070 Optimisation
MATH2070 Optimisation Nonlinear optimisation with constraints Semester 2, 2012 Lecturer: I.W. Guo Lecture slides courtesy of J.R. Wishart Review The full nonlinear optimisation problem with equality constraints
More informationLecture 2: Convergence of Random Variables
Lecture 2: Convergence of Random Variables Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Introduction to Stochastic Processes, Fall 2013 1 / 9 Convergence of Random Variables
More informationInequality Constraints
Chapter 2 Inequality Constraints 2.1 Optimality Conditions Early in multivariate calculus we learn the significance of differentiability in finding minimizers. In this section we begin our study of the
More informationLecture 2: Convex functions
Lecture 2: Convex functions f : R n R is convex if dom f is convex and for all x, y dom f, θ [0, 1] f is concave if f is convex f(θx + (1 θ)y) θf(x) + (1 θ)f(y) x x convex concave neither x examples (on
More informationLagrangian Duality Theory
Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual
More informationIntroduction to Nonlinear Stochastic Programming
School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS
More informationLecture 1: Background on Convex Analysis
Lecture 1: Background on Convex Analysis John Duchi PCMI 2016 Outline I Convex sets 1.1 Definitions and examples 2.2 Basic properties 3.3 Projections onto convex sets 4.4 Separating and supporting hyperplanes
More informationIntroduction and Math Preliminaries
Introduction and Math Preliminaries Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Appendices A, B, and C, Chapter
More informationSome Background Math Notes on Limsups, Sets, and Convexity
EE599 STOCHASTIC NETWORK OPTIMIZATION, MICHAEL J. NEELY, FALL 2008 1 Some Background Math Notes on Limsups, Sets, and Convexity I. LIMITS Let f(t) be a real valued function of time. Suppose f(t) converges
More informationCSCI : Optimization and Control of Networks. Review on Convex Optimization
CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one
More informationNonlinear Programming 3rd Edition. Theoretical Solutions Manual Chapter 6
Nonlinear Programming 3rd Edition Theoretical Solutions Manual Chapter 6 Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts 1 NOTE This manual contains
More informationOnline Convex Optimization
Advanced Course in Machine Learning Spring 2010 Online Convex Optimization Handouts are jointly prepared by Shie Mannor and Shai Shalev-Shwartz A convex repeated game is a two players game that is performed
More informationInformation Theory and Communication
Information Theory and Communication Ritwik Banerjee rbanerjee@cs.stonybrook.edu c Ritwik Banerjee Information Theory and Communication 1/8 General Chain Rules Definition Conditional mutual information
More informationSWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm. Convex Optimization. Computing and Software McMaster University
SWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm Convex Optimization Computing and Software McMaster University General NLO problem (NLO : Non Linear Optimization) (N LO) min
More informationICS-E4030 Kernel Methods in Machine Learning
ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationChapter 13. Convex and Concave. Josef Leydold Mathematical Methods WS 2018/19 13 Convex and Concave 1 / 44
Chapter 13 Convex and Concave Josef Leydold Mathematical Methods WS 2018/19 13 Convex and Concave 1 / 44 Monotone Function Function f is called monotonically increasing, if x 1 x 2 f (x 1 ) f (x 2 ) It
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions
More informationJanuary 29, Introduction to optimization and complexity. Outline. Introduction. Problem formulation. Convexity reminder. Optimality Conditions
Olga Galinina olga.galinina@tut.fi ELT-53656 Network Analysis Dimensioning II Department of Electronics Communications Engineering Tampere University of Technology, Tampere, Finl January 29, 2014 1 2 3
More informationSymmetric Matrices and Eigendecomposition
Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions
More informationWeek 4: Calculus and Optimization (Jehle and Reny, Chapter A2)
Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2) Tsun-Feng Chiang *School of Economics, Henan University, Kaifeng, China September 27, 2015 Microeconomic Theory Week 4: Calculus and Optimization
More information8. Constrained Optimization
8. Constrained Optimization Daisuke Oyama Mathematics II May 11, 2018 Unconstrained Maximization Problem Let X R N be a nonempty set. Definition 8.1 For a function f : X R, x X is a (strict) local maximizer
More informationConvex Functions. Pontus Giselsson
Convex Functions Pontus Giselsson 1 Today s lecture lower semicontinuity, closure, convex hull convexity preserving operations precomposition with affine mapping infimal convolution image function supremum
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 18
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 18 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 31, 2012 Andre Tkacenko
More informationEE514A Information Theory I Fall 2013
EE514A Information Theory I Fall 2013 K. Mohan, Prof. J. Bilmes University of Washington, Seattle Department of Electrical Engineering Fall Quarter, 2013 http://j.ee.washington.edu/~bilmes/classes/ee514a_fall_2013/
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationChapter 14 Unconstrained and Constrained Optimization Problems
main def 2011/9/26 15:22 page 480 #511 Chapter 14 Unconstrained and Constrained Optimization Problems Shuguang Cui, Man-Cho Anthony So, and Rui Zhang Texas A&M University, College Station, USA Chinese
More information