Computational Optimization. Convexity and Unconstrained Optimization 1/29/08 and 2/1(revised)
|
|
- Kerry Doyle
- 5 years ago
- Views:
Transcription
1 Computational Optimization Convexity and Unconstrained Optimization 1/9/08 and /1(revised)
2 Convex Sets A set S is convex if the line segment joining any two points in the set is also in the set, i.e., for any x,y S, λx+(1- λ)y S for all 0 λ 1 }. convex not convex convex not convex not convex
3 Proving Convexity Prove {x Ax<=b} is convex. Let x and y be elements of C= x Ax b. For any λ (0,1), A( λx+ (1 λ) y) = λax+ (1 λ) Ay λb+ (1 λ) b= b λx+ (1 λ) y C { }
4 You Try Prove D= {x x 1} is convex.
5 Convex Functions A function f is (strictly) convex on a convex set S, if and only if for any x,y S, f(λx+(1- λ)y)(<) λf(x)+ (1- λ)f(y) for all 0 λ 1. f(λx+(1- λ)y) f(y) f(x) x λx+(1- λ)y y
6 Proving Function Convex Linear functions n f ( x) = w' x = wx where x R i= 1 i n For any x, y R λ (0,1) f( λx+ (1 λ) y) = w'( λx+ (1 λ) y) = λwx ' + (1 λ) wy ' λ f( x) + (1 λ) f( y) i n
7 You Try = f ( x, x ) x x
8 Hint: x is convex Consider any two points x,y and λ (0,1) λx + (1 λ) y = λ x + (1 λ) y + (1 λ) λx + λ(1 λ) y ( ) ( ) ( λx (1 λ) y) = λx + (1 λ) y λ(1 λ) xy+ (1 λ) λx + λ(1 λ) y = λx + (1 λ) y + λ(1 λ)( x y) + λx = λ x + λ λx First line uses (1 ) and similarly for (1 λ). Second line completes the square of λ + (1 λ). Third line observes the remaining terms are a square. Fouth line follows since λ(1 λ)( ) 0. x y x y y
9 Handy Facts Let g1( x),, gm( x) be convex functions And a>0. m Then f ( x) = g i ( x) is convex. i= 1 And hx ( ) = ag( x) 1 is convex.
10 Convexity and Curvature Convex functions have positive curvature everywhere. Curvature can be measured by the second derivative or Hessian. Properties of the Hessian indicate if a function is convex or not.
11 Convex Functions A function f is (strictly) convex on a convex set S, if and only if for any x,y S, f(λx+(1- λ)y)(<) λf(x)+ (1- λ)f(y) for all 0 λ 1. f(x+(1- λ)y) f(y) f(x) x λx+(1- λ)y y
12 Theorem Let f be twice continuously differentiable. f(x) is convex on S if and only if for all x S, the Hessian at x f ( x) is positive semi-definite.
13 Definition The matrix H is positive semi-definite (p.s.d.) if and only if for any vector y yhy 0 The matrix H is positive definite (p.d.) if and only if for any nonzero vector y yhy > 0 Similarly for negative (semi-) definite.
14 Theorem Let f be twice continuously differentiable. f(x) is strictly convex on S if and only if for all x X, the Hessian at x f ( x) is positive definite.
15 Checking Matrix H is p.s.d/p.d. Manually 4 1 x x x = 4x x x x x + 3x [ ] x = 4x x x + 3x 1 1 = ( x x )^+ 3x + x > 0 [ x x ] , so matrix is positive definite
16 Useful facts The sum of convex functions is convex The composition of convex functions is convex.
17 via eigenvalues The eigenvalues of 4 1 are and so matrix is positive definite
18 Summary: using eigenvalues If all eigenvalues are positive, then matrix is positive definite, p.d. If all eigenvalues are nonnegative, then matrix is positive semi-definite, p.s.d If all eigenvalues are negative, then matrix is negative definite, n.d. If all eigenvalues are nonpositive, then matrix is negative semi-definite, n.s.d Otherwise the matrix is indefinite.
19 Try with Hessians f( x, x ) = x + x 1 1 x 1 f( x) = 4x 0 f( x) = a a b 0 4 b = a + b > for any ab Strictly Convex [ ] 4 0 [ ] 0
20 Check Hessian H = [ 0; 0 4] Eigs(H) are and 4 So Hessian matrix is always p.d. So function is strictly convex
21 Differentiability and Convexity For convex function, linear approximation underestimates function f(x) (x*,f(x*)) g() x = f (*) x + ( x x*) f (*) x
22 Theorem Assume f is continuously differentiable on a Set S. f is convex on S if and only if f( y) f( x) + ( y x)' f ( x) x, y S
23 Theorem y Consider problem min f(x) unconstrained. If f ( x) = 0 and f is convex, then x is a global minimum. Proof: f( y) f( x) + ( y x)' f ( x) by convexity of f = f( x) since f( x) = 0.
24 Unconstrained Optimality Basic Problem: Conditions (1) min f ( x) x S Where S is an open set e.g. R n
25 First Order Necessary Conditions Theorem: Let f be continuously differentiable. If x* is a local minimizer of (1), then f ( x*) = 0
26 Stationary Points Note that this condition is not sufficient f ( x*) = 0 Also true for local max and saddle points
27 Proof Assume false, e.g., f(*) x 0 Let d = f( x*), then f ( x* + λd) = f( x*) + λd f( x*) + λd α( x*, λd) f( x* + λd) f( x*) λ = d f(*) x + d α(*, x λd) f( x* + λd) f( x*) < 0 for λsufficiently small since d f( x*) < 0and α( x*, λd) 0. CONTRADICTION!! x* is a local min.
28 Second Order Sufficient Conditions Theorem: Let f be twice continuously differentiable. If f ( x*) = 0 and f ( x*) is positive definite then x* is a strict local minimizer of (1).
29 Proof Any point x in neighborhood of x* can be written as x*+λd for some vector d with norm 1 and λ<= λ*. Since f is twice continuously differentiable, we can choose λ* such that f( ε )is p.d. for all ε such that ε x* λ* 1 d, λ λ*, f( x* + λd) = f( x*) + λd f( x*) + λ d' f( ε) d since f ( x*) = 0 1 f( x* + λd) f( x*) = λ d' f( x*) d > 0 therefore x* is a strict local min.
30 Second Order Necessary Conditions Theorem: Let f be twice continuously differentiable. If x* is a local minimizer of (1) then f ( x*) = 0 f ( x*) is positive semi definite
31 Proof by contradiction d Assume false, namely there exists some d such that f x d < (*) 0 then f x + λd = f x + λd f x 1 + λ d f x d+ λd α x λd ( * ) (*) (*) ' (*) (*, ) ( * + λ ) ( *) 1 = d ' f ( x *) d+ d α ( x *, λd ) f x d f x λ f( x* + λd) f( x*) < 0 for λsufficiently small since d f( x*) d < 0for some d and α( x*, λd) 0. Contradiction!!! x* is a local min.
32 Example Say we are minimizing 1 f ( x, x ) = x xx + x 15x 4x [8,]???
33 Solve FONC Solve FONC to find stationary point * fxx = 15 = 1 x1 ( 1, ) x x1 * x * = = 4 4
34 Check SOSC The Hessian at x* 1 fx ( 1*, x*) = 1 4 is p.d. since the eigenvalues, and 1.88, are positive. Therefore SOSC are satisfied. x* is a strictly local min;
35 Alternative Argument The Hessian at every value x is 1 fxx ( 1, ) = 1 4 which is p.d. since the eigenvalues, and 1.88, are positive. Therefore the function is strictly convex. Since f(x*)=0 and f is a strictly convex, x* is the unique strict global minimum.
36 You Try Use FONC and matlab to find solution of min f ( x, x, x ) = 10x + 5x + 3x x x + x x 4x x + x min f ( x, x, x ) = 10x + 5x 3x x x + x x 4x x + x Are SOSC satisfied? Are SONC? Is f convex?
37 Optimality Conditions for 1- dimen. functions First Order Necessary Condition If x* is a local min then f (x*)=0. If f (x*)=0 then??????????
38 nd Derivatives - 1D Case Sufficient conditions If f (x*)=0 and f (x*) >0, then x* is a strict local min. If f (x*)=0 and f (x*) <0, then x* is a strict local max. Necessary conditions If x* is a local min, then f (x*)=0 and f (x*) >=0. If x* is a local max, then f (x*)=0 and f (x*) <=0.
39 Optimality Conditions for function of R n First Order Necessary Condition If x* is a local min then f ( x*) = 0 If f ( x*) = 0 then??????????
40 Second Order Conditions Sufficient conditions If f ( x*) = 0 and f (*) x is p.d. then x* is a strict local min. If f ( x*) = 0 and f (*) x is n.d. then x* is a strict local max.
41 Second Order Conditions Necessary conditions If x* is a local min, then f ( x*) = 0 and f (*) x is p.s.d. If x* is a local max, then f ( x*) = 0 and f (*) x is n.s.d.
42 Optimality Conditions for Convex Convexity Let f be continuously differentiable convex function, x* is a global minimum of f if and only if f ( x*) = 0 Let f be continuously differentiable strictly convex function, If f ( x*) = 0 then x* is the unique global minimum of f. Works similarly for max and concave.
43 Line Search Assume f function maps the vector f to a n scalar: f : R R n Current point is x R Have interval Want to find: λ [ ab, ] λ [,] ab min f( x + λd) = g( λ)
44 Example Say we are minimizing 1 fxx ( 1, ) = x1 xx 1 + x 15x1 4x 1 1 x1 x [ ] 1 = [ xx 1 ] x x Solution is [8 ] Say we are at [ 0, -1] and we want to do a linesearch in direction d=[1 0]
45 Line Search We are at [ 0, -1] and we want to do a linesearch in direction d=[1 0] [8,] d [0,-1] [9/4,-1]
46 Descent Directions If the directional derivative is negative then linesearch will lead to decrease in the function f ( x) d < 0 [8,] [0,-1] f () x d
47 Example continued The exact stepsize can be found 0 1 λ x + λd = λ = 1 λ g = f x + d = ( λ) ( λ ) λ 15λ 4 1 ' λ 15 1 g '( λ) = f '( x + λd) = f ( x) d = = λ + 15 = 0 9 λ = 4 '
48 Example continued So new point is x λ d = = 0 1 f([0,-1]=6 f([9/4, -1])=-46.5 f([8,])=-64 In this case, f ( x + λ d ) is a convex function (verify) so this is a Global min.
49 Example Consider min x x x x Find all points satisfy FONC What can you say about that point based on SONC?
50 FONC The first order necessary conditions are 3x 1 x 1 x x 1 + 4x = 0 = 0 So both (0,0) and (6,9) satisfy FONC
51 Second Order Conditions The Hessian is The Hessian at (6,9) Is indefinite so (6,9) is not a local min (or max). = 4 6 ) ( x x x x x f 18 1 (6,9) 1 4 f =
52 Second Order Conditions The Hessian at (0,0) 0 0 f (0,0) = 0 4 is psd. So this point satisfies the SONC But not the SOSC. It might be a local min, (but in fact it is not try (-ε,0) for any ε>0
53 Do Lab
Computational Optimization. Mathematical Programming Fundamentals 1/25 (revised)
Computational Optimization Mathematical Programming Fundamentals 1/5 (revised) If you don t know where you are going, you probably won t get there. -from some book I read in eight grade If you do get there,
More information1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016
AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 8 February 22nd 1 Overview In the previous lecture we saw characterizations of optimality in linear optimization, and we reviewed the
More informationMath 273a: Optimization Basic concepts
Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize
More informationLecture Unconstrained optimization. In this lecture we will study the unconstrained problem. minimize f(x), (2.1)
Lecture 2 In this lecture we will study the unconstrained problem minimize f(x), (2.1) where x R n. Optimality conditions aim to identify properties that potential minimizers need to satisfy in relation
More informationChapter 1. Optimality Conditions: Unconstrained Optimization. 1.1 Differentiable Problems
Chapter 1 Optimality Conditions: Unconstrained Optimization 1.1 Differentiable Problems Consider the problem of minimizing the function f : R n R where f is twice continuously differentiable on R n : P
More informationOptimization Part 1 P. Agius L2.1, Spring 2008
Optimization Part 1 Contents Terms and definitions, formulating the problem Unconstrained optimization conditions FONC, SONC, SOSC Searching for the solution Which direction? What step size? Constrained
More informationMATHEMATICAL ECONOMICS: OPTIMIZATION. Contents
MATHEMATICAL ECONOMICS: OPTIMIZATION JOÃO LOPES DIAS Contents 1. Introduction 2 1.1. Preliminaries 2 1.2. Optimal points and values 2 1.3. The optimization problems 3 1.4. Existence of optimal points 4
More informationOptimality conditions for unconstrained optimization. Outline
Optimality conditions for unconstrained optimization Daniel P. Robinson Department of Applied Mathematics and Statistics Johns Hopkins University September 13, 2018 Outline 1 The problem and definitions
More informationSymmetric Matrices and Eigendecomposition
Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions
More informationMATH 5720: Unconstrained Optimization Hung Phan, UMass Lowell September 13, 2018
MATH 57: Unconstrained Optimization Hung Phan, UMass Lowell September 13, 18 1 Global and Local Optima Let a function f : S R be defined on a set S R n Definition 1 (minimizers and maximizers) (i) x S
More informationExamination paper for TMA4180 Optimization I
Department of Mathematical Sciences Examination paper for TMA4180 Optimization I Academic contact during examination: Phone: Examination date: 26th May 2016 Examination time (from to): 09:00 13:00 Permitted
More informationECE580 Solution to Problem Set 3: Applications of the FONC, SONC, and SOSC
ECE580 Spring 2016 Solution to Problem Set 3 February 8, 2016 1 ECE580 Solution to Problem Set 3: Applications of the FONC, SONC, and SOSC These problems are from the textbook by Chong and Zak, 4th edition,
More informationLecture 4: Convex Functions, Part I February 1
IE 521: Convex Optimization Instructor: Niao He Lecture 4: Convex Functions, Part I February 1 Spring 2017, UIUC Scribe: Shuanglong Wang Courtesy warning: These notes do not necessarily cover everything
More informationOPER 627: Nonlinear Optimization Lecture 2: Math Background and Optimality Conditions
OPER 627: Nonlinear Optimization Lecture 2: Math Background and Optimality Conditions Department of Statistical Sciences and Operations Research Virginia Commonwealth University Aug 28, 2013 (Lecture 2)
More informationMATH 4211/6211 Optimization Basics of Optimization Problems
MATH 4211/6211 Optimization Basics of Optimization Problems Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 A standard minimization
More informationUNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems
UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction
More information2019 Spring MATH2060A Mathematical Analysis II 1
2019 Spring MATH2060A Mathematical Analysis II 1 Notes 1. CONVEX FUNCTIONS First we define what a convex function is. Let f be a function on an interval I. For x < y in I, the straight line connecting
More informationMATH 4211/6211 Optimization Constrained Optimization
MATH 4211/6211 Optimization Constrained Optimization Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 Constrained optimization
More informationProblem Set 0 Solutions
CS446: Machine Learning Spring 2017 Problem Set 0 Solutions Handed Out: January 25 th, 2017 Handed In: NONE 1. [Probability] Assume that the probability of obtaining heads when tossing a coin is λ. a.
More informationECE580 Fall 2015 Solution to Midterm Exam 1 October 23, Please leave fractions as fractions, but simplify them, etc.
ECE580 Fall 2015 Solution to Midterm Exam 1 October 23, 2015 1 Name: Solution Score: /100 This exam is closed-book. You must show ALL of your work for full credit. Please read the questions carefully.
More informationConvex Analysis and Economic Theory AY Elementary properties of convex functions
Division of the Humanities and Social Sciences Ec 181 KC Border Convex Analysis and Economic Theory AY 2018 2019 Topic 6: Convex functions I 6.1 Elementary properties of convex functions We may occasionally
More informationDO NOT OPEN THIS QUESTION BOOKLET UNTIL YOU ARE TOLD TO DO SO
QUESTION BOOKLET EECS 227A Fall 2009 Midterm Tuesday, Ocotober 20, 11:10-12:30pm DO NOT OPEN THIS QUESTION BOOKLET UNTIL YOU ARE TOLD TO DO SO You have 80 minutes to complete the midterm. The midterm consists
More informationUnconstrained Optimization
1 / 36 Unconstrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University February 2, 2015 2 / 36 3 / 36 4 / 36 5 / 36 1. preliminaries 1.1 local approximation
More informationLecture 3: Basics of set-constrained and unconstrained optimization
Lecture 3: Basics of set-constrained and unconstrained optimization (Chap 6 from textbook) Xiaoqun Zhang Shanghai Jiao Tong University Last updated: October 9, 2018 Optimization basics Outline Optimization
More informationConvex envelopes, cardinality constrained optimization and LASSO. An application in supervised learning: support vector machines (SVMs)
ORF 523 Lecture 8 Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Any typos should be emailed to a a a@princeton.edu. 1 Outline Convexity-preserving operations Convex envelopes, cardinality
More informationModule 04 Optimization Problems KKT Conditions & Solvers
Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September
More informationSOLUTIONS TO ADDITIONAL EXERCISES FOR II.1 AND II.2
SOLUTIONS TO ADDITIONAL EXERCISES FOR II.1 AND II.2 Here are the solutions to the additional exercises in betsepexercises.pdf. B1. Let y and z be distinct points of L; we claim that x, y and z are not
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationWeek 4: Calculus and Optimization (Jehle and Reny, Chapter A2)
Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2) Tsun-Feng Chiang *School of Economics, Henan University, Kaifeng, China September 27, 2015 Microeconomic Theory Week 4: Calculus and Optimization
More information4 Newton Method. Unconstrained Convex Optimization 21. H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion:
Unconstrained Convex Optimization 21 4 Newton Method H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion: f(x + p) f(x)+p T f(x)+ 1 2 pt H(x)p ˆf(p) In general, ˆf(p) won
More informationOPER 627: Nonlinear Optimization Lecture 14: Mid-term Review
OPER 627: Nonlinear Optimization Lecture 14: Mid-term Review Department of Statistical Sciences and Operations Research Virginia Commonwealth University Oct 16, 2013 (Lecture 14) Nonlinear Optimization
More informationLecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016
Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 1 Entropy Since this course is about entropy maximization,
More informationSWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm. Convex Optimization. Computing and Software McMaster University
SWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm Convex Optimization Computing and Software McMaster University General NLO problem (NLO : Non Linear Optimization) (N LO) min
More informationBASICS OF CONVEX ANALYSIS
BASICS OF CONVEX ANALYSIS MARKUS GRASMAIR 1. Main Definitions We start with providing the central definitions of convex functions and convex sets. Definition 1. A function f : R n R + } is called convex,
More informationNumerical Optimization
Unconstrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 01, India. NPTEL Course on Unconstrained Minimization Let f : R n R. Consider the optimization problem:
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationCE 191: Civil and Environmental Engineering Systems Analysis. LEC 05 : Optimality Conditions
CE 191: Civil and Environmental Engineering Systems Analysis LEC : Optimality Conditions Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 214 Prof. Moura
More informationECE580 Exam 1 October 4, Please do not write on the back of the exam pages. Extra paper is available from the instructor.
ECE580 Exam 1 October 4, 2012 1 Name: Solution Score: /100 You must show ALL of your work for full credit. This exam is closed-book. Calculators may NOT be used. Please leave fractions as fractions, etc.
More informationPreliminary draft only: please check for final version
ARE211, Fall2012 CALCULUS4: THU, OCT 11, 2012 PRINTED: AUGUST 22, 2012 (LEC# 15) Contents 3. Univariate and Multivariate Differentiation (cont) 1 3.6. Taylor s Theorem (cont) 2 3.7. Applying Taylor theory:
More informationMarch 5, 2012 MATH 408 FINAL EXAM SAMPLE
March 5, 202 MATH 408 FINAL EXAM SAMPLE Partial Solutions to Sample Questions (in progress) See the sample questions for the midterm exam, but also consider the following questions. Obviously, a final
More informationLecture 2 - Unconstrained Optimization Definition[Global Minimum and Maximum]Let f : S R be defined on a set S R n. Then
Lecture 2 - Unconstrained Optimization Definition[Global Minimum and Maximum]Let f : S R be defined on a set S R n. Then 1. x S is a global minimum point of f over S if f (x) f (x ) for any x S. 2. x S
More informationNonlinear Programming Models
Nonlinear Programming Models Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Nonlinear Programming Models p. Introduction Nonlinear Programming Models p. NLP problems minf(x) x S R n Standard form:
More informationIntroduction to unconstrained optimization - direct search methods
Introduction to unconstrained optimization - direct search methods Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Structure of optimization methods Typically Constraint handling converts the
More informationDivision of the Humanities and Social Sciences. Supergradients. KC Border Fall 2001 v ::15.45
Division of the Humanities and Social Sciences Supergradients KC Border Fall 2001 1 The supergradient of a concave function There is a useful way to characterize the concavity of differentiable functions.
More informationIOE 511/Math 652: Continuous Optimization Methods, Section 1
IOE 511/Math 652: Continuous Optimization Methods, Section 1 Marina A. Epelman Fall 2007 These notes can be freely reproduced for any non-commercial purpose; please acknowledge the author if you do so.
More informationChapter 2 Convex Analysis
Chapter 2 Convex Analysis The theory of nonsmooth analysis is based on convex analysis. Thus, we start this chapter by giving basic concepts and results of convexity (for further readings see also [202,
More informationThe general programming problem is the nonlinear programming problem where a given function is maximized subject to a set of inequality constraints.
1 Optimization Mathematical programming refers to the basic mathematical problem of finding a maximum to a function, f, subject to some constraints. 1 In other words, the objective is to find a point,
More informationConvex Analysis and Economic Theory Winter 2018
Division of the Humanities and Social Sciences Ec 181 KC Border Convex Analysis and Economic Theory Winter 2018 Topic 7: Quasiconvex Functions I 7.1 Level sets of functions For an extended real-valued
More informationCHAPTER 2: QUADRATIC PROGRAMMING
CHAPTER 2: QUADRATIC PROGRAMMING Overview Quadratic programming (QP) problems are characterized by objective functions that are quadratic in the design variables, and linear constraints. In this sense,
More informationIowa State University. Instructor: Alex Roitershtein Summer Homework #1. Solutions
Math 501 Iowa State University Introduction to Real Analysis Department of Mathematics Instructor: Alex Roitershtein Summer 015 EXERCISES FROM CHAPTER 1 Homework #1 Solutions The following version of the
More informationMath (P)refresher Lecture 8: Unconstrained Optimization
Math (P)refresher Lecture 8: Unconstrained Optimization September 2006 Today s Topics : Quadratic Forms Definiteness of Quadratic Forms Maxima and Minima in R n First Order Conditions Second Order Conditions
More informationChapter 1 Preliminaries
Chapter 1 Preliminaries 1.1 Conventions and Notations Throughout the book we use the following notations for standard sets of numbers: N the set {1, 2,...} of natural numbers Z the set of integers Q the
More informationPart 3: Trust-region methods for unconstrained optimization. Nick Gould (RAL)
Part 3: Trust-region methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective
More informationThis property turns out to be a general property of eigenvectors of a symmetric A that correspond to distinct eigenvalues as we shall see later.
34 To obtain an eigenvector x 2 0 2 for l 2 = 0, define: B 2 A - l 2 I 2 = È 1, 1, 1 Î 1-0 È 1, 0, 0 Î 1 = È 1, 1, 1 Î 1. To transform B 2 into an upper triangular matrix, subtract the first row of B 2
More informationMathematical Economics. Lecture Notes (in extracts)
Prof. Dr. Frank Werner Faculty of Mathematics Institute of Mathematical Optimization (IMO) http://math.uni-magdeburg.de/ werner/math-ec-new.html Mathematical Economics Lecture Notes (in extracts) Winter
More informationCE 191: Civil & Environmental Engineering Systems Analysis. LEC 17 : Final Review
CE 191: Civil & Environmental Engineering Systems Analysis LEC 17 : Final Review Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 2014 Prof. Moura UC Berkeley
More informationOptimization: Problem Set Solutions
Optimization: Problem Set Solutions Annalisa Molino University of Rome Tor Vergata annalisa.molino@uniroma2.it Fall 20 Compute the maxima minima or saddle points of the following functions a. f(x y) =
More informationPerformance Surfaces and Optimum Points
CSC 302 1.5 Neural Networks Performance Surfaces and Optimum Points 1 Entrance Performance learning is another important class of learning law. Network parameters are adjusted to optimize the performance
More informationSeptember Math Course: First Order Derivative
September Math Course: First Order Derivative Arina Nikandrova Functions Function y = f (x), where x is either be a scalar or a vector of several variables (x,..., x n ), can be thought of as a rule which
More informationNumerical Optimization Techniques
Numerical Optimization Techniques Léon Bottou NEC Labs America COS 424 3/2/2010 Today s Agenda Goals Representation Capacity Control Operational Considerations Computational Considerations Classification,
More informationLecture 15 Newton Method and Self-Concordance. October 23, 2008
Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications
More informationA function(al) f is convex if dom f is a convex set, and. f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) f(x) = x 3
Convex functions The domain dom f of a functional f : R N R is the subset of R N where f is well-defined. A function(al) f is convex if dom f is a convex set, and f(θx + (1 θ)y) θf(x) + (1 θ)f(y) for all
More informationEuler s Theorem for Homogeneous Functions
Division of the Humanities and Social Sciences Euler s Theorem for Homogeneous Functions KC Border October 2000 1 Definition Let X be a subset of R n. A function f : X R is homogeneous of degree k if for
More informationInformation Theory and Communication
Information Theory and Communication Ritwik Banerjee rbanerjee@cs.stonybrook.edu c Ritwik Banerjee Information Theory and Communication 1/8 General Chain Rules Definition Conditional mutual information
More informationARE211, Fall 2005 CONTENTS. 5. Characteristics of Functions Surjective, Injective and Bijective functions. 5.2.
ARE211, Fall 2005 LECTURE #18: THU, NOV 3, 2005 PRINT DATE: NOVEMBER 22, 2005 (COMPSTAT2) CONTENTS 5. Characteristics of Functions. 1 5.1. Surjective, Injective and Bijective functions 1 5.2. Homotheticity
More informationMathematical Foundations -1- Convexity and quasi-convexity. Convex set Convex function Concave function Quasi-concave function Supporting hyperplane
Mathematical Foundations -1- Convexity and quasi-convexity Convex set Convex function Concave function Quasi-concave function Supporting hyperplane Mathematical Foundations -2- Convexity and quasi-convexity
More informationAM 205: lecture 18. Last time: optimization methods Today: conditions for optimality
AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality Existence of Global Minimum For example: f (x, y) = x 2 + y 2 is coercive on R 2 (global min. at (0, 0)) f (x) = x 3
More informationChapter 2: Preliminaries and elements of convex analysis
Chapter 2: Preliminaries and elements of convex analysis Edoardo Amaldi DEIB Politecnico di Milano edoardo.amaldi@polimi.it Website: http://home.deib.polimi.it/amaldi/opt-14-15.shtml Academic year 2014-15
More informationKey words. saddle-point dynamics, asymptotic convergence, convex-concave functions, proximal calculus, center manifold theory, nonsmooth dynamics
SADDLE-POINT DYNAMICS: CONDITIONS FOR ASYMPTOTIC STABILITY OF SADDLE POINTS ASHISH CHERUKURI, BAHMAN GHARESIFARD, AND JORGE CORTÉS Abstract. This paper considers continuously differentiable functions of
More informationCHAPTER 4: HIGHER ORDER DERIVATIVES. Likewise, we may define the higher order derivatives. f(x, y, z) = xy 2 + e zx. y = 2xy.
April 15, 2009 CHAPTER 4: HIGHER ORDER DERIVATIVES In this chapter D denotes an open subset of R n. 1. Introduction Definition 1.1. Given a function f : D R we define the second partial derivatives as
More informationNonlinear Programming Algorithms Handout
Nonlinear Programming Algorithms Handout Michael C. Ferris Computer Sciences Department University of Wisconsin Madison, Wisconsin 5376 September 9 1 Eigenvalues The eigenvalues of a matrix A C n n are
More informationConvex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Convex Functions Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Definition convex function Examples
More informationProblem Set 3
4.02 Problem Set 3 Due Thursday, October 2, 2004, in class Starred (*) problems will not count for the grade on this problem set; they are based on material from lectures on 0/2 and 0/26, and provide practice
More informationBanach-Alaoglu, boundedness, weak-to-strong principles Paul Garrett 1. Banach-Alaoglu theorem
(April 12, 2004) Banach-Alaoglu, boundedness, weak-to-strong principles Paul Garrett Banach-Alaoglu theorem: compactness of polars A variant Banach-Steinhaus theorem Bipolars Weak
More informationis a new metric on X, for reference, see [1, 3, 6]. Since x 1+x
THE TEACHING OF MATHEMATICS 016, Vol. XIX, No., pp. 68 75 STRICT MONOTONICITY OF NONNEGATIVE STRICTLY CONCAVE FUNCTION VANISHING AT THE ORIGIN Yuanhong Zhi Abstract. In this paper we prove that every nonnegative
More informationPractice Exam 1: Continuous Optimisation
Practice Exam : Continuous Optimisation. Let f : R m R be a convex function and let A R m n, b R m be given. Show that the function g(x) := f(ax + b) is a convex function of x on R n. Suppose that f is
More informationComputational Optimization. Constrained Optimization Part 2
Computational Optimization Constrained Optimization Part Optimality Conditions Unconstrained Case X* is global min Conve f X* is local min SOSC f ( *) = SONC Easiest Problem Linear equality constraints
More informationN. L. P. NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP. Optimization. Models of following form:
0.1 N. L. P. Katta G. Murty, IOE 611 Lecture slides Introductory Lecture NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP does not include everything
More informationIntroduction to Nonlinear Stochastic Programming
School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS
More informationJensen s inequality for nonconvex functions
Mathematical Communications 9(004, 119-14 119 Jensen s inequality for nonconvex functions Sanjo Zlobec Abstract. Jensen s inequality is formulated for convexifiable (generally nonconvex functions. Key
More informationReview of Optimization Methods
Review of Optimization Methods Prof. Manuela Pedio 20550 Quantitative Methods for Finance August 2018 Outline of the Course Lectures 1 and 2 (3 hours, in class): Linear and non-linear functions on Limits,
More informationTangent spaces, normals and extrema
Chapter 3 Tangent spaces, normals and extrema If S is a surface in 3-space, with a point a S where S looks smooth, i.e., without any fold or cusp or self-crossing, we can intuitively define the tangent
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationConvex Functions and Optimization
Chapter 5 Convex Functions and Optimization 5.1 Convex Functions Our next topic is that of convex functions. Again, we will concentrate on the context of a map f : R n R although the situation can be generalized
More informationStatic Problem Set 2 Solutions
Static Problem Set Solutions Jonathan Kreamer July, 0 Question (i) Let g, h be two concave functions. Is f = g + h a concave function? Prove it. Yes. Proof: Consider any two points x, x and α [0, ]. Let
More informationSeminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1
Seminars on Mathematics for Economics and Finance Topic 5: Optimization Kuhn-Tucker conditions for problems with inequality constraints 1 Session: 15 Aug 2015 (Mon), 10:00am 1:00pm I. Optimization with
More informationMathematics II, course
Mathematics II, course 2013-2014 Juan Pablo Rincón Zapatero October 24, 2013 Summary: The course has four parts that we describe below. (I) Topology in Rn is a brief review of the main concepts and properties
More informationLecture 2: Convex Sets and Functions
Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are
More information, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are
Quadratic forms We consider the quadratic function f : R 2 R defined by f(x) = 2 xt Ax b T x with x = (x, x 2 ) T, () where A R 2 2 is symmetric and b R 2. We will see that, depending on the eigenvalues
More informationFunctions of Several Variables
Functions of Several Variables The Unconstrained Minimization Problem where In n dimensions the unconstrained problem is stated as f() x variables. minimize f()x x, is a scalar objective function of vector
More informationSolutions to Homework 7
Solutions to Homework 7 Exercise #3 in section 5.2: A rectangular box is inscribed in a hemisphere of radius r. Find the dimensions of the box of maximum volume. Solution: The base of the rectangular box
More informationNonlinear Programming (NLP)
Natalia Lazzati Mathematics for Economics (Part I) Note 6: Nonlinear Programming - Unconstrained Optimization Note 6 is based on de la Fuente (2000, Ch. 7), Madden (1986, Ch. 3 and 5) and Simon and Blume
More informationGradient Descent. Dr. Xiaowei Huang
Gradient Descent Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Three machine learning algorithms: decision tree learning k-nn linear regression only optimization objectives are discussed,
More informationOptimization Methods. Lecture 19: Line Searches and Newton s Method
15.93 Optimization Methods Lecture 19: Line Searches and Newton s Method 1 Last Lecture Necessary Conditions for Optimality (identifies candidates) x local min f(x ) =, f(x ) PSD Slide 1 Sufficient Conditions
More informationThe Steepest Descent Algorithm for Unconstrained Optimization
The Steepest Descent Algorithm for Unconstrained Optimization Robert M. Freund February, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 1 Steepest Descent Algorithm The problem
More informationOptimization. Yuh-Jye Lee. March 21, Data Science and Machine Intelligence Lab National Chiao Tung University 1 / 29
Optimization Yuh-Jye Lee Data Science and Machine Intelligence Lab National Chiao Tung University March 21, 2017 1 / 29 You Have Learned (Unconstrained) Optimization in Your High School Let f (x) = ax
More informationChapter 0. Mathematical Preliminaries. 0.1 Norms
Chapter 0 Mathematical Preliminaries 0.1 Norms Throughout this course we will be working with the vector space R n. For this reason we begin with a brief review of its metric space properties Definition
More informationMidterm 1. Every element of the set of functions is continuous
Econ 200 Mathematics for Economists Midterm Question.- Consider the set of functions F C(0, ) dened by { } F = f C(0, ) f(x) = ax b, a A R and b B R That is, F is a subset of the set of continuous functions
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More information