# MATH2070 Optimisation

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 MATH2070 Optimisation Nonlinear optimisation with constraints Semester 2, 2012 Lecturer: I.W. Guo Lecture slides courtesy of J.R. Wishart

2 Review The full nonlinear optimisation problem with equality constraints Method of Lagrange multipliers Dealing with Inequality Constraints and the Kuhn-Tucker conditions Second order conditions with constraints

3 The full nonlinear optimisation problem with equality constraints Method of Lagrange multipliers Dealing with Inequality Constraints and the Kuhn-Tucker conditions Second order conditions with constraints

4 Non-linear optimisation Interested in optimising a function of several variables with constraints. Multivariate framework Variables x = (x 1, x 2,..., x n ) D R n Objective Function Z = f (x 1, x 2,..., x n ) Constraint equations, g 1 (x 1, x 2,..., x n ) = 0, h 1 (x 1, x 2,..., x n ) 0, g 2 (x 1, x 2,..., x n ) = 0, h 2 (x 1, x 2,..., x n ) 0,.. g m (x 1, x 2,..., x n ) = 0, h p (x 1, x 2,..., x n ) 0.

5 Vector notation Multivariate framework Minimise Z = f (x) such that g(x) = 0, h(x) 0, where g : R n R m and h : R n R p. Objectives? Find extrema of above problem. Consider equality constraints first. Extension: Allow inequality constraints.

6 Initial Example Minimise the heat loss through the surface area of an open lid rectangular container. Formulated as a problem Minimise, S = 2xy + 2yz + xz such that, V = xyz constant and x, y, z > 0. Parameters : (x, y, z) Objective function : f = S Constraints : g = xyz V

7 Graphical interpretation of problem level contours of f(x, y) g(x,y) = c P Feasible points occur at the tangential intersection of g(x, y) and f(x, y). Gradient of f is parallel to gradient of g. Recall from linear algebra and vectors for some constant, λ. f + λ g = 0,

8 Lagrangian Definition Define the Lagrangian for our problem, L(x, y, z, λ) = S(x, y, z) + λg(x, y, z) = 2xy + 2yz + xz + λ (xyz V ) Necessary conditions, L x = L y = L z = L λ = 0. This will be a system of four equations with four unknowns (x, y, z, λ) that needs to be solved

9 Solving the system Calculate derivatives, L x = 2y + z λyz, L y = 2x + 2z λxz, L z = 2y + x λxy, L λ = xyz V, Ensure constraint, L λ = 0 z = V xy (1) Eliminate λ, solve for (x, y, z) L x = 0 2y + z = λyz λ = 2 z + 1 y L y = 0 2x + 2z = λxz λ = 2 z + 2 x L z = 0 2y + 2x = λxy λ = 2 x + 1 y (2) (3) (4)

10 Solving system (continued) From (2) and (3) it follows x = 2y From (2) and (4) it follows x = z Use (1) with x = z = 2y then simplify, ( ) 3 3 V x = (x, y, z) = 2V, 4, 3 2V

11 The full nonlinear optimisation problem with equality constraints Method of Lagrange multipliers Dealing with Inequality Constraints and the Kuhn-Tucker conditions Second order conditions with constraints

12 Constrained optimisation Wish to generalised previous argument to multivariate framework, Minimise Z = f (x), x D R n such that g(x) = 0, h(x) 0, where g : R n R m and h : R n R p. Geometrically, the constraints define a subset Ω of R n, the feasible set, which is of dimension less than n m, in which the minimum lies.

13 Constrained Optimisation A value x D is called feasible if both g(x) = 0 and h(x) 0. An inequality constraint h i (x) 0 is Active if the feasible x D satisfies hi (x) = 0 Inactive otherwise.

14 Equality Constraints A point x which satisfies the constraint g(x ) = 0 is a regular point if { g 1 (x ), g 2 (x )...., g m (x )} is linearly independent. g 1 x 1 Rank.... g m x 1... g 1 x n. g m x n = m.

15 Tangent Subspace Definition Define the constrainted tangent subspace T = {d R n : g(x )d = 0}, where g(x ) is an m n matrix defined, g 1 x 1 g(x ) :=.... g m x 1... ( ) and gi T := gi x 1,..., g i x n g 1 x n. g m x n g 1 (x ) T =. g m (x ) T

16 Tangent Subspace (Example)

17 Generalised Lagrangian Definition Define the Lagrangian for the objective function f(x) with constraint equations g(x) = 0, L(x, λ) := f + m λ i g i = f + λ T g. i=1 Introduce a lagrange multiplier for each equality constraint.

18 First order necessary conditions Theorem Assume x is a regular point of g(x) = 0 and let x be a local extremum of f subject to g(x) = 0, with f, g C 1. Then there exists λ R m such that, L(x, λ) = f(x ) + λ T g(x ) = 0. In sigma notation: L(x ) x j = f(x ) x j + m i=1 λ i g i (x ) x j = 0, j = 1,..., n ; and L(x ) λ i = g i (x ) = 0, i = 1,..., m.

19 Another example Example Find the extremum for f(x, y) = 1 2 (x 1) (y 2)2 + 1 subject to the single constraint x + y = 1. Solution: Construct the Lagrangian: L(x, y, λ) = 1 2 (x 1) (y 2) λ(x + y 1). Then the necessary conditions are: L x = x 1 + λ = 0 L y = y 2 + λ = 0 L λ = 0 x + y = 1.

20 Solve system Can solve this by converting problem into matrix form. i.e. solve for a in Ma = b x y = λ 1 Invert matrix M and solve gives, x y = λ =

21 Comparison with unconstrained problem Example Minimize f(x, y) = 1 2 (x 1) (y 2)2 + 1 subject to the single constraint x + y = 1. In unconstrained problem, minimum attained at x = (1, 2). Minimum at f = f(1, 2) = 1 Constraint modifies solution. Constrained minimum at f = f(0, 1) = 2. Note: Constrained minimum is bounded from below by the unconstrained minumum.

22 Use of convex functions : Example Example Minimise: f(x, y) = x 2 + y 2 + xy 5y such that: x + 2y = 5. The constraint is linear (therefore convex) and Hessian matrix of f is positive definite so f is convex as well. The Lagrangian is L(x, y, λ) = x 2 + y 2 + xy 5y + λ(x + 2y 5).

23 Solve Finding first order necessary conditions, L = 2x + y + λ = 0 x (5) L = 2y + x 5 + 2λ = 0 y (6) L = x + 2y 5 = 0. λ (7) In matrix form the system is, x y = λ 5 Inverting and solving yields, x y = λ 0

24 The full nonlinear optimisation problem with equality constraints Method of Lagrange multipliers Dealing with Inequality Constraints and the Kuhn-Tucker conditions Second order conditions with constraints

25 Inequality constraints We now consider constrained optimisation problems with p inequality constraints, Minimise: f(x 1, x 2,..., x n ) subject to: h 1 (x 1, x 2,..., x n ) 0 In vector form the problem is Minimise: h 2 (x 1, x 2,..., x n ) 0 h p (x 1, x 2,..., x n ) 0. f(x) subject to: h(x) 0..

26 The Kuhn-Tucker Conditions To deal with the inequality constraints, introduce the so called Kuhn-Tucker conditions. f x j + p i=1 λ i h i x j = 0, j = 1,..., n ; λ i h i = 0, i = 1,..., m ; λ i 0, i = 1,..., m. Extra conditions for λ apply in this sense, h i (x) = 0, if λ i > 0, i = 1,..., p or h i (x) < 0, if λ i = 0, i = 1,..., p.

27 Graphical Interpretation level contours of f(x, y) h(x,y) < 0 (λ = 0) P h(x,y) > 0 h(x,y) = 0 (λ > 0) If λ > 0 then find solution on line h(x, y) = 0 If λ = 0 then find solution in region h(x, y) < 0

28 Graphical Interpretation level contours of f(x, y) (x,y ) h(x,y) = 0 (λ > 0) Consider h(x, y) = 0, then λ > 0. Optimum at x = (x, y )

29 Graphical Interpretation (x,y ) level contours of f(x, y) h(x,y) < 0 (λ = 0) h(x,y) = 0 Consider h(x, y) < 0, then λ = 0. Optimum at x = (x, y )

30 Kuhn Tucker Kuhn-Tucker conditions are always necessary. Kuhn-Tucker conditions are sufficient if the objective function and the constraint functions are convex. Need a procedure to deal with these conditions.

31 Introduce slack variables Slack Variables To deal with the inequality constraint introduce a variable, s 2 i for each constraint. New constrained optimisation problem with m equality constraints, Minimise: f(x 1, x 2,..., x n ) subject to: h 1 (x 1, x 2,..., x n ) + s 2 1 = 0 h 2 (x 1, x 2,..., x n ) + s 2 2 = 0 h p (x 1, x 2,..., x n ) + s 2 p = 0..

32 New Lagrangian The Lagrangian for constrained problem is, L(x, λ, s) = f(x) + p i=1 ( λ i hi (x) + s 2 ) i = f(x) + λ T h(x) + λ T s, where s is a vector in R p with i-th component s 2 i. L depends on the n + 2m independent variables x 1, x 2,..., x n, λ 1, λ 2,..., λ p and s 1, s 2,..., s p. To find the constrained minimum we must differentiate L with respect to each of these variables and set the result to zero in each case.

33 Solving the system Need to solve n + 2m variables given by the n + 2m equations generated by, L = L = = L = 0 x 1 x 2 x n L = L = = L = L = L = = L = 0. λ 1 λ 2 λ p s 1 s 2 s p Having λ i 0 ensures that L = f(x) + has a well defined minimum. p i=1 ( λ i hi (x) + s 2 ) i If λ i < 0, we can always make L smaller due to the λ i s 2 i term.

34 New first order equations The differentiation, L/ λ i = 0, ensures the constraints hold. L λi = 0 h i + s 2 i = 0, i = 1,..., p. The differentiation, L s i minimising, = 0, controls observed region of L si = 0 2λ i s i = 0, i = 1,..., p. If λi > 0 we are on the boundary since s i = 0 If λi = 0 we are inside the feasible region since s i 0. If λi < 0 we are on the boundary again, but the point is not a local minimum. So only focus on λ i 0.

35 Simple application Example Consider the general problem Minimise: f(x) subject to: a x b. Re-express the problem to allow the Kuhn-Tucker conditions. Minimise: subject to: f(x) x a x b. The Lagrangian L is given by L = f + λ 1 ( x + s a) + λ 2 (x + s 2 2 b).

36 Solving system Find first order equations L x = f (x) λ 1 + λ 2 = 0, L = x + a + s 2 L 1 = 0 = x b + s 2 2 = 0, λ 1 λ 2 L L = 2λ 1 s 1 = 0 = 2λ 2 s 2 = 0. s 1 s 2 Consider the scenarios for when the constraints are at the boundary or inside the feasible region.

37 Possible scenarios y y y f (x) > 0 f (x) = 0 f (x) < 0 a DOWN b x a IN b x a UP b x Down variable scenario (lower boundary). s 1 = 0 λ 1 > 0 and s 2 2 > 0 λ 2 = 0. Implies that x = a, and f (x) = f (a) = λ 1 > 0, so that the minimum lies at x = a.

38 Possible scenarios y y y f (x) > 0 f (x) = 0 f (x) < 0 a DOWN b x a IN b x a UP b x In variable scenario (Inside feasible region). s 2 1 > 0 λ 1 = 0 and s 2 2 > 0 λ 2 = 0. Implies f (x) = 0, for some x [a, b] determined by the specific f(x), and thus there is a turning point somewhere in the interval [a, b]

39 Possible scenarios y y y f (x) > 0 f (x) = 0 f (x) < 0 a DOWN b x a IN b x a UP b x Up variable scenario (upper boundary). s 2 1 > 0 λ 1 = 0 and s 2 = 0 λ 2 > 0. Implies x = b, so that f (x) = f (b) = λ 2 < 0 and we have the minimum at x = b.

40 Numerical example Example Minimise: f(x 1, x 2, x 3 ) = x 1 (x 1 10) + x 2 (x 2 50) 2x 3 subject to: x 1 + x 2 10 x Write down Lagrangian, L(x, λ, s) = f(x) + λ T (h(x) + s), where, h(x) = ( ) x1 + x 2 10 x 3 10 s = ( ) s 2 1 s 2 2

41 Solve the system of first order equations L = 2x λ 1 = 0 x 1 (8) L = 2x λ 1 = 0 x 2 (9) L = 2 + λ 2 = 0 x 3 (10) The Kuhn-Tucker conditions are λ 1, λ 2 0 and λ 1 (x 1 + x 2 10) = 0 (11) λ 2 (x 3 10) = 0 (12) Immediately from (10) and (12), λ 2 = 2 and x 3 = 10.

42 Two remaining cases to consider Inside the feasible set λ 1 = 0. From equations (8) and (9) find, x = (x 1, x 2, x 3 ) = (5, 25, 10) However, constraint x 1 + x 2 10, is violated Reject this solution! At the boundary, λ 1 > 0 Using constraint equation (11) x 1 + x 2 = 10. Find the system: 2x λ 1 = 0 2x λ 1 = 0, Solving gives, λ 1 = 20 and x 1 = 5, x 2 = 15.

43 The full nonlinear optimisation problem with equality constraints Method of Lagrange multipliers Dealing with Inequality Constraints and the Kuhn-Tucker conditions Second order conditions with constraints

44 Motivation Determine sufficient conditions for minima (maxima) given a set of constraints. Define the operator 2 to denote the Hessian matrices, 2 f := H f. What { are the necessary/sufficient conditions on 2 f and 2 } m g j that ensure a constrained minimum? j=1 Consider equality constraints first. Extend to inequality constraints.

45 Notation Hessian of the constraint equations in the upcoming results needs care. Define as follows, Constraint Hessian involving Lagrange multipliers, λ T 2 g(x) := m λ i 2 g i (x) = i=1 m λ i H gi (x) i=1

46 Necessary second order condition for a constrained minimum with equality constraints Theorem Suppose that x is a regular point of g C 2 and is a local minimum of f such that g(x) = 0 and x is a regular point of g. Then there exists a λ R m such that, f(x ) + λ T g(x ) = 0. Also, from the tangent plane T = {y : g(x )y = 0}, the Lagragian Hessian at x, 2 L(x ) := 2 f(x ) + λ T 2 g(x ), is positive semi-definite on T. i.e. y T 2 L(x )y 0 for all y T.

47 Sufficient second order condition for a constrained minimum with equality constraints Theorem Suppose that x and λ satisfy, f(x ) + λ T g(x ) = 0. Suppose further that the Lagragian Hessian at x, 2 L(x ) := 2 f(x ) + λ T 2 g(x ). If y T 2 L(x )y > 0 for all y T. Then f(x ) is a local minimum that satisfies g(x ) = 0.

48 Determine if positive/negative definite Options for determining definiteness of 2 L(x ) on the subspace T. Complete the square of y T 2 L(x )y. Compute the eigenvalues of y T 2 L(x )y. Example Max: Z = x 1 x 2 + x 1 x 3 + x 2 x 3 subject to: x 1 + x 2 + x 3 = 3.

49 Extension: Inequality constraints Problem: where f, g, h C 1. Min f(x) subject to: h(x) 0 g(x) = 0. Extend arguments to include both equality and inequality constraints. Notation: Define the index set J : Definition Let J (x ) be the index set of active constraints. h j (x ) = 0, j J ; h j (x ) < 0, j / J.

50 First order necessary conditions The point x is called a regular point of the constraints, g(x) = 0 and h(x) 0, if are linearly independent. Theorem { g i } m i=1 and { h j } j J Let x be a local minimum of f(x) subject the constraints g(x) = 0 and h(x) 0. Suppose further that x is a regular point of the constraints. Then there exists λ R m and µ R p with µ 0 such that, f(x ) + λ T g(x ) + µ T h(x ) = 0 µ T h(x ) = 0. (13)

51 Second order necessary conditions Theorem Let x be a regular point of g(x) = 0, h(x) 0 with f, g, h C 2. If x is a local minimum point of the fully constrained problem then there exists λ R m and µ R p such that (13) holds and 2 f(x ) + m λ i 2 g i (x ) + i=1 p µ j 2 h j (x ) j=1 is positive semi-definite on the tangent space of the active constraints. i.e. on the space, { } T = d R n g(x )d = 0; h j (x )d = 0, j J

52 Second order sufficient conditions Theorem Let x satisfy g(x) = 0, h(x) 0. Suppose there exists λ R m and µ R p such that µ 0 and (13) holds and 2 f(x ) + m λ i 2 g i (x ) + i=1 p µ j 2 h j (x ) is positive definite on the tangent space of the active constraints. i.e. on the space, { } T = d R n g(x )d = 0; h j (x )d = 0, j J, j=1 { } where J = j h j (x ) = 0, µ j > 0. Then x is a strict local minimum of f subject to the full constraints.

53 Ways to compute the sufficient conditions The nature of the Hessian of the Lagrangian function and with its auxiliary conditions can be determined. 1. Eigenvalues on the subspace T. 2. Find the nature of the Bordered Hessian.

54 Eigenvalues on subspaces Let {e i } m i=1 be an orthonormal basis of T. Then d T can be written, d = Ez, where z R m and, E = [e 1 e 2... e m ]. Then use, d T 2 Ld =...,

55 Bordered Hessian Alternative method to determining the behaviour of the Lagragian Hessian. Definition Given an objective function f with constraints g : R n R m, define the Bordered Hessian matrix, ( 2 L(x, λ) g(x) T ) B(x, λ) := g(x) 0 Find the nature of the Bordered Hessian.

### Lagrange Multipliers

Lagrange Multipliers (Com S 477/577 Notes) Yan-Bin Jia Nov 9, 2017 1 Introduction We turn now to the study of minimization with constraints. More specifically, we will tackle the following problem: minimize

### The Kuhn-Tucker Problem

Natalia Lazzati Mathematics for Economics (Part I) Note 8: Nonlinear Programming - The Kuhn-Tucker Problem Note 8 is based on de la Fuente (2000, Ch. 7) and Simon and Blume (1994, Ch. 18 and 19). The Kuhn-Tucker

### 3.7 Constrained Optimization and Lagrange Multipliers

3.7 Constrained Optimization and Lagrange Multipliers 71 3.7 Constrained Optimization and Lagrange Multipliers Overview: Constrained optimization problems can sometimes be solved using the methods of the

### Constrained Optimization

Constrained Optimization Joshua Wilde, revised by Isabel Tecu, Takeshi Suzuki and María José Boccardi August 13, 2013 1 General Problem Consider the following general constrained optimization problem:

### Optimality Conditions

Chapter 2 Optimality Conditions 2.1 Global and Local Minima for Unconstrained Problems When a minimization problem does not have any constraints, the problem is to find the minimum of the objective function.

### 3. Minimization with constraints Problem III. Minimize f(x) in R n given that x satisfies the equality constraints. g j (x) = c j, j = 1,...

3. Minimization with constraints Problem III. Minimize f(x) in R n given that x satisfies the equality constraints g j (x) = c j, j = 1,..., m < n, where c 1,..., c m are given numbers. Theorem 3.1. Let

### SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING

Nf SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING f(x R m g HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 5, DR RAPHAEL

### Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

### EC /11. Math for Microeconomics September Course, Part II Lecture Notes. Course Outline

LONDON SCHOOL OF ECONOMICS Professor Leonardo Felli Department of Economics S.478; x7525 EC400 20010/11 Math for Microeconomics September Course, Part II Lecture Notes Course Outline Lecture 1: Tools for

### Math 291-3: Lecture Notes Northwestern University, Spring 2016

Math 291-3: Lecture Notes Northwestern University, Spring 216 Written by Santiago Cañez These are lecture notes for Math 291-3, the third quarter of MENU: Intensive Linear Algebra and Multivariable Calculus,

### Lecture 9: Implicit function theorem, constrained extrema and Lagrange multipliers

Lecture 9: Implicit function theorem, constrained extrema and Lagrange multipliers Rafikul Alam Department of Mathematics IIT Guwahati What does the Implicit function theorem say? Let F : R 2 R be C 1.

### Lagrange Multipliers

Optimization with Constraints As long as algebra and geometry have been separated, their progress have been slow and their uses limited; but when these two sciences have been united, they have lent each

### Tangent spaces, normals and extrema

Chapter 3 Tangent spaces, normals and extrema If S is a surface in 3-space, with a point a S where S looks smooth, i.e., without any fold or cusp or self-crossing, we can intuitively define the tangent

### Computation. For QDA we need to calculate: Lets first consider the case that

Computation For QDA we need to calculate: δ (x) = 1 2 log( Σ ) 1 2 (x µ ) Σ 1 (x µ ) + log(π ) Lets first consider the case that Σ = I,. This is the case where each distribution is spherical, around the

### Lagrange multipliers. Portfolio optimization. The Lagrange multipliers method for finding constrained extrema of multivariable functions.

Chapter 9 Lagrange multipliers Portfolio optimization The Lagrange multipliers method for finding constrained extrema of multivariable functions 91 Lagrange multipliers Optimization problems often require

### Optimization Theory. Lectures 4-6

Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x\$0} Existence of a maximum:

### Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization. Nick Gould (RAL)

Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization Nick Gould (RAL) x IR n f(x) subject to c(x) = Part C course on continuoue optimization CONSTRAINED MINIMIZATION x

### MATH20132 Calculus of Several Variables. 2018

MATH20132 Calculus of Several Variables 2018 Solutions to Problems 8 Lagrange s Method 1 For x R 2 let fx = x 2 3xy +y 2 5x+5y i Find the critical values of fx in R 2, ii Findthecriticalvaluesoffxrestrictedtotheparametriccurvet

### Ph.D. Katarína Bellová Page 1 Mathematics 2 (10-PHY-BIPMA2) EXAM - Solutions, 20 July 2017, 10:00 12:00 All answers to be justified.

PhD Katarína Bellová Page 1 Mathematics 2 (10-PHY-BIPMA2 EXAM - Solutions, 20 July 2017, 10:00 12:00 All answers to be justified Problem 1 [ points]: For which parameters λ R does the following system

### Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

Quadratic Programming Outline Linearly constrained minimization Linear equality constraints Linear inequality constraints Quadratic objective function 2 SideBar: Matrix Spaces Four fundamental subspaces

### Constrained optimization: direct methods (cont.)

Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a

### Numerical optimization

Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal

### Preliminary draft only: please check for final version

ARE211, Fall2012 CALCULUS4: THU, OCT 11, 2012 PRINTED: AUGUST 22, 2012 (LEC# 15) Contents 3. Univariate and Multivariate Differentiation (cont) 1 3.6. Taylor s Theorem (cont) 2 3.7. Applying Taylor theory:

### Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

### Convex Optimization and Modeling

Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

### Lecture 2: Convex Sets and Functions

Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are

### Mathematical Foundations -1- Constrained Optimization. Constrained Optimization. An intuitive approach 2. First Order Conditions (FOC) 7

Mathematical Foundations -- Constrained Optimization Constrained Optimization An intuitive approach First Order Conditions (FOC) 7 Constraint qualifications 9 Formal statement of the FOC for a maximum

### ECE580 Solution to Problem Set 6

ECE580 Fall 2015 Solution to Problem Set 6 December 23 2015 1 ECE580 Solution to Problem Set 6 These problems are from the textbook by Chong and Zak 4th edition which is the textbook for the ECE580 Fall

### Here each term has degree 2 (the sum of exponents is 2 for all summands). A quadratic form of three variables looks as

Reading [SB], Ch. 16.1-16.3, p. 375-393 1 Quadratic Forms A quadratic function f : R R has the form f(x) = a x. Generalization of this notion to two variables is the quadratic form Q(x 1, x ) = a 11 x

### Computational Optimization. Constrained Optimization Part 2

Computational Optimization Constrained Optimization Part Optimality Conditions Unconstrained Case X* is global min Conve f X* is local min SOSC f ( *) = SONC Easiest Problem Linear equality constraints

### Date: July 5, Contents

2 Lagrange Multipliers Date: July 5, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 14 2.3. Informative Lagrange Multipliers...........

### Chapter 3 Numerical Methods

Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2

### 1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016

AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 8 February 22nd 1 Overview In the previous lecture we saw characterizations of optimality in linear optimization, and we reviewed the

### Lagrange Multipliers

Calculus 3 Lia Vas Lagrange Multipliers Constrained Optimization for functions of two variables. To find the maximum and minimum values of z = f(x, y), objective function, subject to a constraint g(x,

### Chap 2. Optimality conditions

Chap 2. Optimality conditions Version: 29-09-2012 2.1 Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1

### Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725

Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =

### ARE211, Fall2015. Contents. 2. Univariate and Multivariate Differentiation (cont) Taylor s Theorem (cont) 2

ARE211, Fall2015 CALCULUS4: THU, SEP 17, 2015 PRINTED: SEPTEMBER 22, 2015 (LEC# 7) Contents 2. Univariate and Multivariate Differentiation (cont) 1 2.4. Taylor s Theorem (cont) 2 2.5. Applying Taylor theory:

### 1 Convexity, concavity and quasi-concavity. (SB )

UNIVERSITY OF MARYLAND ECON 600 Summer 2010 Lecture Two: Unconstrained Optimization 1 Convexity, concavity and quasi-concavity. (SB 21.1-21.3.) For any two points, x, y R n, we can trace out the line of

### CE 191: Civil and Environmental Engineering Systems Analysis. LEC 05 : Optimality Conditions

CE 191: Civil and Environmental Engineering Systems Analysis LEC : Optimality Conditions Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 214 Prof. Moura

### Optimization Tutorial 1. Basic Gradient Descent

E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.

### Math 273a: Optimization Basic concepts

Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize

CHAPTER 2: QUADRATIC PROGRAMMING Overview Quadratic programming (QP) problems are characterized by objective functions that are quadratic in the design variables, and linear constraints. In this sense,

### Newton-Type Algorithms for Nonlinear Constrained Optimization

Newton-Type Algorithms for Nonlinear Constrained Optimization Angelika Altmann-Dieses Faculty of Management Science and Engineering Karlsruhe University of Applied Sciences Moritz Diehl Department of Microsystems

### MATHEMATICS FOR ECONOMISTS. An Introductory Textbook. Third Edition. Malcolm Pemberton and Nicholas Rau. UNIVERSITY OF TORONTO PRESS Toronto Buffalo

MATHEMATICS FOR ECONOMISTS An Introductory Textbook Third Edition Malcolm Pemberton and Nicholas Rau UNIVERSITY OF TORONTO PRESS Toronto Buffalo Contents Preface Dependence of Chapters Answers and Solutions

### Advanced Calculus Notes. Michael P. Windham

Advanced Calculus Notes Michael P. Windham February 25, 2008 Contents 1 Basics 5 1.1 Euclidean Space R n............................ 5 1.2 Functions................................. 7 1.2.1 Sets and functions........................

### Let f(x) = x, but the domain of f is the interval 0 x 1. Note

I.g Maximum and Minimum. Lagrange Multipliers Recall: Suppose we are given y = f(x). We recall that the maximum/minimum points occur at the following points: (1) where f = 0; (2) where f does not exist;

### STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II - KUHN TUCKER THEORY

STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II - KUHN TUCKER THEORY UNIVERSITY OF MARYLAND: ECON 600 1. Some Eamples 1 A general problem that arises countless times in economics takes the form: (Verbally):

### Applied Lagrange Duality for Constrained Optimization

Applied Lagrange Duality for Constrained Optimization February 12, 2002 Overview The Practical Importance of Duality ffl Review of Convexity ffl A Separating Hyperplane Theorem ffl Definition of the Dual

### Review of Optimization Basics

Review of Optimization Basics. Introduction Electricity markets throughout the US are said to have a two-settlement structure. The reason for this is that the structure includes two different markets:

### CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008.

1 ECONOMICS 594: LECTURE NOTES CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS W. Erwin Diewert January 31, 2008. 1. Introduction Many economic problems have the following structure: (i) a linear function

### CONVEX FUNCTIONS AND OPTIMIZATION TECHINIQUES A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF

CONVEX FUNCTIONS AND OPTIMIZATION TECHINIQUES A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN MATHEMATICS SUBMITTED TO NATIONAL INSTITUTE OF TECHNOLOGY,

### 10 Numerical methods for constrained problems

10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside

### Nonlinear Programming (NLP)

Natalia Lazzati Mathematics for Economics (Part I) Note 6: Nonlinear Programming - Unconstrained Optimization Note 6 is based on de la Fuente (2000, Ch. 7), Madden (1986, Ch. 3 and 5) and Simon and Blume

### Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

### Lecture Support Vector Machine (SVM) Classifiers

Introduction to Machine Learning Lecturer: Amir Globerson Lecture 6 Fall Semester Scribe: Yishay Mansour 6.1 Support Vector Machine (SVM) Classifiers Classification is one of the most important tasks in

### In English, this means that if we travel on a straight line between any two points in C, then we never leave C.

Convex sets In this section, we will be introduced to some of the mathematical fundamentals of convex sets. In order to motivate some of the definitions, we will look at the closest point problem from

### You should be able to...

Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set

### Implicit Functions, Curves and Surfaces

Chapter 11 Implicit Functions, Curves and Surfaces 11.1 Implicit Function Theorem Motivation. In many problems, objects or quantities of interest can only be described indirectly or implicitly. It is then

### 0, otherwise. Find each of the following limits, or explain that the limit does not exist.

Midterm Solutions 1, y x 4 1. Let f(x, y) = 1, y 0 0, otherwise. Find each of the following limits, or explain that the limit does not exist. (a) (b) (c) lim f(x, y) (x,y) (0,1) lim f(x, y) (x,y) (2,3)

Lecture: Quadratic optimization 1. Positive definite och semidefinite matrices 2. LDL T factorization 3. Quadratic optimization without constraints 4. Quadratic optimization with constraints 5. Least-squares

### 39.1 Absolute maxima/minima

Module 13 : Maxima, Minima Saddle Points, Constrained maxima minima Lecture 39 : Absolute maxima / minima [Section 39.1] Objectives In this section you will learn the following : The notion of absolute

### Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:

CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

### Topic III.2: Maximum Entropy Models

Topic III.2: Maximum Entropy Models Discrete Topics in Data Mining Universität des Saarlandes, Saarbrücken Winter Semester 2012/13 T III.2-1 Topic III.2: Maximum Entropy Models 1. The Maximum Entropy Principle

### Assignment 1: From the Definition of Convexity to Helley Theorem

Assignment 1: From the Definition of Convexity to Helley Theorem Exercise 1 Mark in the following list the sets which are convex: 1. {x R 2 : x 1 + i 2 x 2 1, i = 1,..., 10} 2. {x R 2 : x 2 1 + 2ix 1x

### 1 Invariant subspaces

MATH 2040 Linear Algebra II Lecture Notes by Martin Li Lecture 8 Eigenvalues, eigenvectors and invariant subspaces 1 In previous lectures we have studied linear maps T : V W from a vector space V to another

### MAT 419 Lecture Notes Transcribed by Eowyn Cenek 6/1/2012

(Homework 1: Chapter 1: Exercises 1-7, 9, 11, 19, due Monday June 11th See also the course website for lectures, assignments, etc) Note: today s lecture is primarily about definitions Lots of definitions

### Linear vector spaces and subspaces.

Math 2051 W2008 Margo Kondratieva Week 1 Linear vector spaces and subspaces. Section 1.1 The notion of a linear vector space. For the purpose of these notes we regard (m 1)-matrices as m-dimensional vectors,

### Dimensions = xyz dv. xyz dv as an iterated integral in rectangular coordinates.

Math Show Your Work! Page of 8. () A rectangular box is to hold 6 cubic meters. The material used for the top and bottom of the box is twice as expensive per square meter than the material used for the

### NONLINEAR PROGRAMMING WITH CONSTRAINTS

NONLINEAR PROGRAMMING WITH CONSTRAINTS 8.1 Directsubstitution... 265 8.2 First-Order ~ecessary Conditions for a Local Extremum... 267 8.3 Quadratic Programming... 284 8.4 Penalty. Barrier. and Augmented

### 3E4: Modelling Choice. Introduction to nonlinear programming. Announcements

3E4: Modelling Choice Lecture 7 Introduction to nonlinear programming 1 Announcements Solutions to Lecture 4-6 Homework will be available from http://www.eng.cam.ac.uk/~dr241/3e4 Looking ahead to Lecture

### EC400 Math for Microeconomics Syllabus The course is based on 6 sixty minutes lectures and on 6 ninety minutes classes.

London School of Economics Department of Economics Dr Francesco Nava Offi ce: 32L.3.20 EC400 Math for Microeconomics Syllabus 2016 The course is based on 6 sixty minutes lectures and on 6 ninety minutes

### Optimization using Calculus. Optimization of Functions of Multiple Variables subject to Equality Constraints

Optimization using Calculus Optimization of Functions of Multiple Variables subject to Equality Constraints 1 Objectives Optimization of functions of multiple variables subjected to equality constraints

### EE 227A: Convex Optimization and Applications October 14, 2008

EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider

Essential Microeconomics -- CHAPTER -: SHADOW PRICES An intuitive approach: profit maimizing firm with a fied supply of an input Shadow prices 5 Concave maimization problem 7 Constraint qualifications

### MULTIVARIABLE CALCULUS, LINEAR ALGEBRA, AND DIFFERENTIAL EQUATIONS

T H I R D E D I T I O N MULTIVARIABLE CALCULUS, LINEAR ALGEBRA, AND DIFFERENTIAL EQUATIONS STANLEY I. GROSSMAN University of Montana and University College London SAUNDERS COLLEGE PUBLISHING HARCOURT BRACE

### Differentiation. f(x + h) f(x) Lh = L.

Analysis in R n Math 204, Section 30 Winter Quarter 2008 Paul Sally, e-mail: sally@math.uchicago.edu John Boller, e-mail: boller@math.uchicago.edu website: http://www.math.uchicago.edu/ boller/m203 Differentiation

### Convex Optimization Lecture 6: KKT Conditions, and applications

Convex Optimization Lecture 6: KKT Conditions, and applications Dr. Michel Baes, IFOR / ETH Zürich Quick recall of last week s lecture Various aspects of convexity: The set of minimizers is convex. Convex

### Lecture 6 Positive Definite Matrices

Linear Algebra Lecture 6 Positive Definite Matrices Prof. Chun-Hung Liu Dept. of Electrical and Computer Engineering National Chiao Tung University Spring 2017 2017/6/8 Lecture 6: Positive Definite Matrices

### Mathematical Economics: Lecture 16

Mathematical Economics: Lecture 16 Yu Ren WISE, Xiamen University November 26, 2012 Outline 1 Chapter 21: Concave and Quasiconcave Functions New Section Chapter 21: Concave and Quasiconcave Functions Concave

### z = f (x; y) f (x ; y ) f (x; y) f (x; y )

BEEM0 Optimization Techiniques for Economists Lecture Week 4 Dieter Balkenborg Departments of Economics University of Exeter Since the fabric of the universe is most perfect, and is the work of a most

### (Here > 0 is given.) In cases where M is convex, there is a nice theory for this problem; the theory has much more general applicability too.

Convex Analysis with Applications UBC Math 604 Lecture Notes by Philip D. Loewen In trust region methods, we minimize a quadratic model function M = M(p) over the set of all p R n satisfying a constraint

### Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88

Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant

### Optimisation theory. The International College of Economics and Finance. I. An explanatory note. The Lecturer and Class teacher: Kantorovich G.G.

The International College of Economics and Finance The course syllabus Optimisation theory The eighth semester I. An explanatory note The Lecturer and Class teacher: Kantorovich G.G. Requirements to students:

### HOMEWORK 7 SOLUTIONS

HOMEWORK 7 SOLUTIONS MA11: ADVANCED CALCULUS, HILARY 17 (1) Using the method of Lagrange multipliers, find the largest and smallest values of the function f(x, y) xy on the ellipse x + y 1. Solution: The

### 1 The linear algebra of linear programs (March 15 and 22, 2015)

1 The linear algebra of linear programs (March 15 and 22, 2015) Many optimization problems can be formulated as linear programs. The main features of a linear program are the following: Variables are real

### Support Vector Machines and Kernel Methods

2018 CS420 Machine Learning, Lecture 3 Hangout from Prof. Andrew Ng. http://cs229.stanford.edu/notes/cs229-notes3.pdf Support Vector Machines and Kernel Methods Weinan Zhang Shanghai Jiao Tong University

### 8 Barrier Methods for Constrained Optimization

IOE 519: NL, Winter 2012 c Marina A. Epelman 55 8 Barrier Methods for Constrained Optimization In this subsection, we will restrict our attention to instances of constrained problem () that have inequality

### Lecture 6: Conic Optimization September 8

IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

### 4 Partial Differentiation

4 Partial Differentiation Many equations in engineering, physics and mathematics tie together more than two variables. For example Ohm s Law (V = IR) and the equation for an ideal gas, PV = nrt, which

### Symmetric and Asymmetric Duality

journal of mathematical analysis and applications 220, 125 131 (1998) article no. AY975824 Symmetric and Asymmetric Duality Massimo Pappalardo Department of Mathematics, Via Buonarroti 2, 56127, Pisa,

### Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2)

Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2) Tsun-Feng Chiang *School of Economics, Henan University, Kaifeng, China September 27, 2015 Microeconomic Theory Week 4: Calculus and Optimization

### Module 04 Optimization Problems KKT Conditions & Solvers

Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September

### A geometric proof of the spectral theorem for real symmetric matrices

0 0 0 A geometric proof of the spectral theorem for real symmetric matrices Robert Sachs Department of Mathematical Sciences George Mason University Fairfax, Virginia 22030 rsachs@gmu.edu January 6, 2011

### Math 263 Assignment #4 Solutions. 0 = f 1 (x,y,z) = 2x 1 0 = f 2 (x,y,z) = z 2 0 = f 3 (x,y,z) = y 1

Math 263 Assignment #4 Solutions 1. Find and classify the critical points of each of the following functions: (a) f(x,y,z) = x 2 + yz x 2y z + 7 (c) f(x,y) = e x2 y 2 (1 e x2 ) (b) f(x,y) = (x + y) 3 (x

### Maths for Signals and Systems Linear Algebra in Engineering

Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE

### Finite Dimensional Optimization Part I: The KKT Theorem 1

John Nachbar Washington University March 26, 2018 1 Introduction Finite Dimensional Optimization Part I: The KKT Theorem 1 These notes characterize maxima and minima in terms of first derivatives. I focus

### Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R