MATH2070 Optimisation

Save this PDF as:
 WORD  PNG  TXT  JPG

Size: px
Start display at page:

Download "MATH2070 Optimisation"

Transcription

1 MATH2070 Optimisation Nonlinear optimisation with constraints Semester 2, 2012 Lecturer: I.W. Guo Lecture slides courtesy of J.R. Wishart

2 Review The full nonlinear optimisation problem with equality constraints Method of Lagrange multipliers Dealing with Inequality Constraints and the Kuhn-Tucker conditions Second order conditions with constraints

3 The full nonlinear optimisation problem with equality constraints Method of Lagrange multipliers Dealing with Inequality Constraints and the Kuhn-Tucker conditions Second order conditions with constraints

4 Non-linear optimisation Interested in optimising a function of several variables with constraints. Multivariate framework Variables x = (x 1, x 2,..., x n ) D R n Objective Function Z = f (x 1, x 2,..., x n ) Constraint equations, g 1 (x 1, x 2,..., x n ) = 0, h 1 (x 1, x 2,..., x n ) 0, g 2 (x 1, x 2,..., x n ) = 0, h 2 (x 1, x 2,..., x n ) 0,.. g m (x 1, x 2,..., x n ) = 0, h p (x 1, x 2,..., x n ) 0.

5 Vector notation Multivariate framework Minimise Z = f (x) such that g(x) = 0, h(x) 0, where g : R n R m and h : R n R p. Objectives? Find extrema of above problem. Consider equality constraints first. Extension: Allow inequality constraints.

6 Initial Example Minimise the heat loss through the surface area of an open lid rectangular container. Formulated as a problem Minimise, S = 2xy + 2yz + xz such that, V = xyz constant and x, y, z > 0. Parameters : (x, y, z) Objective function : f = S Constraints : g = xyz V

7 Graphical interpretation of problem level contours of f(x, y) g(x,y) = c P Feasible points occur at the tangential intersection of g(x, y) and f(x, y). Gradient of f is parallel to gradient of g. Recall from linear algebra and vectors for some constant, λ. f + λ g = 0,

8 Lagrangian Definition Define the Lagrangian for our problem, L(x, y, z, λ) = S(x, y, z) + λg(x, y, z) = 2xy + 2yz + xz + λ (xyz V ) Necessary conditions, L x = L y = L z = L λ = 0. This will be a system of four equations with four unknowns (x, y, z, λ) that needs to be solved

9 Solving the system Calculate derivatives, L x = 2y + z λyz, L y = 2x + 2z λxz, L z = 2y + x λxy, L λ = xyz V, Ensure constraint, L λ = 0 z = V xy (1) Eliminate λ, solve for (x, y, z) L x = 0 2y + z = λyz λ = 2 z + 1 y L y = 0 2x + 2z = λxz λ = 2 z + 2 x L z = 0 2y + 2x = λxy λ = 2 x + 1 y (2) (3) (4)

10 Solving system (continued) From (2) and (3) it follows x = 2y From (2) and (4) it follows x = z Use (1) with x = z = 2y then simplify, ( ) 3 3 V x = (x, y, z) = 2V, 4, 3 2V

11 The full nonlinear optimisation problem with equality constraints Method of Lagrange multipliers Dealing with Inequality Constraints and the Kuhn-Tucker conditions Second order conditions with constraints

12 Constrained optimisation Wish to generalised previous argument to multivariate framework, Minimise Z = f (x), x D R n such that g(x) = 0, h(x) 0, where g : R n R m and h : R n R p. Geometrically, the constraints define a subset Ω of R n, the feasible set, which is of dimension less than n m, in which the minimum lies.

13 Constrained Optimisation A value x D is called feasible if both g(x) = 0 and h(x) 0. An inequality constraint h i (x) 0 is Active if the feasible x D satisfies hi (x) = 0 Inactive otherwise.

14 Equality Constraints A point x which satisfies the constraint g(x ) = 0 is a regular point if { g 1 (x ), g 2 (x )...., g m (x )} is linearly independent. g 1 x 1 Rank.... g m x 1... g 1 x n. g m x n = m.

15 Tangent Subspace Definition Define the constrainted tangent subspace T = {d R n : g(x )d = 0}, where g(x ) is an m n matrix defined, g 1 x 1 g(x ) :=.... g m x 1... ( ) and gi T := gi x 1,..., g i x n g 1 x n. g m x n g 1 (x ) T =. g m (x ) T

16 Tangent Subspace (Example)

17 Generalised Lagrangian Definition Define the Lagrangian for the objective function f(x) with constraint equations g(x) = 0, L(x, λ) := f + m λ i g i = f + λ T g. i=1 Introduce a lagrange multiplier for each equality constraint.

18 First order necessary conditions Theorem Assume x is a regular point of g(x) = 0 and let x be a local extremum of f subject to g(x) = 0, with f, g C 1. Then there exists λ R m such that, L(x, λ) = f(x ) + λ T g(x ) = 0. In sigma notation: L(x ) x j = f(x ) x j + m i=1 λ i g i (x ) x j = 0, j = 1,..., n ; and L(x ) λ i = g i (x ) = 0, i = 1,..., m.

19 Another example Example Find the extremum for f(x, y) = 1 2 (x 1) (y 2)2 + 1 subject to the single constraint x + y = 1. Solution: Construct the Lagrangian: L(x, y, λ) = 1 2 (x 1) (y 2) λ(x + y 1). Then the necessary conditions are: L x = x 1 + λ = 0 L y = y 2 + λ = 0 L λ = 0 x + y = 1.

20 Solve system Can solve this by converting problem into matrix form. i.e. solve for a in Ma = b x y = λ 1 Invert matrix M and solve gives, x y = λ =

21 Comparison with unconstrained problem Example Minimize f(x, y) = 1 2 (x 1) (y 2)2 + 1 subject to the single constraint x + y = 1. In unconstrained problem, minimum attained at x = (1, 2). Minimum at f = f(1, 2) = 1 Constraint modifies solution. Constrained minimum at f = f(0, 1) = 2. Note: Constrained minimum is bounded from below by the unconstrained minumum.

22 Use of convex functions : Example Example Minimise: f(x, y) = x 2 + y 2 + xy 5y such that: x + 2y = 5. The constraint is linear (therefore convex) and Hessian matrix of f is positive definite so f is convex as well. The Lagrangian is L(x, y, λ) = x 2 + y 2 + xy 5y + λ(x + 2y 5).

23 Solve Finding first order necessary conditions, L = 2x + y + λ = 0 x (5) L = 2y + x 5 + 2λ = 0 y (6) L = x + 2y 5 = 0. λ (7) In matrix form the system is, x y = λ 5 Inverting and solving yields, x y = λ 0

24 The full nonlinear optimisation problem with equality constraints Method of Lagrange multipliers Dealing with Inequality Constraints and the Kuhn-Tucker conditions Second order conditions with constraints

25 Inequality constraints We now consider constrained optimisation problems with p inequality constraints, Minimise: f(x 1, x 2,..., x n ) subject to: h 1 (x 1, x 2,..., x n ) 0 In vector form the problem is Minimise: h 2 (x 1, x 2,..., x n ) 0 h p (x 1, x 2,..., x n ) 0. f(x) subject to: h(x) 0..

26 The Kuhn-Tucker Conditions To deal with the inequality constraints, introduce the so called Kuhn-Tucker conditions. f x j + p i=1 λ i h i x j = 0, j = 1,..., n ; λ i h i = 0, i = 1,..., m ; λ i 0, i = 1,..., m. Extra conditions for λ apply in this sense, h i (x) = 0, if λ i > 0, i = 1,..., p or h i (x) < 0, if λ i = 0, i = 1,..., p.

27 Graphical Interpretation level contours of f(x, y) h(x,y) < 0 (λ = 0) P h(x,y) > 0 h(x,y) = 0 (λ > 0) If λ > 0 then find solution on line h(x, y) = 0 If λ = 0 then find solution in region h(x, y) < 0

28 Graphical Interpretation level contours of f(x, y) (x,y ) h(x,y) = 0 (λ > 0) Consider h(x, y) = 0, then λ > 0. Optimum at x = (x, y )

29 Graphical Interpretation (x,y ) level contours of f(x, y) h(x,y) < 0 (λ = 0) h(x,y) = 0 Consider h(x, y) < 0, then λ = 0. Optimum at x = (x, y )

30 Kuhn Tucker Kuhn-Tucker conditions are always necessary. Kuhn-Tucker conditions are sufficient if the objective function and the constraint functions are convex. Need a procedure to deal with these conditions.

31 Introduce slack variables Slack Variables To deal with the inequality constraint introduce a variable, s 2 i for each constraint. New constrained optimisation problem with m equality constraints, Minimise: f(x 1, x 2,..., x n ) subject to: h 1 (x 1, x 2,..., x n ) + s 2 1 = 0 h 2 (x 1, x 2,..., x n ) + s 2 2 = 0 h p (x 1, x 2,..., x n ) + s 2 p = 0..

32 New Lagrangian The Lagrangian for constrained problem is, L(x, λ, s) = f(x) + p i=1 ( λ i hi (x) + s 2 ) i = f(x) + λ T h(x) + λ T s, where s is a vector in R p with i-th component s 2 i. L depends on the n + 2m independent variables x 1, x 2,..., x n, λ 1, λ 2,..., λ p and s 1, s 2,..., s p. To find the constrained minimum we must differentiate L with respect to each of these variables and set the result to zero in each case.

33 Solving the system Need to solve n + 2m variables given by the n + 2m equations generated by, L = L = = L = 0 x 1 x 2 x n L = L = = L = L = L = = L = 0. λ 1 λ 2 λ p s 1 s 2 s p Having λ i 0 ensures that L = f(x) + has a well defined minimum. p i=1 ( λ i hi (x) + s 2 ) i If λ i < 0, we can always make L smaller due to the λ i s 2 i term.

34 New first order equations The differentiation, L/ λ i = 0, ensures the constraints hold. L λi = 0 h i + s 2 i = 0, i = 1,..., p. The differentiation, L s i minimising, = 0, controls observed region of L si = 0 2λ i s i = 0, i = 1,..., p. If λi > 0 we are on the boundary since s i = 0 If λi = 0 we are inside the feasible region since s i 0. If λi < 0 we are on the boundary again, but the point is not a local minimum. So only focus on λ i 0.

35 Simple application Example Consider the general problem Minimise: f(x) subject to: a x b. Re-express the problem to allow the Kuhn-Tucker conditions. Minimise: subject to: f(x) x a x b. The Lagrangian L is given by L = f + λ 1 ( x + s a) + λ 2 (x + s 2 2 b).

36 Solving system Find first order equations L x = f (x) λ 1 + λ 2 = 0, L = x + a + s 2 L 1 = 0 = x b + s 2 2 = 0, λ 1 λ 2 L L = 2λ 1 s 1 = 0 = 2λ 2 s 2 = 0. s 1 s 2 Consider the scenarios for when the constraints are at the boundary or inside the feasible region.

37 Possible scenarios y y y f (x) > 0 f (x) = 0 f (x) < 0 a DOWN b x a IN b x a UP b x Down variable scenario (lower boundary). s 1 = 0 λ 1 > 0 and s 2 2 > 0 λ 2 = 0. Implies that x = a, and f (x) = f (a) = λ 1 > 0, so that the minimum lies at x = a.

38 Possible scenarios y y y f (x) > 0 f (x) = 0 f (x) < 0 a DOWN b x a IN b x a UP b x In variable scenario (Inside feasible region). s 2 1 > 0 λ 1 = 0 and s 2 2 > 0 λ 2 = 0. Implies f (x) = 0, for some x [a, b] determined by the specific f(x), and thus there is a turning point somewhere in the interval [a, b]

39 Possible scenarios y y y f (x) > 0 f (x) = 0 f (x) < 0 a DOWN b x a IN b x a UP b x Up variable scenario (upper boundary). s 2 1 > 0 λ 1 = 0 and s 2 = 0 λ 2 > 0. Implies x = b, so that f (x) = f (b) = λ 2 < 0 and we have the minimum at x = b.

40 Numerical example Example Minimise: f(x 1, x 2, x 3 ) = x 1 (x 1 10) + x 2 (x 2 50) 2x 3 subject to: x 1 + x 2 10 x Write down Lagrangian, L(x, λ, s) = f(x) + λ T (h(x) + s), where, h(x) = ( ) x1 + x 2 10 x 3 10 s = ( ) s 2 1 s 2 2

41 Solve the system of first order equations L = 2x λ 1 = 0 x 1 (8) L = 2x λ 1 = 0 x 2 (9) L = 2 + λ 2 = 0 x 3 (10) The Kuhn-Tucker conditions are λ 1, λ 2 0 and λ 1 (x 1 + x 2 10) = 0 (11) λ 2 (x 3 10) = 0 (12) Immediately from (10) and (12), λ 2 = 2 and x 3 = 10.

42 Two remaining cases to consider Inside the feasible set λ 1 = 0. From equations (8) and (9) find, x = (x 1, x 2, x 3 ) = (5, 25, 10) However, constraint x 1 + x 2 10, is violated Reject this solution! At the boundary, λ 1 > 0 Using constraint equation (11) x 1 + x 2 = 10. Find the system: 2x λ 1 = 0 2x λ 1 = 0, Solving gives, λ 1 = 20 and x 1 = 5, x 2 = 15.

43 The full nonlinear optimisation problem with equality constraints Method of Lagrange multipliers Dealing with Inequality Constraints and the Kuhn-Tucker conditions Second order conditions with constraints

44 Motivation Determine sufficient conditions for minima (maxima) given a set of constraints. Define the operator 2 to denote the Hessian matrices, 2 f := H f. What { are the necessary/sufficient conditions on 2 f and 2 } m g j that ensure a constrained minimum? j=1 Consider equality constraints first. Extend to inequality constraints.

45 Notation Hessian of the constraint equations in the upcoming results needs care. Define as follows, Constraint Hessian involving Lagrange multipliers, λ T 2 g(x) := m λ i 2 g i (x) = i=1 m λ i H gi (x) i=1

46 Necessary second order condition for a constrained minimum with equality constraints Theorem Suppose that x is a regular point of g C 2 and is a local minimum of f such that g(x) = 0 and x is a regular point of g. Then there exists a λ R m such that, f(x ) + λ T g(x ) = 0. Also, from the tangent plane T = {y : g(x )y = 0}, the Lagragian Hessian at x, 2 L(x ) := 2 f(x ) + λ T 2 g(x ), is positive semi-definite on T. i.e. y T 2 L(x )y 0 for all y T.

47 Sufficient second order condition for a constrained minimum with equality constraints Theorem Suppose that x and λ satisfy, f(x ) + λ T g(x ) = 0. Suppose further that the Lagragian Hessian at x, 2 L(x ) := 2 f(x ) + λ T 2 g(x ). If y T 2 L(x )y > 0 for all y T. Then f(x ) is a local minimum that satisfies g(x ) = 0.

48 Determine if positive/negative definite Options for determining definiteness of 2 L(x ) on the subspace T. Complete the square of y T 2 L(x )y. Compute the eigenvalues of y T 2 L(x )y. Example Max: Z = x 1 x 2 + x 1 x 3 + x 2 x 3 subject to: x 1 + x 2 + x 3 = 3.

49 Extension: Inequality constraints Problem: where f, g, h C 1. Min f(x) subject to: h(x) 0 g(x) = 0. Extend arguments to include both equality and inequality constraints. Notation: Define the index set J : Definition Let J (x ) be the index set of active constraints. h j (x ) = 0, j J ; h j (x ) < 0, j / J.

50 First order necessary conditions The point x is called a regular point of the constraints, g(x) = 0 and h(x) 0, if are linearly independent. Theorem { g i } m i=1 and { h j } j J Let x be a local minimum of f(x) subject the constraints g(x) = 0 and h(x) 0. Suppose further that x is a regular point of the constraints. Then there exists λ R m and µ R p with µ 0 such that, f(x ) + λ T g(x ) + µ T h(x ) = 0 µ T h(x ) = 0. (13)

51 Second order necessary conditions Theorem Let x be a regular point of g(x) = 0, h(x) 0 with f, g, h C 2. If x is a local minimum point of the fully constrained problem then there exists λ R m and µ R p such that (13) holds and 2 f(x ) + m λ i 2 g i (x ) + i=1 p µ j 2 h j (x ) j=1 is positive semi-definite on the tangent space of the active constraints. i.e. on the space, { } T = d R n g(x )d = 0; h j (x )d = 0, j J

52 Second order sufficient conditions Theorem Let x satisfy g(x) = 0, h(x) 0. Suppose there exists λ R m and µ R p such that µ 0 and (13) holds and 2 f(x ) + m λ i 2 g i (x ) + i=1 p µ j 2 h j (x ) is positive definite on the tangent space of the active constraints. i.e. on the space, { } T = d R n g(x )d = 0; h j (x )d = 0, j J, j=1 { } where J = j h j (x ) = 0, µ j > 0. Then x is a strict local minimum of f subject to the full constraints.

53 Ways to compute the sufficient conditions The nature of the Hessian of the Lagrangian function and with its auxiliary conditions can be determined. 1. Eigenvalues on the subspace T. 2. Find the nature of the Bordered Hessian.

54 Eigenvalues on subspaces Let {e i } m i=1 be an orthonormal basis of T. Then d T can be written, d = Ez, where z R m and, E = [e 1 e 2... e m ]. Then use, d T 2 Ld =...,

55 Bordered Hessian Alternative method to determining the behaviour of the Lagragian Hessian. Definition Given an objective function f with constraints g : R n R m, define the Bordered Hessian matrix, ( 2 L(x, λ) g(x) T ) B(x, λ) := g(x) 0 Find the nature of the Bordered Hessian.

Lagrange Multipliers

Lagrange Multipliers Lagrange Multipliers (Com S 477/577 Notes) Yan-Bin Jia Nov 9, 2017 1 Introduction We turn now to the study of minimization with constraints. More specifically, we will tackle the following problem: minimize

More information

The Kuhn-Tucker Problem

The Kuhn-Tucker Problem Natalia Lazzati Mathematics for Economics (Part I) Note 8: Nonlinear Programming - The Kuhn-Tucker Problem Note 8 is based on de la Fuente (2000, Ch. 7) and Simon and Blume (1994, Ch. 18 and 19). The Kuhn-Tucker

More information

3.7 Constrained Optimization and Lagrange Multipliers

3.7 Constrained Optimization and Lagrange Multipliers 3.7 Constrained Optimization and Lagrange Multipliers 71 3.7 Constrained Optimization and Lagrange Multipliers Overview: Constrained optimization problems can sometimes be solved using the methods of the

More information

Constrained Optimization

Constrained Optimization Constrained Optimization Joshua Wilde, revised by Isabel Tecu, Takeshi Suzuki and María José Boccardi August 13, 2013 1 General Problem Consider the following general constrained optimization problem:

More information

Optimality Conditions

Optimality Conditions Chapter 2 Optimality Conditions 2.1 Global and Local Minima for Unconstrained Problems When a minimization problem does not have any constraints, the problem is to find the minimum of the objective function.

More information

3. Minimization with constraints Problem III. Minimize f(x) in R n given that x satisfies the equality constraints. g j (x) = c j, j = 1,...

3. Minimization with constraints Problem III. Minimize f(x) in R n given that x satisfies the equality constraints. g j (x) = c j, j = 1,... 3. Minimization with constraints Problem III. Minimize f(x) in R n given that x satisfies the equality constraints g j (x) = c j, j = 1,..., m < n, where c 1,..., c m are given numbers. Theorem 3.1. Let

More information

SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING

SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING Nf SECTION C: CONTINUOUS OPTIMISATION LECTURE 9: FIRST ORDER OPTIMALITY CONDITIONS FOR CONSTRAINED NONLINEAR PROGRAMMING f(x R m g HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 5, DR RAPHAEL

More information

Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006 Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

More information

EC /11. Math for Microeconomics September Course, Part II Lecture Notes. Course Outline

EC /11. Math for Microeconomics September Course, Part II Lecture Notes. Course Outline LONDON SCHOOL OF ECONOMICS Professor Leonardo Felli Department of Economics S.478; x7525 EC400 20010/11 Math for Microeconomics September Course, Part II Lecture Notes Course Outline Lecture 1: Tools for

More information

Math 291-3: Lecture Notes Northwestern University, Spring 2016

Math 291-3: Lecture Notes Northwestern University, Spring 2016 Math 291-3: Lecture Notes Northwestern University, Spring 216 Written by Santiago Cañez These are lecture notes for Math 291-3, the third quarter of MENU: Intensive Linear Algebra and Multivariable Calculus,

More information

Lecture 9: Implicit function theorem, constrained extrema and Lagrange multipliers

Lecture 9: Implicit function theorem, constrained extrema and Lagrange multipliers Lecture 9: Implicit function theorem, constrained extrema and Lagrange multipliers Rafikul Alam Department of Mathematics IIT Guwahati What does the Implicit function theorem say? Let F : R 2 R be C 1.

More information

Lagrange Multipliers

Lagrange Multipliers Optimization with Constraints As long as algebra and geometry have been separated, their progress have been slow and their uses limited; but when these two sciences have been united, they have lent each

More information

Tangent spaces, normals and extrema

Tangent spaces, normals and extrema Chapter 3 Tangent spaces, normals and extrema If S is a surface in 3-space, with a point a S where S looks smooth, i.e., without any fold or cusp or self-crossing, we can intuitively define the tangent

More information

Computation. For QDA we need to calculate: Lets first consider the case that

Computation. For QDA we need to calculate: Lets first consider the case that Computation For QDA we need to calculate: δ (x) = 1 2 log( Σ ) 1 2 (x µ ) Σ 1 (x µ ) + log(π ) Lets first consider the case that Σ = I,. This is the case where each distribution is spherical, around the

More information

Lagrange multipliers. Portfolio optimization. The Lagrange multipliers method for finding constrained extrema of multivariable functions.

Lagrange multipliers. Portfolio optimization. The Lagrange multipliers method for finding constrained extrema of multivariable functions. Chapter 9 Lagrange multipliers Portfolio optimization The Lagrange multipliers method for finding constrained extrema of multivariable functions 91 Lagrange multipliers Optimization problems often require

More information

Optimization Theory. Lectures 4-6

Optimization Theory. Lectures 4-6 Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x$0} Existence of a maximum:

More information

Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization. Nick Gould (RAL)

Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization. Nick Gould (RAL) Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization Nick Gould (RAL) x IR n f(x) subject to c(x) = Part C course on continuoue optimization CONSTRAINED MINIMIZATION x

More information

MATH20132 Calculus of Several Variables. 2018

MATH20132 Calculus of Several Variables. 2018 MATH20132 Calculus of Several Variables 2018 Solutions to Problems 8 Lagrange s Method 1 For x R 2 let fx = x 2 3xy +y 2 5x+5y i Find the critical values of fx in R 2, ii Findthecriticalvaluesoffxrestrictedtotheparametriccurvet

More information

Ph.D. Katarína Bellová Page 1 Mathematics 2 (10-PHY-BIPMA2) EXAM - Solutions, 20 July 2017, 10:00 12:00 All answers to be justified.

Ph.D. Katarína Bellová Page 1 Mathematics 2 (10-PHY-BIPMA2) EXAM - Solutions, 20 July 2017, 10:00 12:00 All answers to be justified. PhD Katarína Bellová Page 1 Mathematics 2 (10-PHY-BIPMA2 EXAM - Solutions, 20 July 2017, 10:00 12:00 All answers to be justified Problem 1 [ points]: For which parameters λ R does the following system

More information

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

More information

Quadratic Programming

Quadratic Programming Quadratic Programming Outline Linearly constrained minimization Linear equality constraints Linear inequality constraints Quadratic objective function 2 SideBar: Matrix Spaces Four fundamental subspaces

More information

Constrained optimization: direct methods (cont.)

Constrained optimization: direct methods (cont.) Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a

More information

Numerical optimization

Numerical optimization Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal

More information

Preliminary draft only: please check for final version

Preliminary draft only: please check for final version ARE211, Fall2012 CALCULUS4: THU, OCT 11, 2012 PRINTED: AUGUST 22, 2012 (LEC# 15) Contents 3. Univariate and Multivariate Differentiation (cont) 1 3.6. Taylor s Theorem (cont) 2 3.7. Applying Taylor theory:

More information

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

More information

Convex Optimization and Modeling

Convex Optimization and Modeling Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

More information

Lecture 2: Convex Sets and Functions

Lecture 2: Convex Sets and Functions Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are

More information

Mathematical Foundations -1- Constrained Optimization. Constrained Optimization. An intuitive approach 2. First Order Conditions (FOC) 7

Mathematical Foundations -1- Constrained Optimization. Constrained Optimization. An intuitive approach 2. First Order Conditions (FOC) 7 Mathematical Foundations -- Constrained Optimization Constrained Optimization An intuitive approach First Order Conditions (FOC) 7 Constraint qualifications 9 Formal statement of the FOC for a maximum

More information

ECE580 Solution to Problem Set 6

ECE580 Solution to Problem Set 6 ECE580 Fall 2015 Solution to Problem Set 6 December 23 2015 1 ECE580 Solution to Problem Set 6 These problems are from the textbook by Chong and Zak 4th edition which is the textbook for the ECE580 Fall

More information

Here each term has degree 2 (the sum of exponents is 2 for all summands). A quadratic form of three variables looks as

Here each term has degree 2 (the sum of exponents is 2 for all summands). A quadratic form of three variables looks as Reading [SB], Ch. 16.1-16.3, p. 375-393 1 Quadratic Forms A quadratic function f : R R has the form f(x) = a x. Generalization of this notion to two variables is the quadratic form Q(x 1, x ) = a 11 x

More information

Computational Optimization. Constrained Optimization Part 2

Computational Optimization. Constrained Optimization Part 2 Computational Optimization Constrained Optimization Part Optimality Conditions Unconstrained Case X* is global min Conve f X* is local min SOSC f ( *) = SONC Easiest Problem Linear equality constraints

More information

Date: July 5, Contents

Date: July 5, Contents 2 Lagrange Multipliers Date: July 5, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 14 2.3. Informative Lagrange Multipliers...........

More information

Chapter 3 Numerical Methods

Chapter 3 Numerical Methods Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2

More information

1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016

1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016 AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 8 February 22nd 1 Overview In the previous lecture we saw characterizations of optimality in linear optimization, and we reviewed the

More information

Lagrange Multipliers

Lagrange Multipliers Calculus 3 Lia Vas Lagrange Multipliers Constrained Optimization for functions of two variables. To find the maximum and minimum values of z = f(x, y), objective function, subject to a constraint g(x,

More information

Chap 2. Optimality conditions

Chap 2. Optimality conditions Chap 2. Optimality conditions Version: 29-09-2012 2.1 Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1

More information

Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725

Karush-Kuhn-Tucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36-725 Karush-Kuhn-Tucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10-725/36-725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =

More information

ARE211, Fall2015. Contents. 2. Univariate and Multivariate Differentiation (cont) Taylor s Theorem (cont) 2

ARE211, Fall2015. Contents. 2. Univariate and Multivariate Differentiation (cont) Taylor s Theorem (cont) 2 ARE211, Fall2015 CALCULUS4: THU, SEP 17, 2015 PRINTED: SEPTEMBER 22, 2015 (LEC# 7) Contents 2. Univariate and Multivariate Differentiation (cont) 1 2.4. Taylor s Theorem (cont) 2 2.5. Applying Taylor theory:

More information

1 Convexity, concavity and quasi-concavity. (SB )

1 Convexity, concavity and quasi-concavity. (SB ) UNIVERSITY OF MARYLAND ECON 600 Summer 2010 Lecture Two: Unconstrained Optimization 1 Convexity, concavity and quasi-concavity. (SB 21.1-21.3.) For any two points, x, y R n, we can trace out the line of

More information

CE 191: Civil and Environmental Engineering Systems Analysis. LEC 05 : Optimality Conditions

CE 191: Civil and Environmental Engineering Systems Analysis. LEC 05 : Optimality Conditions CE 191: Civil and Environmental Engineering Systems Analysis LEC : Optimality Conditions Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 214 Prof. Moura

More information

Optimization Tutorial 1. Basic Gradient Descent

Optimization Tutorial 1. Basic Gradient Descent E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.

More information

Math 273a: Optimization Basic concepts

Math 273a: Optimization Basic concepts Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize

More information

CHAPTER 2: QUADRATIC PROGRAMMING

CHAPTER 2: QUADRATIC PROGRAMMING CHAPTER 2: QUADRATIC PROGRAMMING Overview Quadratic programming (QP) problems are characterized by objective functions that are quadratic in the design variables, and linear constraints. In this sense,

More information

Newton-Type Algorithms for Nonlinear Constrained Optimization

Newton-Type Algorithms for Nonlinear Constrained Optimization Newton-Type Algorithms for Nonlinear Constrained Optimization Angelika Altmann-Dieses Faculty of Management Science and Engineering Karlsruhe University of Applied Sciences Moritz Diehl Department of Microsystems

More information

MATHEMATICS FOR ECONOMISTS. An Introductory Textbook. Third Edition. Malcolm Pemberton and Nicholas Rau. UNIVERSITY OF TORONTO PRESS Toronto Buffalo

MATHEMATICS FOR ECONOMISTS. An Introductory Textbook. Third Edition. Malcolm Pemberton and Nicholas Rau. UNIVERSITY OF TORONTO PRESS Toronto Buffalo MATHEMATICS FOR ECONOMISTS An Introductory Textbook Third Edition Malcolm Pemberton and Nicholas Rau UNIVERSITY OF TORONTO PRESS Toronto Buffalo Contents Preface Dependence of Chapters Answers and Solutions

More information

Advanced Calculus Notes. Michael P. Windham

Advanced Calculus Notes. Michael P. Windham Advanced Calculus Notes Michael P. Windham February 25, 2008 Contents 1 Basics 5 1.1 Euclidean Space R n............................ 5 1.2 Functions................................. 7 1.2.1 Sets and functions........................

More information

Let f(x) = x, but the domain of f is the interval 0 x 1. Note

Let f(x) = x, but the domain of f is the interval 0 x 1. Note I.g Maximum and Minimum. Lagrange Multipliers Recall: Suppose we are given y = f(x). We recall that the maximum/minimum points occur at the following points: (1) where f = 0; (2) where f does not exist;

More information

STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II - KUHN TUCKER THEORY

STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II - KUHN TUCKER THEORY STATIC LECTURE 4: CONSTRAINED OPTIMIZATION II - KUHN TUCKER THEORY UNIVERSITY OF MARYLAND: ECON 600 1. Some Eamples 1 A general problem that arises countless times in economics takes the form: (Verbally):

More information

Applied Lagrange Duality for Constrained Optimization

Applied Lagrange Duality for Constrained Optimization Applied Lagrange Duality for Constrained Optimization February 12, 2002 Overview The Practical Importance of Duality ffl Review of Convexity ffl A Separating Hyperplane Theorem ffl Definition of the Dual

More information

Review of Optimization Basics

Review of Optimization Basics Review of Optimization Basics. Introduction Electricity markets throughout the US are said to have a two-settlement structure. The reason for this is that the structure includes two different markets:

More information

CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008.

CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008. 1 ECONOMICS 594: LECTURE NOTES CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS W. Erwin Diewert January 31, 2008. 1. Introduction Many economic problems have the following structure: (i) a linear function

More information

CONVEX FUNCTIONS AND OPTIMIZATION TECHINIQUES A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF

CONVEX FUNCTIONS AND OPTIMIZATION TECHINIQUES A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF CONVEX FUNCTIONS AND OPTIMIZATION TECHINIQUES A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE IN MATHEMATICS SUBMITTED TO NATIONAL INSTITUTE OF TECHNOLOGY,

More information

10 Numerical methods for constrained problems

10 Numerical methods for constrained problems 10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside

More information

Nonlinear Programming (NLP)

Nonlinear Programming (NLP) Natalia Lazzati Mathematics for Economics (Part I) Note 6: Nonlinear Programming - Unconstrained Optimization Note 6 is based on de la Fuente (2000, Ch. 7), Madden (1986, Ch. 3 and 5) and Simon and Blume

More information

Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

More information

Lecture Support Vector Machine (SVM) Classifiers

Lecture Support Vector Machine (SVM) Classifiers Introduction to Machine Learning Lecturer: Amir Globerson Lecture 6 Fall Semester Scribe: Yishay Mansour 6.1 Support Vector Machine (SVM) Classifiers Classification is one of the most important tasks in

More information

In English, this means that if we travel on a straight line between any two points in C, then we never leave C.

In English, this means that if we travel on a straight line between any two points in C, then we never leave C. Convex sets In this section, we will be introduced to some of the mathematical fundamentals of convex sets. In order to motivate some of the definitions, we will look at the closest point problem from

More information

You should be able to...

You should be able to... Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set

More information

Implicit Functions, Curves and Surfaces

Implicit Functions, Curves and Surfaces Chapter 11 Implicit Functions, Curves and Surfaces 11.1 Implicit Function Theorem Motivation. In many problems, objects or quantities of interest can only be described indirectly or implicitly. It is then

More information

0, otherwise. Find each of the following limits, or explain that the limit does not exist.

0, otherwise. Find each of the following limits, or explain that the limit does not exist. Midterm Solutions 1, y x 4 1. Let f(x, y) = 1, y 0 0, otherwise. Find each of the following limits, or explain that the limit does not exist. (a) (b) (c) lim f(x, y) (x,y) (0,1) lim f(x, y) (x,y) (2,3)

More information

Lecture: Quadratic optimization

Lecture: Quadratic optimization Lecture: Quadratic optimization 1. Positive definite och semidefinite matrices 2. LDL T factorization 3. Quadratic optimization without constraints 4. Quadratic optimization with constraints 5. Least-squares

More information

39.1 Absolute maxima/minima

39.1 Absolute maxima/minima Module 13 : Maxima, Minima Saddle Points, Constrained maxima minima Lecture 39 : Absolute maxima / minima [Section 39.1] Objectives In this section you will learn the following : The notion of absolute

More information

Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:

Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem: CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

More information

Topic III.2: Maximum Entropy Models

Topic III.2: Maximum Entropy Models Topic III.2: Maximum Entropy Models Discrete Topics in Data Mining Universität des Saarlandes, Saarbrücken Winter Semester 2012/13 T III.2-1 Topic III.2: Maximum Entropy Models 1. The Maximum Entropy Principle

More information

Assignment 1: From the Definition of Convexity to Helley Theorem

Assignment 1: From the Definition of Convexity to Helley Theorem Assignment 1: From the Definition of Convexity to Helley Theorem Exercise 1 Mark in the following list the sets which are convex: 1. {x R 2 : x 1 + i 2 x 2 1, i = 1,..., 10} 2. {x R 2 : x 2 1 + 2ix 1x

More information

1 Invariant subspaces

1 Invariant subspaces MATH 2040 Linear Algebra II Lecture Notes by Martin Li Lecture 8 Eigenvalues, eigenvectors and invariant subspaces 1 In previous lectures we have studied linear maps T : V W from a vector space V to another

More information

MAT 419 Lecture Notes Transcribed by Eowyn Cenek 6/1/2012

MAT 419 Lecture Notes Transcribed by Eowyn Cenek 6/1/2012 (Homework 1: Chapter 1: Exercises 1-7, 9, 11, 19, due Monday June 11th See also the course website for lectures, assignments, etc) Note: today s lecture is primarily about definitions Lots of definitions

More information

Linear vector spaces and subspaces.

Linear vector spaces and subspaces. Math 2051 W2008 Margo Kondratieva Week 1 Linear vector spaces and subspaces. Section 1.1 The notion of a linear vector space. For the purpose of these notes we regard (m 1)-matrices as m-dimensional vectors,

More information

Dimensions = xyz dv. xyz dv as an iterated integral in rectangular coordinates.

Dimensions = xyz dv. xyz dv as an iterated integral in rectangular coordinates. Math Show Your Work! Page of 8. () A rectangular box is to hold 6 cubic meters. The material used for the top and bottom of the box is twice as expensive per square meter than the material used for the

More information

NONLINEAR PROGRAMMING WITH CONSTRAINTS

NONLINEAR PROGRAMMING WITH CONSTRAINTS NONLINEAR PROGRAMMING WITH CONSTRAINTS 8.1 Directsubstitution... 265 8.2 First-Order ~ecessary Conditions for a Local Extremum... 267 8.3 Quadratic Programming... 284 8.4 Penalty. Barrier. and Augmented

More information

3E4: Modelling Choice. Introduction to nonlinear programming. Announcements

3E4: Modelling Choice. Introduction to nonlinear programming. Announcements 3E4: Modelling Choice Lecture 7 Introduction to nonlinear programming 1 Announcements Solutions to Lecture 4-6 Homework will be available from http://www.eng.cam.ac.uk/~dr241/3e4 Looking ahead to Lecture

More information

EC400 Math for Microeconomics Syllabus The course is based on 6 sixty minutes lectures and on 6 ninety minutes classes.

EC400 Math for Microeconomics Syllabus The course is based on 6 sixty minutes lectures and on 6 ninety minutes classes. London School of Economics Department of Economics Dr Francesco Nava Offi ce: 32L.3.20 EC400 Math for Microeconomics Syllabus 2016 The course is based on 6 sixty minutes lectures and on 6 ninety minutes

More information

Optimization using Calculus. Optimization of Functions of Multiple Variables subject to Equality Constraints

Optimization using Calculus. Optimization of Functions of Multiple Variables subject to Equality Constraints Optimization using Calculus Optimization of Functions of Multiple Variables subject to Equality Constraints 1 Objectives Optimization of functions of multiple variables subjected to equality constraints

More information

EE 227A: Convex Optimization and Applications October 14, 2008

EE 227A: Convex Optimization and Applications October 14, 2008 EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider

More information

CHAPTER 1-2: SHADOW PRICES

CHAPTER 1-2: SHADOW PRICES Essential Microeconomics -- CHAPTER -: SHADOW PRICES An intuitive approach: profit maimizing firm with a fied supply of an input Shadow prices 5 Concave maimization problem 7 Constraint qualifications

More information

MULTIVARIABLE CALCULUS, LINEAR ALGEBRA, AND DIFFERENTIAL EQUATIONS

MULTIVARIABLE CALCULUS, LINEAR ALGEBRA, AND DIFFERENTIAL EQUATIONS T H I R D E D I T I O N MULTIVARIABLE CALCULUS, LINEAR ALGEBRA, AND DIFFERENTIAL EQUATIONS STANLEY I. GROSSMAN University of Montana and University College London SAUNDERS COLLEGE PUBLISHING HARCOURT BRACE

More information

Differentiation. f(x + h) f(x) Lh = L.

Differentiation. f(x + h) f(x) Lh = L. Analysis in R n Math 204, Section 30 Winter Quarter 2008 Paul Sally, e-mail: sally@math.uchicago.edu John Boller, e-mail: boller@math.uchicago.edu website: http://www.math.uchicago.edu/ boller/m203 Differentiation

More information

Convex Optimization Lecture 6: KKT Conditions, and applications

Convex Optimization Lecture 6: KKT Conditions, and applications Convex Optimization Lecture 6: KKT Conditions, and applications Dr. Michel Baes, IFOR / ETH Zürich Quick recall of last week s lecture Various aspects of convexity: The set of minimizers is convex. Convex

More information

Lecture 6 Positive Definite Matrices

Lecture 6 Positive Definite Matrices Linear Algebra Lecture 6 Positive Definite Matrices Prof. Chun-Hung Liu Dept. of Electrical and Computer Engineering National Chiao Tung University Spring 2017 2017/6/8 Lecture 6: Positive Definite Matrices

More information

Mathematical Economics: Lecture 16

Mathematical Economics: Lecture 16 Mathematical Economics: Lecture 16 Yu Ren WISE, Xiamen University November 26, 2012 Outline 1 Chapter 21: Concave and Quasiconcave Functions New Section Chapter 21: Concave and Quasiconcave Functions Concave

More information

z = f (x; y) f (x ; y ) f (x; y) f (x; y )

z = f (x; y) f (x ; y ) f (x; y) f (x; y ) BEEM0 Optimization Techiniques for Economists Lecture Week 4 Dieter Balkenborg Departments of Economics University of Exeter Since the fabric of the universe is most perfect, and is the work of a most

More information

(Here > 0 is given.) In cases where M is convex, there is a nice theory for this problem; the theory has much more general applicability too.

(Here > 0 is given.) In cases where M is convex, there is a nice theory for this problem; the theory has much more general applicability too. Convex Analysis with Applications UBC Math 604 Lecture Notes by Philip D. Loewen In trust region methods, we minimize a quadratic model function M = M(p) over the set of all p R n satisfying a constraint

More information

Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88

Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88 Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant

More information

Optimisation theory. The International College of Economics and Finance. I. An explanatory note. The Lecturer and Class teacher: Kantorovich G.G.

Optimisation theory. The International College of Economics and Finance. I. An explanatory note. The Lecturer and Class teacher: Kantorovich G.G. The International College of Economics and Finance The course syllabus Optimisation theory The eighth semester I. An explanatory note The Lecturer and Class teacher: Kantorovich G.G. Requirements to students:

More information

HOMEWORK 7 SOLUTIONS

HOMEWORK 7 SOLUTIONS HOMEWORK 7 SOLUTIONS MA11: ADVANCED CALCULUS, HILARY 17 (1) Using the method of Lagrange multipliers, find the largest and smallest values of the function f(x, y) xy on the ellipse x + y 1. Solution: The

More information

1 The linear algebra of linear programs (March 15 and 22, 2015)

1 The linear algebra of linear programs (March 15 and 22, 2015) 1 The linear algebra of linear programs (March 15 and 22, 2015) Many optimization problems can be formulated as linear programs. The main features of a linear program are the following: Variables are real

More information

Support Vector Machines and Kernel Methods

Support Vector Machines and Kernel Methods 2018 CS420 Machine Learning, Lecture 3 Hangout from Prof. Andrew Ng. http://cs229.stanford.edu/notes/cs229-notes3.pdf Support Vector Machines and Kernel Methods Weinan Zhang Shanghai Jiao Tong University

More information

8 Barrier Methods for Constrained Optimization

8 Barrier Methods for Constrained Optimization IOE 519: NL, Winter 2012 c Marina A. Epelman 55 8 Barrier Methods for Constrained Optimization In this subsection, we will restrict our attention to instances of constrained problem () that have inequality

More information

Lecture 6: Conic Optimization September 8

Lecture 6: Conic Optimization September 8 IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

More information

4 Partial Differentiation

4 Partial Differentiation 4 Partial Differentiation Many equations in engineering, physics and mathematics tie together more than two variables. For example Ohm s Law (V = IR) and the equation for an ideal gas, PV = nrt, which

More information

Symmetric and Asymmetric Duality

Symmetric and Asymmetric Duality journal of mathematical analysis and applications 220, 125 131 (1998) article no. AY975824 Symmetric and Asymmetric Duality Massimo Pappalardo Department of Mathematics, Via Buonarroti 2, 56127, Pisa,

More information

Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2)

Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2) Week 4: Calculus and Optimization (Jehle and Reny, Chapter A2) Tsun-Feng Chiang *School of Economics, Henan University, Kaifeng, China September 27, 2015 Microeconomic Theory Week 4: Calculus and Optimization

More information

Module 04 Optimization Problems KKT Conditions & Solvers

Module 04 Optimization Problems KKT Conditions & Solvers Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September

More information

A geometric proof of the spectral theorem for real symmetric matrices

A geometric proof of the spectral theorem for real symmetric matrices 0 0 0 A geometric proof of the spectral theorem for real symmetric matrices Robert Sachs Department of Mathematical Sciences George Mason University Fairfax, Virginia 22030 rsachs@gmu.edu January 6, 2011

More information

Math 263 Assignment #4 Solutions. 0 = f 1 (x,y,z) = 2x 1 0 = f 2 (x,y,z) = z 2 0 = f 3 (x,y,z) = y 1

Math 263 Assignment #4 Solutions. 0 = f 1 (x,y,z) = 2x 1 0 = f 2 (x,y,z) = z 2 0 = f 3 (x,y,z) = y 1 Math 263 Assignment #4 Solutions 1. Find and classify the critical points of each of the following functions: (a) f(x,y,z) = x 2 + yz x 2y z + 7 (c) f(x,y) = e x2 y 2 (1 e x2 ) (b) f(x,y) = (x + y) 3 (x

More information

Maths for Signals and Systems Linear Algebra in Engineering

Maths for Signals and Systems Linear Algebra in Engineering Maths for Signals and Systems Linear Algebra in Engineering Lectures 13 15, Tuesday 8 th and Friday 11 th November 016 DR TANIA STATHAKI READER (ASSOCIATE PROFFESOR) IN SIGNAL PROCESSING IMPERIAL COLLEGE

More information

Finite Dimensional Optimization Part I: The KKT Theorem 1

Finite Dimensional Optimization Part I: The KKT Theorem 1 John Nachbar Washington University March 26, 2018 1 Introduction Finite Dimensional Optimization Part I: The KKT Theorem 1 These notes characterize maxima and minima in terms of first derivatives. I focus

More information

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

More information

MS&E 318 (CME 338) Large-Scale Numerical Optimization

MS&E 318 (CME 338) Large-Scale Numerical Optimization Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods

More information

B553 Lecture 5: Matrix Algebra Review

B553 Lecture 5: Matrix Algebra Review B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations

More information