# E5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization

Save this PDF as:
Size: px
Start display at page:

Download "E5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization"

## Transcription

1 E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007

2 Unconstrained convex program Consider a convex optimization problem on the form (CP ) minimize x IR n f(x), where f : IR n IR is convex and twice continuously differentiable. Proposition. Let p denote the optimal value of (CP ). Let x in IR n, let m = η min ( 2 f(x)) and let M = η max ( 2 f(x)). Then, f(x) 1 2m f(x) 2 2 p f(x) 1 2M f(x) 2 2. Note that ỹ = x (1/m) f(x) minimizes f(x) T (y x) + m 2 y x 2 2. The direction (1/m) f(x) is a steepest descent step. A. Forsgren, KTH 2 Lecture 8 Convex optimization 2006/2007

3 Iterative methods (CP ) minimize f(x) subject to x IR n, where f C 2, f convex on IR n. An iterative method generates x 0, x 1, x 2,... such that lim k x k = x, where f(x ) = 0. Terminates when suitable convergence criteria is fulfilled, e.g., f(x k ) < ɛ. A. Forsgren, KTH 3 Lecture 8 Convex optimization 2006/2007

4 Linesearch methods A linesearch method generates in each iteration a search direction and performs a linesearch along the search direction. Iteration k takes the following form at x k. ˆ Compute search direction p k such that f(x k ) T p k < 0. ˆ Approximatively solve min α 0 f(x k + αp k ), which gives α k. ˆ x k+1 x k + α k p k. Different methods vary in choice of p k and α k. A. Forsgren, KTH 4 Lecture 8 Convex optimization 2006/2007

5 Classes of linesearch methods We will initially consider two fundamental methods. ˆ The steepest-descent method, where p k = f(x k ), and ˆ Newton s method, where 2 f(x k )p k = f(x k ). Steepest descent: + Search direction inexpensive to compute, Slow convergence. Newton s method: Search direction more expensive to compute, + Faster convergence. There are methods in-between, e.g., quasi-newton methods that aim at mimicking Newton s method without computing second derivatives. A. Forsgren, KTH 5 Lecture 8 Convex optimization 2006/2007

6 Quadratic objective function Consider model problem with quadratic objective function (QP ) minimize subject to x IR n, f(x) = 1 2 xt Hx + c T x where H 0. Proposition. The following holds for (QP ) depending on H and c: (i) If H 0 then (QP ) has a unique minimizer x given by Hx = c. (ii) If H 0, H 0 each x that fulfills Hx = c is a global minimizer to (QP ). (There may possibly be no such x ). Proof. The condition f(x ) = 0 gives the results. We assume H 0 in the discussion. A. Forsgren, KTH 6 Lecture 8 Convex optimization 2006/2007

7 Linesearch method on quadratic objective function Consider (QP ) with f(x) = 1 2 xt Hx + c T x, where H 0. We obtain: x minimizer to (QP ) 0 = f(x ) = Hx + c. Suppose search direction p k satisfies f(x k ) T p k < 0. Let ϕ(α) = f(x k + αp k ) = f(x k ) + α f(x k ) T p k + α2 2 pt khp k. Then α k = f(x k) T p k p T k Hp gives the minimizer to min α 0 f(x k + αp k ), k i.e., we can perform exact linesearch. A. Forsgren, KTH 7 Lecture 8 Convex optimization 2006/2007

8 Steepest descent on quadratic objective function Assume that f(x) = 1 2 xt Hx + c T x, where H 0. Further assume that steepest descent with exact linesearch is used, i.e., p k = f(x k ) and α k = f(x k) T p k p T k Hp k Then it can be shown ( that ) f(x k+1 ) f(x 2 cond(h) 1 ) (f(x k ) f(x )). cond(h) + 1 cond(h) 1 cond(h) 1 cond(h) , i.e., slow linear convergence. For a nonlinear function, we typically get slow linear convergence, where H is replaced by 2 f(x k ). A. Forsgren, KTH 8 Lecture 8 Convex optimization 2006/2007

9 Speed of convergence Definition. Assume that x k IR n, k = 0, 1,..., and assume that lim k x k = x. We say that {x k } k=0 converges to x with speed of convergence r if lim k x k+1 x x k x r = C, where C <. We have x k+1 x C x k x r. We want r large (and C close to zero). Of interest: ˆ r = 1, 0 < C < 1, linear convergence. (Steepest descent.) ˆ r = 1, C = 0, superlinear convergence. (Quasi-Newton.) ˆ r = 2, quadratic convergence. (Newton s method.) A. Forsgren, KTH 9 Lecture 8 Convex optimization 2006/2007

10 Newton s method for solving a nonlinear equation Consider solving the nonlinear equation f(u) = 0, where f : IR n IR, f C 2. Then, f(u + p) = f(u) + 2 f(u)p + o( p ). Linearization given by f(u) + 2 f(u)p. Choose p so that f(u) + 2 f(u)p = 0, i.e., solve 2 f(u)p = f(u). A Newton iteration takes the following form for a given u. ˆ p solves 2 f(u)p = f(u). ˆ u u + p. (The nonlinear equation need not be a gradient.) A. Forsgren, KTH 10 Lecture 8 Convex optimization 2006/2007

11 Speed of convergence for Newton s method Theorem. Assume that f C 3 and that 2 f(u ) is nonsingular. Then, if Newton s method (with steplength one) is started at a point sufficiently close to u, then it is well defined and converges to u with convergence rate at least two, i.e., there is a constant C such that u k+1 u C u k u 2. The proof can be given by studying a Taylor-series expansion, u k+1 u = u k 2 f(u k ) 1 f(u k ) u = 2 f(u k ) 1 ( f(u ) f(u k ) + 2 f(u k )(u u k )). For u k sufficiently close to u, f(u ) f(u k ) + 2 f(u k )(u u k ) C u k u 2. A. Forsgren, KTH 11 Lecture 8 Convex optimization 2006/2007

12 One-dimensional example for Newton s method For a positive number d, consider computing 1/d by minimizing f(u) = du ln u. Then, f (u) = d 1 u, f (u) = 1 u 2. We see that u = 1 d. u k+1 = u k f (u k ) f (u k ) = u k d 1 u k 1 u 2 k = 2u k u 2 kd. Then, u k+1 1 d = 2u k u 2 kd 1 d = d ( u k 1 d) 2. A. Forsgren, KTH 12 Lecture 8 Convex optimization 2006/2007

13 One-dimensional example for Newton s method, cont. Graphical picture for d = 2. A. Forsgren, KTH 13 Lecture 8 Convex optimization 2006/2007

14 Sufficient descent direction For a direction p k to ensure convergence is must be a sufficient descent direction. Typical conditions are f(x k) T p k f(x k ) p k σ, where σ is a positive constant. This means that p k must be sufficiently similar to the negative gradient. For search direction p k from B k p k = f(x k ) this is required by ensuring that B k M and Bk 1 m, where m and M are positive constants. In a modified Newton method modifications of 2 f(x k ) can be made, if needed, by a modified Cholesky factorization. A. Forsgren, KTH 14 Lecture 8 Convex optimization 2006/2007

15 Linesearch In the linesearch α k is determined as an approximate solution to min α 0 ϕ(α), where ϕ(α) = f(x k + αp k ). We want f(x k+1 ) < f(x k ), i.e., ϕ(α k ) < ϕ(0). This is not sufficient to ensure convergence. Example requirement for step not too long: ϕ(α) ϕ(0) + µαϕ (0), i.e., f(x k + αp k ) f(x k ) + µα f(x k ) T p k, where µ (0, 1). 2 (Armijo condition) Example requirement for step not too short: ϕ (α) ηϕ (0), i.e., (Wolfe condition) f(x k + αp k ) T p k η f(x k ) T p k, where η (µ, 1). Alternative requirement for not too short step: Take smallest nonnegative integer i such that f(x k + 2 i p k ) f(x k ) + µ2 i f(x k ) T p k. ( Backtracking ) A. Forsgren, KTH 15 Lecture 8 Convex optimization 2006/2007

16 Illustration of linesearch conditions The linesearch conditions of Wolfe-Armijo type can be illustrated in the following picture. A. Forsgren, KTH 16 Lecture 8 Convex optimization 2006/2007

17 Linesearch conditions To find ᾱ such that the Wolfe and Armijo conditions are fulfilled we may consider ˆϕ(α) = ϕ(α) ϕ(0) µαϕ (0). Then ˆϕ(0) = 0 and ˆϕ (0) < 0. In addition, there must exist ᾱ > 0 such that ˆϕ(ᾱ) = 0, otherwise ϕ is unbounded from below. By the mean-value theorem there is an ˆα (0, ᾱ) such that ˆϕ( ˆα) < 0 and ˆϕ ( ˆα) = 0. Since µ < η we obtain ϕ(α) ϕ(0) + µαϕ (0) and ϕ (α) ηϕ (0) for α in a neighborhood of ˆα. For example, bisection in combination with polynomial interpolation can be used on ˆϕ to find a suitable α. A. Forsgren, KTH 17 Lecture 8 Convex optimization 2006/2007

18 Newton s method and steepest descent The steepest descent direction solves minimize p IR n f(x) T p subject to p T p 1. The Newton direction solves minimize p IR n f(x) T p subject to p T 2 f(x)p 1. The Newton step is a steepest-descent step in the norm defined by 2 f(x), i.e., u 2 f(x) = (u T 2 f(x)u) 1/2. A. Forsgren, KTH 18 Lecture 8 Convex optimization 2006/2007

19 Self-concordant functions When proving polynomial complexity of interior methods for convex optimization, the notion of self-concordant functions is an important concept. Definition. A three times differentiable function f : C IR, which is convex on the convex set C, is self-concordant if f (x) 2f (x) 3/2. In essence, this means that the third derivatives are not too large. An important self-concordant function is f(x) = ln x for x > 0. A. Forsgren, KTH 19 Lecture 8 Convex optimization 2006/2007

20 Suggested reading Suggested reading in the textbook: ˆ Sections A. Forsgren, KTH 20 Lecture 8 Convex optimization 2006/2007

21 Equality-constrained convex program Consider a convex optimization problem on the form (CP = ) minimize x IR n f(x) subject to Ax = b, where f : IR n IR is convex and twice continuously differentiable. If the Lagrangian function is defined as l(x, λ) = f(x) λ T (Ax b), the first-order optimality conditions are l(x, λ) = 0. We write them as xl(x, λ) λ l(x, λ) = f(x) A(x)T λ Ax b = 0 0. A. Forsgren, KTH 21 Lecture 8 Convex optimization 2006/2007

22 Newton iteration A Newton iteration on the optimality conditions takes the form A T 2 f(x) A 0 p ν = f(x) AT λ Ax b. We may use f(x) AT λ Ax b 2 as merit function, i.e., to measure how good a point is. A. Forsgren, KTH 22 Lecture 8 Convex optimization 2006/2007

23 Variable elimination Note that for a feasible point x, it holds that A(x x) = 0 for all feasible x. Let Z be a matrix whose columns form a basis for null(a). Then x = x + Zv, with a one-to-one correspondence between x and v. Let ϕ(v) = f( x + Zv). We may then rewrite the problem as (CP =) minimize v IR n m ϕ(v). Differentiation gives ϕ(v) = Z T f( x + Zv), 2 ϕ(v) = Z T 2 f( x + Zv)Z. This is an unconstrained problem. We may solve (CP =) and identify x = x + Zv, where v is associated with (CP =). Z T f(x) is called the reduced gradient to f in x. Z T 2 f(x)z is called the reduced Hessian to f in x. A. Forsgren, KTH 23 Lecture 8 Convex optimization 2006/2007

24 First-order optimality conditions as a system of equations, cont. The resulting Newton system may equivalently be written as A T 2 f(x) A 0 p (λ + ν) = f(x) (Ax b), alternatively A T 2 f(x) A 0 p λ + = f(x) (Ax b). We prefer the form with λ +, since it can be directly generalized to problems with inequality constraints. A. Forsgren, KTH 24 Lecture 8 Convex optimization 2006/2007

25 Quadratic programming with equality constraints Compare with an equality-constrained quadratic programming problem (EQP ) minimize 1 2 pt Hp + c T p subject to Ap = b, p IR n, where the unique optimal solution p and multiplier vector λ + are given by H AT A 0 p λ + = c b if Z T HZ 0 and A has full row rank, where Z is a matrix whose columns form a basis for null(a)., A. Forsgren, KTH 25 Lecture 8 Convex optimization 2006/2007

26 Newton iteration and equality-constrained quadratic program Compare A T 2 f(x) A 0 p λ + = f(x) (Ax b) with H AT A 0 p λ + = c b. Identify: 2 f(x) H f(x) c A A (Ax b) b. A. Forsgren, KTH 26 Lecture 8 Convex optimization 2006/2007

27 Newton iteration as a QP problem A Newton iteration for solving the first-order necessary optimality conditions to (CP = ) may be viewed as solving the QP problem (QP = ) minimize 1 2 pt 2 f(x)p + f(x) T p subject to Ap = (Ax b), p IR n, and letting x + = x + p, and λ + are given by the multipliers of (QP = ). Problem (QP = ) is well defined with unique optimal solution p and multiplier vector λ + if Z T 2 f(x)z 0 and A has full row rank, where Z is a matrix whose columns form a basis for null(a). A. Forsgren, KTH 27 Lecture 8 Convex optimization 2006/2007

28 An SQP iteration for problems with equality constraints Given x, λ such that Z T 2 f(x)z 0 and A has full row rank, a Newton iteration takes the following form. ˆ Compute optimal solution p and multiplier vector λ + to (QP = ) minimize 1 2 pt 2 f(x)p + f(x) T p subject to Ap = (Ax b), p IR n, ˆ x x + p, λ λ +. We call this method sequential quadratic programming (SQP). NB! (QP = ) is solved by solving a system of linear equations. NB! x and λ have given numerical values in (QP = ). A. Forsgren, KTH 28 Lecture 8 Convex optimization 2006/2007

29 ed of convergence for SQP method for equality-constrained problem Theorem. Assume that f C 3 is convex on R n and that A IR m n has full row rank. Further, assume that x is a minimizer of (CP = ) such that Z T 2 f(x)z 0, where Z is a matrix whose columns form a basis for null(a). If the SQP method (with steplength one) is started at a point sufficiently close to x, λ, then it is well defined and converges to x, λ with convergence rate at least two. Proof. In a neighborhood of x, λ it holds that Z T 2 f(x)z 0 and 2 f(x) A T is nonsingular. The subproblem (QP = ) is hence well A 0 defined and the result follows from the quadratic rate of convergence of Newton s method. A. Forsgren, KTH 29 Lecture 8 Convex optimization 2006/2007

### SF2822 Applied Nonlinear Optimization. Preparatory question. Lecture 9: Sequential quadratic programming. Anders Forsgren

SF2822 Applied Nonlinear Optimization Lecture 9: Sequential quadratic programming Anders Forsgren SF2822 Applied Nonlinear Optimization, KTH / 24 Lecture 9, 207/208 Preparatory question. Try to solve theory

### Nonlinear Optimization: What s important?

Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global

### Nonlinear optimization

Nonlinear optimization Anders Forsgren Optimization and Systems Theory Department of Mathematics Royal Institute of Technology (KTH) Stockholm, Sweden evita Winter School 2009 Geilo, Norway January 11

### Constrained optimization: direct methods (cont.)

Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a

### 4TE3/6TE3. Algorithms for. Continuous Optimization

4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca

### E5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming

E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program

### Optimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30

Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained

### 8 Numerical methods for unconstrained problems

8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields

### Part 2: Linesearch methods for unconstrained optimization. Nick Gould (RAL)

Part 2: Linesearch methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective

### SF2822 Applied nonlinear optimization, final exam Saturday December

SF2822 Applied nonlinear optimization, final exam Saturday December 5 27 8. 3. Examiner: Anders Forsgren, tel. 79 7 27. Allowed tools: Pen/pencil, ruler and rubber; plus a calculator provided by the department.

### Line Search Methods for Unconstrained Optimisation

Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic

### ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints

ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained

### 5 Quasi-Newton Methods

Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min

### Algorithms for constrained local optimization

Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained

### 1 Newton s Method. Suppose we want to solve: x R. At x = x, f (x) can be approximated by:

Newton s Method Suppose we want to solve: (P:) min f (x) At x = x, f (x) can be approximated by: n x R. f (x) h(x) := f ( x)+ f ( x) T (x x)+ (x x) t H ( x)(x x), 2 which is the quadratic Taylor expansion

### 4 Newton Method. Unconstrained Convex Optimization 21. H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion:

Unconstrained Convex Optimization 21 4 Newton Method H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion: f(x + p) f(x)+p T f(x)+ 1 2 pt H(x)p ˆf(p) In general, ˆf(p) won

### Unconstrained minimization of smooth functions

Unconstrained minimization of smooth functions We want to solve min x R N f(x), where f is convex. In this section, we will assume that f is differentiable (so its gradient exists at every point), and

### Numerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen

Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen

### The Steepest Descent Algorithm for Unconstrained Optimization

The Steepest Descent Algorithm for Unconstrained Optimization Robert M. Freund February, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 1 Steepest Descent Algorithm The problem

### Written Examination

Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes

### SF2822 Applied Nonlinear Optimization. Preparatory question. Lecture 10: Interior methods. Anders Forsgren. 1. Try to solve theory question 7.

SF2822 Applied Nonlinear Optimization Lecture 10: Interior methods Anders Forsgren SF2822 Applied Nonlinear Optimization, KTH 1 / 24 Lecture 10, 2017/2018 Preparatory question 1. Try to solve theory question

### Unconstrained optimization

Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout

### 1 Computing with constraints

Notes for 2017-04-26 1 Computing with constraints Recall that our basic problem is minimize φ(x) s.t. x Ω where the feasible set Ω is defined by equality and inequality conditions Ω = {x R n : c i (x)

### Algorithms for Constrained Optimization

1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic

### 5 Handling Constraints

5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest

### Convex Optimization. Newton s method. ENSAE: Optimisation 1/44

Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)

### Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms: A Comparative Study

International Journal of Mathematics And Its Applications Vol.2 No.4 (2014), pp.47-56. ISSN: 2347-1557(online) Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms:

### Scientific Computing: Optimization

Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture

### IE 5531: Engineering Optimization I

IE 5531: Engineering Optimization I Lecture 15: Nonlinear optimization Prof. John Gunnar Carlsson November 1, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I November 1, 2010 1 / 24

### Newton s Method. Javier Peña Convex Optimization /36-725

Newton s Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and

### Optimization Problems with Constraints - introduction to theory, numerical Methods and applications

Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)

### Lecture 15 Newton Method and Self-Concordance. October 23, 2008

Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications

### Optimization Tutorial 1. Basic Gradient Descent

E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.

### Introduction to Nonlinear Optimization Paul J. Atzberger

Introduction to Nonlinear Optimization Paul J. Atzberger Comments should be sent to: atzberg@math.ucsb.edu Introduction We shall discuss in these notes a brief introduction to nonlinear optimization concepts,

### Multidisciplinary System Design Optimization (MSDO)

Multidisciplinary System Design Optimization (MSDO) Numerical Optimization II Lecture 8 Karen Willcox 1 Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Today s Topics Sequential

### Newton s Method. Ryan Tibshirani Convex Optimization /36-725

Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x

### Some Theoretical Properties of an Augmented Lagrangian Merit Function

Some Theoretical Properties of an Augmented Lagrangian Merit Function Philip E. GILL Walter MURRAY Michael A. SAUNDERS Margaret H. WRIGHT Technical Report SOL 86-6R September 1986 Abstract Sequential quadratic

### minimize x subject to (x 2)(x 4) u,

Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for

### Lecture 14: October 17

1-725/36-725: Convex Optimization Fall 218 Lecture 14: October 17 Lecturer: Lecturer: Ryan Tibshirani Scribes: Pengsheng Guo, Xian Zhou Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:

### Optimization Methods. Lecture 19: Line Searches and Newton s Method

15.93 Optimization Methods Lecture 19: Line Searches and Newton s Method 1 Last Lecture Necessary Conditions for Optimality (identifies candidates) x local min f(x ) =, f(x ) PSD Slide 1 Sufficient Conditions

### MATH 4211/6211 Optimization Basics of Optimization Problems

MATH 4211/6211 Optimization Basics of Optimization Problems Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 A standard minimization

### Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization. Nick Gould (RAL)

Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization Nick Gould (RAL) x IR n f(x) subject to c(x) = Part C course on continuoue optimization CONSTRAINED MINIMIZATION x

### 10-725/ Optimization Midterm Exam

10-725/36-725 Optimization Midterm Exam November 6, 2012 NAME: ANDREW ID: Instructions: This exam is 1hr 20mins long Except for a single two-sided sheet of notes, no other material or discussion is permitted

### 6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE. Three Alternatives/Remedies for Gradient Projection

6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE Three Alternatives/Remedies for Gradient Projection Two-Metric Projection Methods Manifold Suboptimization Methods

### Unconstrained Optimization

1 / 36 Unconstrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University February 2, 2015 2 / 36 3 / 36 4 / 36 5 / 36 1. preliminaries 1.1 local approximation

### 10. Unconstrained minimization

Convex Optimization Boyd & Vandenberghe 10. Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method self-concordant functions implementation

### Computational Finance

Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples

### 5 Overview of algorithms for unconstrained optimization

IOE 59: NLP, Winter 22 c Marina A. Epelman 9 5 Overview of algorithms for unconstrained optimization 5. General optimization algorithm Recall: we are attempting to solve the problem (P) min f(x) s.t. x

### Examination paper for TMA4180 Optimization I

Department of Mathematical Sciences Examination paper for TMA4180 Optimization I Academic contact during examination: Phone: Examination date: 26th May 2016 Examination time (from to): 09:00 13:00 Permitted

### Higher-Order Methods

Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth

### min f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;

Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many

### March 8, 2010 MATH 408 FINAL EXAM SAMPLE

March 8, 200 MATH 408 FINAL EXAM SAMPLE EXAM OUTLINE The final exam for this course takes place in the regular course classroom (MEB 238) on Monday, March 2, 8:30-0:20 am. You may bring two-sided 8 page

### On the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method

Optimization Methods and Software Vol. 00, No. 00, Month 200x, 1 11 On the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method ROMAN A. POLYAK Department of SEOR and Mathematical

### Lecture 3. Optimization Problems and Iterative Algorithms

Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex

### Nonlinear Programming

Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week

### Numerical Methods I Solving Nonlinear Equations

Numerical Methods I Solving Nonlinear Equations Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 16th, 2014 A. Donev (Courant Institute)

### Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2

Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Coralia Cartis, University of Oxford INFOMM CDT: Modelling, Analysis and Computation of Continuous Real-World Problems Methods

### REGULARIZED SEQUENTIAL QUADRATIC PROGRAMMING METHODS

REGULARIZED SEQUENTIAL QUADRATIC PROGRAMMING METHODS Philip E. Gill Daniel P. Robinson UCSD Department of Mathematics Technical Report NA-11-02 October 2011 Abstract We present the formulation and analysis

Quadratic Programming Outline Linearly constrained minimization Linear equality constraints Linear inequality constraints Quadratic objective function 2 SideBar: Matrix Spaces Four fundamental subspaces

### Constrained Optimization

1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange

### Arc Search Algorithms

Arc Search Algorithms Nick Henderson and Walter Murray Stanford University Institute for Computational and Mathematical Engineering November 10, 2011 Unconstrained Optimization minimize x D F (x) where

### Numerical Optimization: Basic Concepts and Algorithms

May 27th 2015 Numerical Optimization: Basic Concepts and Algorithms R. Duvigneau R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 1 Outline Some basic concepts in optimization Some

### AM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods

AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality

### Lecture 15: SQP methods for equality constrained optimization

Lecture 15: SQP methods for equality constrained optimization Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lecture 15: SQP methods for equality constrained

### CONVEXIFICATION SCHEMES FOR SQP METHODS

CONVEXIFICAION SCHEMES FOR SQP MEHODS Philip E. Gill Elizabeth Wong UCSD Center for Computational Mathematics echnical Report CCoM-14-06 July 18, 2014 Abstract Sequential quadratic programming (SQP) methods

### 2.3 Linear Programming

2.3 Linear Programming Linear Programming (LP) is the term used to define a wide range of optimization problems in which the objective function is linear in the unknown variables and the constraints are

### Part 4: Active-set methods for linearly constrained optimization. Nick Gould (RAL)

Part 4: Active-set methods for linearly constrained optimization Nick Gould RAL fx subject to Ax b Part C course on continuoue optimization LINEARLY CONSTRAINED MINIMIZATION fx subject to Ax { } b where

### Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted

### Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted

### 10 Numerical methods for constrained problems

10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside

### Numerical optimization

Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal

### On the use of piecewise linear models in nonlinear programming

Math. Program., Ser. A (2013) 137:289 324 DOI 10.1007/s10107-011-0492-9 FULL LENGTH PAPER On the use of piecewise linear models in nonlinear programming Richard H. Byrd Jorge Nocedal Richard A. Waltz Yuchen

### 12. Interior-point methods

12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

### CONSTRAINED NONLINEAR PROGRAMMING

149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach

### AM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods

AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α

### Optimality Conditions for Constrained Optimization

72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)

### Self-Concordant Barrier Functions for Convex Optimization

Appendix F Self-Concordant Barrier Functions for Convex Optimization F.1 Introduction In this Appendix we present a framework for developing polynomial-time algorithms for the solution of convex optimization

### An Inexact Newton Method for Nonlinear Constrained Optimization

An Inexact Newton Method for Nonlinear Constrained Optimization Frank E. Curtis Numerical Analysis Seminar, January 23, 2009 Outline Motivation and background Algorithm development and theoretical results

### Math 273a: Optimization Basic concepts

Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize

### AM 205: lecture 18. Last time: optimization methods Today: conditions for optimality

AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality Existence of Global Minimum For example: f (x, y) = x 2 + y 2 is coercive on R 2 (global min. at (0, 0)) f (x) = x 3

### Numerical Optimization

Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,

### TMA4180 Solutions to recommended exercises in Chapter 3 of N&W

TMA480 Solutions to recommended exercises in Chapter 3 of N&W Exercise 3. The steepest descent and Newtons method with the bactracing algorithm is implemented in rosenbroc_newton.m. With initial point

### Miscellaneous Nonlinear Programming Exercises

Miscellaneous Nonlinear Programming Exercises Henry Wolkowicz 2 08 21 University of Waterloo Department of Combinatorics & Optimization Waterloo, Ontario N2L 3G1, Canada Contents 1 Numerical Analysis Background

### Programming, numerics and optimization

Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428

### MS&E 318 (CME 338) Large-Scale Numerical Optimization

Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods

### Outline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems

Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction

### Lecture 3: Basics of set-constrained and unconstrained optimization

Lecture 3: Basics of set-constrained and unconstrained optimization (Chap 6 from textbook) Xiaoqun Zhang Shanghai Jiao Tong University Last updated: October 9, 2018 Optimization basics Outline Optimization

### Constrained Optimization and Lagrangian Duality

CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

### Lecture 13: Constrained optimization

2010-12-03 Basic ideas A nonlinearly constrained problem must somehow be converted relaxed into a problem which we can solve (a linear/quadratic or unconstrained problem) We solve a sequence of such problems

### Some new facts about sequential quadratic programming methods employing second derivatives

To appear in Optimization Methods and Software Vol. 00, No. 00, Month 20XX, 1 24 Some new facts about sequential quadratic programming methods employing second derivatives A.F. Izmailov a and M.V. Solodov

### LINEAR AND NONLINEAR PROGRAMMING

LINEAR AND NONLINEAR PROGRAMMING Stephen G. Nash and Ariela Sofer George Mason University The McGraw-Hill Companies, Inc. New York St. Louis San Francisco Auckland Bogota Caracas Lisbon London Madrid Mexico

### Optimization. Next: Curve Fitting Up: Numerical Analysis for Chemical Previous: Linear Algebraic and Equations. Subsections

Next: Curve Fitting Up: Numerical Analysis for Chemical Previous: Linear Algebraic and Equations Subsections One-dimensional Unconstrained Optimization Golden-Section Search Quadratic Interpolation Newton's

### A GLOBALLY CONVERGENT STABILIZED SQP METHOD

A GLOBALLY CONVERGENT STABILIZED SQP METHOD Philip E. Gill Daniel P. Robinson July 6, 2013 Abstract Sequential quadratic programming SQP methods are a popular class of methods for nonlinearly constrained

### Part 3: Trust-region methods for unconstrained optimization. Nick Gould (RAL)

Part 3: Trust-region methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective

### Optimization methods

Lecture notes 3 February 8, 016 1 Introduction Optimization methods In these notes we provide an overview of a selection of optimization methods. We focus on methods which rely on first-order information,

### Lectures 9 and 10: Constrained optimization problems and their optimality conditions

Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained

### Conditional Gradient (Frank-Wolfe) Method

Conditional Gradient (Frank-Wolfe) Method Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 1 Outline Today: Conditional gradient method Convergence analysis Properties

### The Conjugate Gradient Method

The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)