AM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods
|
|
- Stewart Nicholson
- 5 years ago
- Views:
Transcription
1 AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods
2 Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α k B 1 k f (x k) where α k is a line search parameter and B k is some approximation to the Hessian Quasi-Newton methods generally lose quadratic convergence of Newton s method, but often superlinear convergence is achieved We now consider some specific quasi-newton methods
3 BFGS The Broyden Fletcher Goldfarb Shanno (BFGS) method is one of the most popular quasi-newton methods: 1: choose initial guess x 0 2: choose B 0, initial Hessian guess, e.g. B 0 = I 3: for k = 0, 1, 2,... do 4: solve B k s k = f (x k ) 5: x k+1 = x k + s k 6: y k = f (x k+1 ) f (x k ) 7: B k+1 = B k + B k 8: end for where B k y ky T k y T k s k B ks k s T k B k s T k B ks k
4 BFGS See lecture: derivation of the Broyden root-finding algorithm See lecture: derivation of the BFGS algorithm Basic idea is that B k accumulates second derivative information on successive iterations, eventually approximates H f well
5 BFGS Actual implementation of BFGS: store and update inverse Hessian to avoid solving linear system: 1: choose initial guess x 0 2: choose H 0, initial inverse Hessian guess, e.g. H 0 = I 3: for k = 0, 1, 2,... do 4: calculate s k = H k f (x k ) 5: x k+1 = x k + s k 6: y k = f (x k+1 ) f (x k ) 7: H k+1 = H k 8: end for where H k (I s k ρ k y t k )H k(i ρ k y k s T k ) + ρ ks k s T k, ρ k = 1 y t k s k
6 BFGS BFGS is implemented as the fmin bfgs function in scipy.optimize Also, BFGS (+ trust region) is implemented in Matlab s fminunc function, e.g. x0 = [5;5]; options = optimset( GradObj, on ); [x,fval,exitflag,output] =... fminunc(@himmelblau_function,x0,options);
7 Conjugate Gradient Method The conjugate gradient (CG) method is another alternative to Newton s method that does not require the Hessian: 1: choose initial guess x 0 2: g 0 = f (x 0 ) 3: x 0 = g 0 4: for k = 0, 1, 2,... do 5: choose η k to minimize f (x k + η k s k ) 6: x k+1 = x k + η k s k 7: g k+1 = f (x k+1 ) 8: β k+1 = (gk+1 T g k+1)/(gk T g k) 9: s k+1 = g k+1 + β k+1 s k 10: end for
8 Constrained Optimization
9 Equality Constrained Optimization We now consider equality constrained minimization: min f (x) subject to g(x) = 0, x Rn where f : R n R and g : R n R m With the Lagrangian L(x, λ) = f (x) + λ T g(x), we recall from that necessary condition for optimality is [ ] f (x) + J T L(x, λ) = g (x)λ = 0 g(x) Once again, this is a nonlinear system of equations that can be solved via Newton s method
10 Sequential Quadratic Programming To derive the Jacobian of this system, we write [ f (x) + m L(x, λ) = k=1 λ k g k (x) g(x) ] R n+m Then we need to differentiate wrt to x R n and λ R m For i = 1,..., n, we have ( L(x, λ)) i = f (x) x i + m k=1 λ k g k (x) x i Differentiating wrt x j, for i, j = 1,..., n, gives x j ( L(x, λ)) i = 2 f (x) x i x j + m k=1 λ k 2 g k (x) x i x j
11 Sequential Quadratic Programming Hence the top-left n n block of the Jacobian of L(x, λ) is B(x, λ) H f (x) + m λ k H gk (x) R n n k=1 Differentiating ( L(x, λ)) i wrt λ j, for i = 1,..., n, j = 1,..., m, gives ( L(x, λ)) i = g j(x) λ j x i Hence the top-right n m block of the Jacobian of L(x, λ) is J g (x) T R n m
12 Sequential Quadratic Programming For i = n + 1,..., n + m, we have ( L(x, λ)) i = g i (x) Differentiating ( L(x, λ)) i wrt x j, for i = n + 1,..., n + m, j = 1,..., n, gives x j ( L(x, λ)) i = g i(x) x j Hence the bottom-left m n block of the Jacobian of L(x, λ) is J g (x) R m n... and the final m m bottom right block is just zero (differentiation of g i (x) w.r.t. λ j )
13 Sequential Quadratic Programming Hence, we have derived the following Jacobian matrix for L(x, λ): [ ] B(x, λ) J T g (x) R (m+n) (m+n) J g (x) 0 Note the 2 2 block structure of this matrix (matrices with this structure are often called KKT matrices 1 ) 1 Karush, Kuhn, Tucker: did seminal work on nonlinear optimization
14 Sequential Quadratic Programming Therefore, Newton s method for L(x, λ) = 0 is: [ B(xk, λ k ) Jg T ] [ ] [ (x k ) sk f (xk ) + Jg = T (x k )λ k J g (x k ) 0 g(x k ) δ k ] for k = 0, 1, 2,... Here (s k, δ k ) R n+m is the k th Newton step
15 Sequential Quadratic Programming Now, consider the constrained minimization problem, where (x k, λ k ) is our Newton iterate at step k: { } 1 min s 2 st B(x k, λ k )s + s T ( f (x k ) + Jg T (x k )λ k ) subject to J g (x k )s + g(x k ) = 0 The objective function is quadratic in s (here x k, λ k are constants) This minimization problem has Lagrangian L k (s, δ) 1 2 st B(x k, λ k )s + s T ( f (x k ) + J T g (x k )λ k ) + δ T (J g (x k )s + g(x k ))
16 Sequential Quadratic Programming Then solving L k (s, δ) = 0 (i.e. first-order necessary conditions) gives a linear system, which is the same as the kth Newton step Hence at each step of Newton s method, we exactly solve a minimization problem (quadratic objective fn., linear constraints) An optimization problem of this type is called a quadratic program This motivates the name for applying Newton s method to L(x, λ) = 0: Sequential Quadratic Programming (SQP)
17 Sequential Quadratic Programming SQP is an important method, and there are many issues to be considered to obtain an efficient and reliable implementation: Efficient solution of the linear systems at each Newton iteration matrix block structure can be exploited Quasi-Newton approximations to the Hessian (as in the unconstrained case) Trust region, line search etc to improve robustness Treatment of constraints (equality and inequality) during the iterative process Selection of good starting guess for λ
18 Penalty Methods Another computational strategy for constrained optimization is to employ penalty methods This converts a constrained problem into an unconstrained problem Key idea: Introduce a new objective function which is a weighted sum of objective function and constraint
19 Penalty Methods Given the minimization problem min x f (x) subject to g(x) = 0 we can consider the related unconstrained problem min x φ ρ (x) = f (x) ρg(x)t g(x) ( ) Let x and x ρ denote the solution of ( ) and ( ), respectively Under appropriate conditions, it can be shown that lim x ρ ρ = x
20 Penalty Methods In practice, we can solve the unconstrained problem for a large value of ρ to get a good approximation of x Another strategy is to solve for a sequence of penalty parameters, ρ k, where x ρ k serves as a starting guess for x ρ k+1 Note that the major drawback of penalty methods is that a large factor ρ will increase the condition number of the Hessian H φρ On the other hand, penalty methods can be convenient, primarily due to their simplicity
21 Linear Programming
22 Linear Programming As we mentioned earlier, the optimization problem min f (x) subject to g(x) = 0 and h(x) 0, ( ) x Rn with f, g, h affine, is called a linear program The feasible region is a convex polyhedron 2 Since the objective function maps out a hyperplane, its global minimum must occur at a vertex of the feasible region 2 Polyhedron: a solid with flat sides, straight edges
23 Linear Programming This can be seen most easily with a picture (in R 2 )
24 Linear Programming The standard approach for solving linear programs is conceptually simple: examine a sequence of the vertices to find the minimum This is called the simplex method Despite its conceptual simplicity, it is non-trivial to develop an efficient implementation of this algorithm We will not discuss the implementation details of the simplex method...
25 Linear Programming In the worst case, the computational work required for the simplex method grows exponentially with the size of the problem But this worst-case behavior is extremely rare; in practice simplex is very efficient (computational work typically grows linearly) Newer methods, called interior point methods, have been developed that are polynomial in the worst case Nevertheless, simplex is still the standard approach since it is more efficient than interior point for most problems
26 Linear Programming Python example: Using cvxopt, solve the linear program subject to and 0 x 1, 0 x 2, 0 x 3 min x f (x) = 5x 1 4x 2 6x 3 x 1 x 2 + x x 1 + 2x 2 + 4x x 1 + 2x 2 30 (LP solvers are efficient, main challenge is to formulate an optimization problem as a linear program in the first place!)
27 PDE Constrained Optimization
28 PDE Constrained Optimization We will now consider optimization based on a function that depends on the solution of a PDE Let us denote a parameter dependent PDE as PDE(u(p); p) = 0 p R n is a parameter vector; could encode, for example, the flow speed and direction in a convection diffusion problem u(p) is the PDE solution for a given p
29 PDE Constrained Optimization We then consider an output functional g, 3 which maps an arbitrary function v to R And we introduce a parameter dependent output, G(p) R, where G(p) g(u(p)) R, which we seek to minimize At the end of the day, this gives a standard optimization problem: min G(p) p R n 3 A functional is just a map from a vector space to R
30 PDE Constrained Optimization One could equivalently write this PDE-based optimization problem as min g(u) subject to PDE(u; p) = 0 p,u For this reason, this type of optimization problem is typically referred to as PDE constrained optimization objective function g depends on u u and p are related by the PDE constraint Based on this formulation, we could introduce Lagrange multipliers and proceed in the usual way for constrained optimization...
31 PDE Constrained Optimization Here we will focus on the form we introduced first: min G(p) p R n Optimization methods usually need some derivative information, such as using finite differences to approximate G(p)
32 PDE Constrained Optimization But using finite differences can be expensive, especially if we have many parameters: G(p) p i G(p + he i) G(p), h hence we need n + 1 evaluations of G to approximate G(p)! We saw from the Himmelblau example that supplying the gradient G(p) cuts down on the number of function evaluations required The extra function calls due to F.D. isn t a big deal for Himmelblau s function, each evaluation is very cheap But in PDE constrained optimization, each p G(p) requires a full PDE solve!
33 PDE Constrained Optimization Hence for PDE constrained optimization with many parameters, it is important to be able to compute the gradient more efficiently There are two main approaches: the direct method the adjoint method The direct method is simpler, but the adjoint method is much more efficient if we have many parameters
34 PDE Output Derivatives Consider the ODE BVP u (x; p) + r(p)u(x; p) = f (x), u(a) = u(b) = 0 which we will refer to as the primal equation Here p R n is the parameter vector, and r : R n R We define an output functional based on an integral g(v) b a σ(x)u(x)dx, for some function σ; then G(p) g(u(p)) R
35 The Direct Method We observe that G(p) b = σ(x) u dx p i a p i hence if we can compute u p i, i = 1, 2,..., n, then we can obtain the gradient Assuming sufficient smoothness, we can differentiate the ODE BVP wrt p i to obtain, for i = 1, 2,..., n u (x; p) + r(p) u (x; p) = r u(x; p) p i p i p i
36 The Direct Method Once we compute each u p i we can then evaluate G(p) by evaluating a sequence of n integrals However, this is not much better than using finite differences: We still need to solve n separate ODE BVPs (Though only the right-hand side changes, so could LU factorize the system matrix once and back/forward sub. for each i)
37 Adjoint-Based Method However, a more efficient approach when n is large is the adjoint method We introduce the adjoint equation: z (x; p) + r(p)z(x; p) = σ(x), z(a) = z(b) = 0
38 Adjoint-Based Method Now, G(p) p i = = = b a b a b a σ(x) u dx p i [ z (x; p) + r(p)z(x; p) ] u dx p i [ z(x; p) u (x; p) + r(p) u ] (x; p) dx, p i p i where the last line follows by integrating by parts twice (boundary terms vanish because u p i and z are zero at a and b) (The adjoint equation is defined based on this integration by parts relationship to the primal equation)
39 Adjoint-Based Method Also, recalling the derivative of the primal problem with respect to p i : u (x; p) + r(p) u (x; p) = r u(x; p), p i p i p i we get G(p) = r b z(x; p)u(x; p)dx p i p i a Therefore, we only need to solve two differential equations (primal and adjoint) to obtain G(p)! For more complicated PDEs the adjoint formulation is more complicated but the basic ideas stay the same
AM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality
More information5 Handling Constraints
5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest
More informationConstrained Optimization
1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationOptimization Problems with Constraints - introduction to theory, numerical Methods and applications
Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)
More informationOutline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems
Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction
More informationQuasi-Newton Methods
Newton s Method Pros and Cons Quasi-Newton Methods MA 348 Kurt Bryan Newton s method has some very nice properties: It s extremely fast, at least once it gets near the minimum, and with the simple modifications
More informationNewton s Method. Ryan Tibshirani Convex Optimization /36-725
Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x
More informationOptimization Methods
Optimization Methods Decision making Examples: determining which ingredients and in what quantities to add to a mixture being made so that it will meet specifications on its composition allocating available
More informationNumerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09
Numerical Optimization 1 Working Horse in Computer Vision Variational Methods Shape Analysis Machine Learning Markov Random Fields Geometry Common denominator: optimization problems 2 Overview of Methods
More information5 Quasi-Newton Methods
Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min
More informationScientific Computing: Optimization
Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture
More informationOptimization and Root Finding. Kurt Hornik
Optimization and Root Finding Kurt Hornik Basics Root finding and unconstrained smooth optimization are closely related: Solving ƒ () = 0 can be accomplished via minimizing ƒ () 2 Slide 2 Basics Root finding
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationLecture V. Numerical Optimization
Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationLecture 18: Optimization Programming
Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More information1 Computing with constraints
Notes for 2017-04-26 1 Computing with constraints Recall that our basic problem is minimize φ(x) s.t. x Ω where the feasible set Ω is defined by equality and inequality conditions Ω = {x R n : c i (x)
More informationNonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More informationMultidisciplinary System Design Optimization (MSDO)
Multidisciplinary System Design Optimization (MSDO) Numerical Optimization II Lecture 8 Karen Willcox 1 Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Today s Topics Sequential
More informationOptimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30
Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints
ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained
More informationCONSTRAINED NONLINEAR PROGRAMMING
149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach
More informationNewton s Method. Javier Peña Convex Optimization /36-725
Newton s Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and
More information2.098/6.255/ Optimization Methods Practice True/False Questions
2.098/6.255/15.093 Optimization Methods Practice True/False Questions December 11, 2009 Part I For each one of the statements below, state whether it is true or false. Include a 1-3 line supporting sentence
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods
More informationLecture 14: October 17
1-725/36-725: Convex Optimization Fall 218 Lecture 14: October 17 Lecturer: Lecturer: Ryan Tibshirani Scribes: Pengsheng Guo, Xian Zhou Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationWhat s New in Active-Set Methods for Nonlinear Optimization?
What s New in Active-Set Methods for Nonlinear Optimization? Philip E. Gill Advances in Numerical Computation, Manchester University, July 5, 2011 A Workshop in Honor of Sven Hammarling UCSD Center for
More informationOptimisation in Higher Dimensions
CHAPTER 6 Optimisation in Higher Dimensions Beyond optimisation in 1D, we will study two directions. First, the equivalent in nth dimension, x R n such that f(x ) f(x) for all x R n. Second, constrained
More informationLinear Programming: Simplex
Linear Programming: Simplex Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Linear Programming: Simplex IMA, August 2016
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca
More informationUnconstrained optimization
Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout
More informationReview for Exam 2 Ben Wang and Mark Styczynski
Review for Exam Ben Wang and Mark Styczynski This is a rough approximation of what we went over in the review session. This is actually more detailed in portions than what we went over. Also, please note
More informationQuasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG. Limited memory BFGS (L-BFGS)
Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno (BFGS) Limited memory BFGS (L-BFGS) February 6, 2014 Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb
More informationPrimal-Dual Interior-Point Methods for Linear Programming based on Newton s Method
Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach
More information2.3 Linear Programming
2.3 Linear Programming Linear Programming (LP) is the term used to define a wide range of optimization problems in which the objective function is linear in the unknown variables and the constraints are
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationDetermination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms: A Comparative Study
International Journal of Mathematics And Its Applications Vol.2 No.4 (2014), pp.47-56. ISSN: 2347-1557(online) Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms:
More informationInterior-Point Methods for Linear Optimization
Interior-Point Methods for Linear Optimization Robert M. Freund and Jorge Vera March, 204 c 204 Robert M. Freund and Jorge Vera. All rights reserved. Linear Optimization with a Logarithmic Barrier Function
More informationLecture 15: SQP methods for equality constrained optimization
Lecture 15: SQP methods for equality constrained optimization Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lecture 15: SQP methods for equality constrained
More informationICS-E4030 Kernel Methods in Machine Learning
ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More informationChapter 4. Unconstrained optimization
Chapter 4. Unconstrained optimization Version: 28-10-2012 Material: (for details see) Chapter 11 in [FKS] (pp.251-276) A reference e.g. L.11.2 refers to the corresponding Lemma in the book [FKS] PDF-file
More informationLecture 13: Constrained optimization
2010-12-03 Basic ideas A nonlinearly constrained problem must somehow be converted relaxed into a problem which we can solve (a linear/quadratic or unconstrained problem) We solve a sequence of such problems
More informationEAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science
EAD 115 Numerical Solution of Engineering and Scientific Problems David M. Rocke Department of Applied Science Multidimensional Unconstrained Optimization Suppose we have a function f() of more than one
More informationLecture 11 and 12: Penalty methods and augmented Lagrangian methods for nonlinear programming
Lecture 11 and 12: Penalty methods and augmented Lagrangian methods for nonlinear programming Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lecture 11 and
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More informationComputational Finance
Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples
More information1.2 Derivation. d p f = d p f(x(p)) = x fd p x (= f x x p ). (1) Second, g x x p + g p = 0. d p f = f x g 1. The expression f x gx
PDE-constrained optimization and the adjoint method Andrew M. Bradley November 16, 21 PDE-constrained optimization and the adjoint method for solving these and related problems appear in a wide range of
More informationChapter 2. Optimization. Gradients, convexity, and ALS
Chapter 2 Optimization Gradients, convexity, and ALS Contents Background Gradient descent Stochastic gradient descent Newton s method Alternating least squares KKT conditions 2 Motivation We can solve
More informationLecture 7 Unconstrained nonlinear programming
Lecture 7 Unconstrained nonlinear programming Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University,
More informationAlgorithms for nonlinear programming problems II
Algorithms for nonlinear programming problems II Martin Branda Charles University Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Computational Aspects of Optimization
More informationConstrained Nonlinear Optimization Algorithms
Department of Industrial Engineering and Management Sciences Northwestern University waechter@iems.northwestern.edu Institute for Mathematics and its Applications University of Minnesota August 4, 2016
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationNumerical Optimization
Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,
More informationConstrained optimization: direct methods (cont.)
Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a
More informationSupport Vector Machines: Maximum Margin Classifiers
Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and Fu-Jie Huang 1 Outline What is behind
More informationPenalty and Barrier Methods General classical constrained minimization problem minimize f(x) subject to g(x) 0 h(x) =0 Penalty methods are motivated by the desire to use unconstrained optimization techniques
More informationLectures 9 and 10: Constrained optimization problems and their optimality conditions
Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained
More informationSF2822 Applied Nonlinear Optimization. Preparatory question. Lecture 9: Sequential quadratic programming. Anders Forsgren
SF2822 Applied Nonlinear Optimization Lecture 9: Sequential quadratic programming Anders Forsgren SF2822 Applied Nonlinear Optimization, KTH / 24 Lecture 9, 207/208 Preparatory question. Try to solve theory
More informationCS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares
CS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares Robert Bridson October 29, 2008 1 Hessian Problems in Newton Last time we fixed one of plain Newton s problems by introducing line search
More informationNonlinear optimization
Nonlinear optimization Anders Forsgren Optimization and Systems Theory Department of Mathematics Royal Institute of Technology (KTH) Stockholm, Sweden evita Winter School 2009 Geilo, Norway January 11
More informationNumerical optimization
Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationCS-E4830 Kernel Methods in Machine Learning
CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationAM 205: lecture 18. Last time: optimization methods Today: conditions for optimality
AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality Existence of Global Minimum For example: f (x, y) = x 2 + y 2 is coercive on R 2 (global min. at (0, 0)) f (x) = x 3
More informationImproving the Convergence of Back-Propogation Learning with Second Order Methods
the of Back-Propogation Learning with Second Order Methods Sue Becker and Yann le Cun, Sept 1988 Kasey Bray, October 2017 Table of Contents 1 with Back-Propagation 2 the of BP 3 A Computationally Feasible
More informationLINEAR AND NONLINEAR PROGRAMMING
LINEAR AND NONLINEAR PROGRAMMING Stephen G. Nash and Ariela Sofer George Mason University The McGraw-Hill Companies, Inc. New York St. Louis San Francisco Auckland Bogota Caracas Lisbon London Madrid Mexico
More informationPart 4: Active-set methods for linearly constrained optimization. Nick Gould (RAL)
Part 4: Active-set methods for linearly constrained optimization Nick Gould RAL fx subject to Ax b Part C course on continuoue optimization LINEARLY CONSTRAINED MINIMIZATION fx subject to Ax { } b where
More informationA Trust-region-based Sequential Quadratic Programming Algorithm
Downloaded from orbit.dtu.dk on: Oct 19, 2018 A Trust-region-based Sequential Quadratic Programming Algorithm Henriksen, Lars Christian; Poulsen, Niels Kjølstad Publication date: 2010 Document Version
More informationarxiv: v1 [math.oc] 10 Apr 2017
A Method to Guarantee Local Convergence for Sequential Quadratic Programming with Poor Hessian Approximation Tuan T. Nguyen, Mircea Lazar and Hans Butler arxiv:1704.03064v1 math.oc] 10 Apr 2017 Abstract
More informationSolution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark
Solution Methods Richard Lusby Department of Management Engineering Technical University of Denmark Lecture Overview (jg Unconstrained Several Variables Quadratic Programming Separable Programming SUMT
More informationNumerical Optimization
Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function
More informationInfeasibility Detection and an Inexact Active-Set Method for Large-Scale Nonlinear Optimization
Infeasibility Detection and an Inexact Active-Set Method for Large-Scale Nonlinear Optimization Frank E. Curtis, Lehigh University involving joint work with James V. Burke, University of Washington Daniel
More informationOn nonlinear optimization since M.J.D. Powell
On nonlinear optimization since 1959 1 M.J.D. Powell Abstract: This view of the development of algorithms for nonlinear optimization is based on the research that has been of particular interest to the
More informationInterior Point Methods. We ll discuss linear programming first, followed by three nonlinear problems. Algorithms for Linear Programming Problems
AMSC 607 / CMSC 764 Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 4: Introduction to Interior Point Methods Dianne P. O Leary c 2008 Interior Point Methods We ll discuss
More informationConstrained optimization. Unconstrained optimization. One-dimensional. Multi-dimensional. Newton with equality constraints. Active-set method.
Optimization Unconstrained optimization One-dimensional Multi-dimensional Newton s method Basic Newton Gauss- Newton Quasi- Newton Descent methods Gradient descent Conjugate gradient Constrained optimization
More informationConstrained optimization
Constrained optimization In general, the formulation of constrained optimization is as follows minj(w), subject to H i (w) = 0, i = 1,..., k. where J is the cost function and H i are the constraints. Lagrange
More information6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE. Three Alternatives/Remedies for Gradient Projection
6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE Three Alternatives/Remedies for Gradient Projection Two-Metric Projection Methods Manifold Suboptimization Methods
More informationApplications of Linear Programming
Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal
More informationLinear programming II
Linear programming II Review: LP problem 1/33 The standard form of LP problem is (primal problem): max z = cx s.t. Ax b, x 0 The corresponding dual problem is: min b T y s.t. A T y c T, y 0 Strong Duality
More informationNumerical Optimization of Partial Differential Equations
Numerical Optimization of Partial Differential Equations Part I: basic optimization concepts in R n Bartosz Protas Department of Mathematics & Statistics McMaster University, Hamilton, Ontario, Canada
More information2. Quasi-Newton methods
L. Vandenberghe EE236C (Spring 2016) 2. Quasi-Newton methods variable metric methods quasi-newton methods BFGS update limited-memory quasi-newton methods 2-1 Newton method for unconstrained minimization
More informationMATH 3795 Lecture 13. Numerical Solution of Nonlinear Equations in R N.
MATH 3795 Lecture 13. Numerical Solution of Nonlinear Equations in R N. Dmitriy Leykekhman Fall 2008 Goals Learn about different methods for the solution of F (x) = 0, their advantages and disadvantages.
More informationCS711008Z Algorithm Design and Analysis
CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief
More informationAn Introduction to Algebraic Multigrid (AMG) Algorithms Derrick Cerwinsky and Craig C. Douglas 1/84
An Introduction to Algebraic Multigrid (AMG) Algorithms Derrick Cerwinsky and Craig C. Douglas 1/84 Introduction Almost all numerical methods for solving PDEs will at some point be reduced to solving A
More informationOPER 627: Nonlinear Optimization Lecture 14: Mid-term Review
OPER 627: Nonlinear Optimization Lecture 14: Mid-term Review Department of Statistical Sciences and Operations Research Virginia Commonwealth University Oct 16, 2013 (Lecture 14) Nonlinear Optimization
More informationSome new facts about sequential quadratic programming methods employing second derivatives
To appear in Optimization Methods and Software Vol. 00, No. 00, Month 20XX, 1 24 Some new facts about sequential quadratic programming methods employing second derivatives A.F. Izmailov a and M.V. Solodov
More informationNumerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems
1 Numerical optimization Alexander & Michael Bronstein, 2006-2009 Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Numerical optimization 048921 Advanced topics in vision Processing and Analysis of
More informationAlgorithms for nonlinear programming problems II
Algorithms for nonlinear programming problems II Martin Branda Charles University in Prague Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Computational Aspects
More informationComputational Optimization. Augmented Lagrangian NW 17.3
Computational Optimization Augmented Lagrangian NW 17.3 Upcoming Schedule No class April 18 Friday, April 25, in class presentations. Projects due unless you present April 25 (free extension until Monday
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Quasi Newton Methods Barnabás Póczos & Ryan Tibshirani Quasi Newton Methods 2 Outline Modified Newton Method Rank one correction of the inverse Rank two correction of the
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More information