Chapter 4. Unconstrained optimization
|
|
- Derick Washington
- 6 years ago
- Views:
Transcription
1 Chapter 4. Unconstrained optimization Version: Material: (for details see) Chapter 11 in [FKS] (pp ) A reference e.g. L.11.2 refers to the corresponding Lemma in the book [FKS] PDF-file of the Book [FKS]: Faigle/Kern/Still, Algorithmic principles of Mathematical Programming. on: stillgj/priv/ CO, Chapter 4 p 1/23
2 4.1 Introduction We consider the nonlinear minimization problem (with f C 1 or f C 2 ): (P) min f (x), x R n Recall: Usually in (nonconvex) unconstrained optimization we try to find a local minimizer. Global minimization is much more difficult. Theoretical method: (based on optimality conditions) Find a point x satisfying f (x) = 0 (critical point) Check whether 2 f (x) 0. CO, Chapter 4 p 2/23
3 CONCEPTUAL ALGORITHM: Choose x 0 R n. Iterate step k: Given x k R n, find a new point x k+1 with f (x k+1 ) < f (x k ). We hope that: x k x with x a local mininimizer. Def. Let x k x for k. The sequence (x k ) has a: linear convergence if with a constant 0 C < 1 and some K N: x k+1 x C x k x, k K. C is the convergence factor. quadratic convergence if with a constant c 0, x k+1 x c x k x 2, k N. superlinear convergence if lim k x k+1 x x k x = 0. CO, Chapter 4 p 3/23
4 4.2 General descent method (f C 1 ) Def. A vector d k R n is called a descent direction for f in x k if f (x k ) T d k < 0 ( ) Rem. If ( ) holds then: f (x k + td k ) < f (x k ) for t > 0 small. Abbreviation: g(x) := f (x), g k := g(x k ) Conceptual DESCENT METHOD: Choose a starting point x 0 R n and ɛ > 0. Iterate step k: Given x k R n, proceed as follows: if g(x k ) < ɛ, stop with x x k. Choose a descent direction d k in x k : g T k d k < 0 Find a solution t k of the (one-dimensional) minimization problem min t 0 f (x k + td k ) and put x k+1 = x k + t k d k. ( ) CO, Chapter 4 p 4/23
5 Remark By this descent method, minimization in R n is reduced to (line) minimization in R (in each step k). Steepest descent method: Use in the descent method as descent direction (see Ex.11.7): d k = f (x k ) Ex.11.7 Assuming f (x k ) 0, show that d k = [ f (x k )]/ f (x k ) solves the problem: min f (x d R n k ) T d s.t. d = 1 Convergence behavior: L.11.3 In the line-minimization step (**) we have f (x k+1 ) T d k = 0 For the steepest descent method this means: d T k+1 d k = 0 (ziggzagging) CO, Chapter 4 p 5/23
6 Th.11.1 Let f C 1. Apply the steepest descent method. If the iterates x k converge, i.e., x k x, then f (x) = 0 Ex.11.8 Given the quadratic function on R n, q(x) = x T Ax + b T x, A 0. Show that the minimizer t k of min q(x k + td k ) is given by t k = gt k d k. t 0 2dk T Ad k CO, Chapter 4 p 6/23
7 Speed of concergence: The next example shows that in general (even for min of quadratic functions), the steepest descent method cannot be expected to converge better than linearly. Ex Apply the steepest descent method to q(x) = x T ( r ) x, r 1 Then with x 0 = (r, 1) it follows ( ) r 1 k x k = (r, ( 1) k ). r + 1 (Linear convergence to x = 0 with factor C = (r 1)/(r + 1).) HINT: Make use of [FKS,L.11.8] and apply induction wrt. k. CO, Chapter 4 p 7/23
8 4.3 Method of conjugate directions Aim: Find an algorithm which (at least for quadratic functions) has better convergence than steepest descent Case: f (x) = q(x) := 1 2 x T Ax + b T x, A 0 (pd.) Idea. Try to generate d k s such that (not only q(x k+1 ) T d k = 0 but) q(x k+1 ) T d j = 0 0 j k Then, after n steps we have q(x n ) T d j = 0 0 j n 1 and (if the d j s are lin. indep.) q(x n ) = 0. So x n = A 1 b is the minimizer of q. CO, Chapter 4 p 8/23
9 L Apply the descent method to q(x). The following are equivalent: (i) g T j+1 d i = 0 for all 0 i j k; (ii) d T j Ad i = 0 for all 0 i < j k. Definition. Vectors d 0,..., d n 1 0 are called A-conjugate (or A-orthogonal) if: dj T Ad i = 0 i j. Ex. A collection of A-conjugate vectors d 0,..., d n 1 0 in R n are linearly independent. CO, Chapter 4 p 9/23
10 Construction of A-conjugate d k s. To construct vectors satisfying the conditions in L.11.4, simply try: d k = g k + α k d k 1 Then d T k Ad k 1 = 0 implies α k = gt k Ad k 1 d T k 1 Ad k 1. Th.11.3 Apply the descent method to q(x) with d k = g k + α k d k 1, α k = gt k Ad k 1 d T k 1 Ad k 1 Then the d k s are A-conjugate. In particular, the algorithm stops after (at most) n steps with the unique minimizer x = A 1 b of q. CO, Chapter 4 p 10/23
11 Conjugate Gradient Method (CG) INIT: Choose x 0 R n, ε > 0, d 0 := g 0 ; ITER: WHILE g k ε DO BEGIN Determine a solution t k for the problem END ( ) min f (x k + td k ) t 0 Set x k+1 = x k + t k d k. Set d k+1 = g k+1 + α k+1 d k. Ex Under the assumptions of Th.11.3, show that the iteration point x k+1 is the (global) minimizer of the quadratic function q on the affine subspace S k = {x 0 + γ 0 d γ k d k γ 0,.., γ k R} CO, Chapter 4 p 11/23
12 4.3.2 Case: non-quadratic functions f (x) Note that for quadratic function f = q we have: α k+1 = gt k+1 Ad k d T k Ad k = gt k+1 (g k+1 g k ) d T k (g k+1 g k ) = gt k+1 (g k+1 g k ) = g k+1 2 g k 2 g k 2 So, for non-quadratic f (x) in the CG-algorithm we can use d k+1 = g k+1 + α k+1 d k with: Hestenes-Stiefel (1952): α k+1 = gt k+1 (g k+1 g k ) d T k (g k+1 g k ) Fletcher-Reeves (1964): α k+1 = g k+1 2 g k 2 Polak-Ribiere (1969): α k+1 = gt k+1 (g k+1 g k ) g k 2 CO, Chapter 4 p 12/23
13 Application to sparse systems Ax = b, A 0 Def. A = (a ij ) is sparse if less than α% of the a ij -s are 0 with (say) α 5 CG-method: apply the CG-method to min 1 2 x T Ax b T x with solution x = A 1 b CO, Chapter 4 p 13/23
14 CG Method for sparse linear systems Ax = b, A 0 INIT: Choose x 0 R n and ε > 0 and set d 0 = g 0 ; ITER: WHILE g k ε DO BEGIN Set x k+1 = x k + t k d k Set g k+1 = g k + t k Ad k with t k = gt k d k d T k Ad k Set d k+1 = g k+1 + α k+1 d k with α k+1 = gt k+1 g k+1 gk T g k END Rem. Complexity: α 100 n2 flop s (floating point operations) per ITER. CO, Chapter 4 p 14/23
15 4.4 Line minimization In the general descent method (see Ch.4.2) we have to repeatedly solve: where h (0) < 0. min t 0 h(t) with h(t) = f (x k + td k ) This can be done by: exact line minimization of numerical analysis e.g., bisection, golden section, Newton-, secant method (see Ch.4.3, Ch ) or more efficiently by inexact line search Goldstein-, Goldstein-Wolfe test (see Ch ) CO, Chapter 4 p 15/23
16 4.5 Newton s method: General remark: Newton s method for solving systems of nonlinear equations is one of the most important tools of applied mathematics. Newtons s Iteration: for solving F (x) = 0, F : R n R n, F C 1, a system of n equations in n unknowns x = (x 1,..., x n ): start with some x 0 and iterate Th.11.4 x k+1 = x k [ F (x k )] 1 F (x k ) k = 0, 1,... (local convergence of Newton s method) Given F : R n R n, F C 2 such that F (x) = 0 and F (x) is non-singular. Then the Newton iteration x k converges quadratically to x for any x 0 sufficiently close to x. CO, Chapter 4 p 16/23
17 Newton for solving : min f (x) or F (x) := f (x) = 0, x k+1 = x k [ 2 f (x k )] 1 f (x k ) (local) quadratic convergence to x if: f C 3, f (x) = 0 with 2 f (x) non-singular. Problems with this Newton method for: x k x possibly a local maximizer. x k x k+1 with increasing f Newton descent method: min f (x) The Newton direction d k = [ 2 f (x k )] 1 f (x k ) is a descent direction at x k (g T k d k < 0) if (assume f (x k ) 0): [ 2 f (x k )] 1 or equivalently 2 f (x k ) is positive definite. CO, Chapter 4 p 17/23
18 Algorithm: (Levenberg-Marquardt variant) step k: Given x k R n with g k determine σ k > 0 such that ( 2 f (x k ) + σ k I) 0, compute d k = ( 2 f (x k ) + σ k I ) 1 gk ( ) 2. Find a minimizer t k of min t 0 f (x k + td k ) and put x k+1 = x k + t k d k. Ex.11.n1 [connection with the trust region method ] Consider the quadratic Taylor approximation of f near x k : q(x) := f (x k ) + f (x k ) T (x x k ) +1/2(x x k ) T 2 f (x k )(x x k ) Compute the descent step d k according to ( ) (Levenberg-Marquardt) and put x k+1 = x k + d k, τ := d k. Show that x k+1 is a local minimizer of the trust region problem: min q(x) s.t. x x k τ CO, Chapter 4 p 18/23
19 Disadvantage of the Newton methods: 2 f (x k ) needed work per step: linear system F k x = b k n 3 flop s CO, Chapter 4 p 19/23
20 4.6 Quasi-Newton method. Find a method which only makes use of first derivatives and only needs O(n 2 ) flop s per iter. Consider the descent method with: desired properties for H k : i H k 0 d k = H k g k ii H k+1 = H k + E k simple update rule iii for quadratic f conjugate directions d j iv the Quasi-Newton condition: (x k+1 x k ) = H k+1 (g k+1 g k ) Notation: δ k := (x k+1 x k ), γ k := (g k+1 g k ) CO, Chapter 4 p 20/23
21 Quasi-Newton Method INIT: Choose some x 0 R n, H 0 0, ε > 0 ITER: WHILE g k ε DO BEGIN Set d k = H k g k, Determine a solution t k for the problem END min f (x k + td k ) t 0 Set x k+1 = x k + t k d k H k+1 = H k + E k. and update For the update H k + E k we try: with α, β, µ R E k = αuu T + βvv T + µ(uv T + vu T ) ( ) where u := δ k, v := H k γ k Note that E k is symmetric with rank 2. CO, Chapter 4 p 21/23
22 L.11.5 Apply the Quasi-Newton method to q(x) = 1 2 x T Ax + b T x, A 0 with E k of the form ( ) and H k+1 satisfying iv: δ k = H k+1 γ k Then the directions d j are A-conjugate : d T j Ad i = 0 0 i < j k Last step in the construction of E k : Find α, β, µ in ( ) such that (iv) holds. This leads to the following update formula CO, Chapter 4 p 22/23
23 Broyden family: with Φ R H k+1 = H k + δ k δ T k δ T k γ k where w := ( δk δ T k γ k As special cases we obtain: H k γ k γ T k H k γ T k H k γ k + Φ ww T ( ) ) H k γ k (γ T γk T H k γ k k H k γ k ) 1 2. Φ = 0, the DFP-method (1963) (Davidon, Fletcher, Powell) Φ = 1, the BFGS-method (1970) (Broyden, Fletcher, Goldfarb, Shanno) Finally we show that property i), H k 0, is preserved. L.11.6 In the Quasi-Newton method, if we use ( ) with Φ 0, then: H k 0 H k+1 0 CO, Chapter 4 p 23/23
Nonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationOptimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23
Optimization: Nonlinear Optimization without Constraints Nonlinear Optimization without Constraints 1 / 23 Nonlinear optimization without constraints Unconstrained minimization min x f(x) where f(x) is
More information5 Quasi-Newton Methods
Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min
More informationOptimization II: Unconstrained Multivariable
Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 1
More informationQuasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG. Limited memory BFGS (L-BFGS)
Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno (BFGS) Limited memory BFGS (L-BFGS) February 6, 2014 Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb
More informationFALL 2018 MATH 4211/6211 Optimization Homework 4
FALL 2018 MATH 4211/6211 Optimization Homework 4 This homework assignment is open to textbook, reference books, slides, and online resources, excluding any direct solution to the problem (such as solution
More informationUnconstrained optimization
Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout
More informationLecture 7 Unconstrained nonlinear programming
Lecture 7 Unconstrained nonlinear programming Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University,
More informationNumerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09
Numerical Optimization 1 Working Horse in Computer Vision Variational Methods Shape Analysis Machine Learning Markov Random Fields Geometry Common denominator: optimization problems 2 Overview of Methods
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Quasi Newton Methods Barnabás Póczos & Ryan Tibshirani Quasi Newton Methods 2 Outline Modified Newton Method Rank one correction of the inverse Rank two correction of the
More informationMethods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent
Nonlinear Optimization Steepest Descent and Niclas Börlin Department of Computing Science Umeå University niclas.borlin@cs.umu.se A disadvantage with the Newton method is that the Hessian has to be derived
More informationJanuary 29, Non-linear conjugate gradient method(s): Fletcher Reeves Polak Ribière January 29, 2014 Hestenes Stiefel 1 / 13
Non-linear conjugate gradient method(s): Fletcher Reeves Polak Ribière Hestenes Stiefel January 29, 2014 Non-linear conjugate gradient method(s): Fletcher Reeves Polak Ribière January 29, 2014 Hestenes
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 3. Gradient Method
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 3 Gradient Method Shiqian Ma, MAT-258A: Numerical Optimization 2 3.1. Gradient method Classical gradient method: to minimize a differentiable convex
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More information1 Numerical optimization
Contents 1 Numerical optimization 5 1.1 Optimization of single-variable functions............ 5 1.1.1 Golden Section Search................... 6 1.1. Fibonacci Search...................... 8 1. Algorithms
More informationChapter 10 Conjugate Direction Methods
Chapter 10 Conjugate Direction Methods An Introduction to Optimization Spring, 2012 1 Wei-Ta Chu 2012/4/13 Introduction Conjugate direction methods can be viewed as being intermediate between the method
More informationQuasi-Newton methods for minimization
Quasi-Newton methods for minimization Lectures for PHD course on Numerical optimization Enrico Bertolazzi DIMS Universitá di Trento November 21 December 14, 2011 Quasi-Newton methods for minimization 1
More informationStatistics 580 Optimization Methods
Statistics 580 Optimization Methods Introduction Let fx be a given real-valued function on R p. The general optimization problem is to find an x ɛ R p at which fx attain a maximum or a minimum. It is of
More informationOptimization II: Unconstrained Multivariable
Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Optimization II: Unconstrained
More information2. Quasi-Newton methods
L. Vandenberghe EE236C (Spring 2016) 2. Quasi-Newton methods variable metric methods quasi-newton methods BFGS update limited-memory quasi-newton methods 2-1 Newton method for unconstrained minimization
More information1 Numerical optimization
Contents Numerical optimization 5. Optimization of single-variable functions.............................. 5.. Golden Section Search..................................... 6.. Fibonacci Search........................................
More informationNumerical solutions of nonlinear systems of equations
Numerical solutions of nonlinear systems of equations Tsung-Ming Huang Department of Mathematics National Taiwan Normal University, Taiwan E-mail: min@math.ntnu.edu.tw August 28, 2011 Outline 1 Fixed points
More information8 Numerical methods for unconstrained problems
8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields
More informationMATH 4211/6211 Optimization Quasi-Newton Method
MATH 4211/6211 Optimization Quasi-Newton Method Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 Quasi-Newton Method Motivation:
More informationQuasi-Newton Methods
Newton s Method Pros and Cons Quasi-Newton Methods MA 348 Kurt Bryan Newton s method has some very nice properties: It s extremely fast, at least once it gets near the minimum, and with the simple modifications
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationNewton s Method. Ryan Tibshirani Convex Optimization /36-725
Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x
More informationThe Conjugate Gradient Algorithm
Optimization over a Subspace Conjugate Direction Methods Conjugate Gradient Algorithm Non-Quadratic Conjugate Gradient Algorithm Optimization over a Subspace Consider the problem min f (x) subject to x
More informationGradient-Based Optimization
Multidisciplinary Design Optimization 48 Chapter 3 Gradient-Based Optimization 3. Introduction In Chapter we described methods to minimize (or at least decrease) a function of one variable. While problems
More informationEECS260 Optimization Lecture notes
EECS260 Optimization Lecture notes Based on Numerical Optimization (Nocedal & Wright, Springer, 2nd ed., 2006) Miguel Á. Carreira-Perpiñán EECS, University of California, Merced May 2, 2010 1 Introduction
More informationOptimization and Root Finding. Kurt Hornik
Optimization and Root Finding Kurt Hornik Basics Root finding and unconstrained smooth optimization are closely related: Solving ƒ () = 0 can be accomplished via minimizing ƒ () 2 Slide 2 Basics Root finding
More informationMethods for Unconstrained Optimization Numerical Optimization Lectures 1-2
Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Coralia Cartis, University of Oxford INFOMM CDT: Modelling, Analysis and Computation of Continuous Real-World Problems Methods
More informationLecture 14: October 17
1-725/36-725: Convex Optimization Fall 218 Lecture 14: October 17 Lecturer: Lecturer: Ryan Tibshirani Scribes: Pengsheng Guo, Xian Zhou Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationORIE 6326: Convex Optimization. Quasi-Newton Methods
ORIE 6326: Convex Optimization Quasi-Newton Methods Professor Udell Operations Research and Information Engineering Cornell April 10, 2017 Slides on steepest descent and analysis of Newton s method adapted
More informationHomework 3 Conjugate Gradient Descent, Accelerated Gradient Descent Newton, Quasi Newton and Projected Gradient Descent
Homework 3 Conjugate Gradient Descent, Accelerated Gradient Descent Newton, Quasi Newton and Projected Gradient Descent CMU 10-725/36-725: Convex Optimization (Fall 2017) OUT: Sep 29 DUE: Oct 13, 5:00
More informationGlobal Convergence of Perry-Shanno Memoryless Quasi-Newton-type Method. 1 Introduction
ISSN 1749-3889 (print), 1749-3897 (online) International Journal of Nonlinear Science Vol.11(2011) No.2,pp.153-158 Global Convergence of Perry-Shanno Memoryless Quasi-Newton-type Method Yigui Ou, Jun Zhang
More informationStatic unconstrained optimization
Static unconstrained optimization 2 In unconstrained optimization an objective function is minimized without any additional restriction on the decision variables, i.e. min f(x) x X ad (2.) with X ad R
More informationComparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming Problems
International Journal of Scientific and Research Publications, Volume 3, Issue 10, October 013 1 ISSN 50-3153 Comparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming
More information(One Dimension) Problem: for a function f(x), find x 0 such that f(x 0 ) = 0. f(x)
Solving Nonlinear Equations & Optimization One Dimension Problem: or a unction, ind 0 such that 0 = 0. 0 One Root: The Bisection Method This one s guaranteed to converge at least to a singularity, i not
More informationOPER 627: Nonlinear Optimization Lecture 14: Mid-term Review
OPER 627: Nonlinear Optimization Lecture 14: Mid-term Review Department of Statistical Sciences and Operations Research Virginia Commonwealth University Oct 16, 2013 (Lecture 14) Nonlinear Optimization
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationNumerical Optimization of Partial Differential Equations
Numerical Optimization of Partial Differential Equations Part I: basic optimization concepts in R n Bartosz Protas Department of Mathematics & Statistics McMaster University, Hamilton, Ontario, Canada
More informationPart 4: IIR Filters Optimization Approach. Tutorial ISCAS 2007
Part 4: IIR Filters Optimization Approach Tutorial ISCAS 2007 Copyright 2007 Andreas Antoniou Victoria, BC, Canada Email: aantoniou@ieee.org July 24, 2007 Frame # 1 Slide # 1 A. Antoniou Part4: IIR Filters
More informationHigher-Order Methods
Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth
More informationNotes on Numerical Optimization
Notes on Numerical Optimization University of Chicago, 2014 Viva Patel October 18, 2014 1 Contents Contents 2 List of Algorithms 4 I Fundamentals of Optimization 5 1 Overview of Numerical Optimization
More informationA globally and R-linearly convergent hybrid HS and PRP method and its inexact version with applications
A globally and R-linearly convergent hybrid HS and PRP method and its inexact version with applications Weijun Zhou 28 October 20 Abstract A hybrid HS and PRP type conjugate gradient method for smooth
More informationSECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS
SECTION: CONTINUOUS OPTIMISATION LECTURE 4: QUASI-NEWTON METHODS HONOUR SCHOOL OF MATHEMATICS, OXFORD UNIVERSITY HILARY TERM 2005, DR RAPHAEL HAUSER 1. The Quasi-Newton Idea. In this lecture we will discuss
More informationReview of Classical Optimization
Part II Review of Classical Optimization Multidisciplinary Design Optimization of Aircrafts 51 2 Deterministic Methods 2.1 One-Dimensional Unconstrained Minimization 2.1.1 Motivation Most practical optimization
More informationNew hybrid conjugate gradient methods with the generalized Wolfe line search
Xu and Kong SpringerPlus (016)5:881 DOI 10.1186/s40064-016-5-9 METHODOLOGY New hybrid conjugate gradient methods with the generalized Wolfe line search Open Access Xiao Xu * and Fan yu Kong *Correspondence:
More informationLecture 7: Optimization methods for non linear estimation or function estimation
Lecture 7: Optimization methods for non linear estimation or function estimation Y. Favennec 1, P. Le Masson 2 and Y. Jarny 1 1 LTN UMR CNRS 6607 Polytetch Nantes 44306 Nantes France 2 LIMATB Université
More informationOptimization 2. CS5240 Theoretical Foundations in Multimedia. Leow Wee Kheng
Optimization 2 CS5240 Theoretical Foundations in Multimedia Leow Wee Kheng Department of Computer Science School of Computing National University of Singapore Leow Wee Kheng (NUS) Optimization 2 1 / 38
More informationAM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationLecture 18: November Review on Primal-dual interior-poit methods
10-725/36-725: Convex Optimization Fall 2016 Lecturer: Lecturer: Javier Pena Lecture 18: November 2 Scribes: Scribes: Yizhu Lin, Pan Liu Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationISyE 6663 (Winter 2017) Nonlinear Optimization
Winter 017 TABLE OF CONTENTS ISyE 6663 (Winter 017) Nonlinear Optimization Prof. R. D. C. Monteiro Georgia Institute of Technology L A TEXer: W. KONG http://wwkong.github.io Last Revision: May 3, 017 Table
More informationAM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α
More informationSearch Directions for Unconstrained Optimization
8 CHAPTER 8 Search Directions for Unconstrained Optimization In this chapter we study the choice of search directions used in our basic updating scheme x +1 = x + t d. for solving P min f(x). x R n All
More informationMultivariate Newton Minimanization
Multivariate Newton Minimanization Optymalizacja syntezy biosurfaktantu Rhamnolipid Rhamnolipids are naturally occuring glycolipid produced commercially by the Pseudomonas aeruginosa species of bacteria.
More informationQuasi-Newton Methods
Quasi-Newton Methods Werner C. Rheinboldt These are excerpts of material relating to the boos [OR00 and [Rhe98 and of write-ups prepared for courses held at the University of Pittsburgh. Some further references
More informationOpen Problems in Nonlinear Conjugate Gradient Algorithms for Unconstrained Optimization
BULLETIN of the Malaysian Mathematical Sciences Society http://math.usm.my/bulletin Bull. Malays. Math. Sci. Soc. (2) 34(2) (2011), 319 330 Open Problems in Nonlinear Conjugate Gradient Algorithms for
More informationMA/OR/ST 706: Nonlinear Programming Midterm Exam Instructor: Dr. Kartik Sivaramakrishnan INSTRUCTIONS
MA/OR/ST 706: Nonlinear Programming Midterm Exam Instructor: Dr. Kartik Sivaramakrishnan INSTRUCTIONS 1. Please write your name and student number clearly on the front page of the exam. 2. The exam is
More informationUniversity of Maryland at College Park. limited amount of computer memory, thereby allowing problems with a very large number
Limited-Memory Matrix Methods with Applications 1 Tamara Gibson Kolda 2 Applied Mathematics Program University of Maryland at College Park Abstract. The focus of this dissertation is on matrix decompositions
More informationEAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science
EAD 115 Numerical Solution of Engineering and Scientific Problems David M. Rocke Department of Applied Science Multidimensional Unconstrained Optimization Suppose we have a function f() of more than one
More informationOptimization Methods
Optimization Methods Decision making Examples: determining which ingredients and in what quantities to add to a mixture being made so that it will meet specifications on its composition allocating available
More informationLecture V. Numerical Optimization
Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize
More informationConditional Gradient (Frank-Wolfe) Method
Conditional Gradient (Frank-Wolfe) Method Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 1 Outline Today: Conditional gradient method Convergence analysis Properties
More informationON THE CONNECTION BETWEEN THE CONJUGATE GRADIENT METHOD AND QUASI-NEWTON METHODS ON QUADRATIC PROBLEMS
ON THE CONNECTION BETWEEN THE CONJUGATE GRADIENT METHOD AND QUASI-NEWTON METHODS ON QUADRATIC PROBLEMS Anders FORSGREN Tove ODLAND Technical Report TRITA-MAT-203-OS-03 Department of Mathematics KTH Royal
More informationNonlinearOptimization
1/35 NonlinearOptimization Pavel Kordík Department of Computer Systems Faculty of Information Technology Czech Technical University in Prague Jiří Kašpar, Pavel Tvrdík, 2011 Unconstrained nonlinear optimization,
More informationOn nonlinear optimization since M.J.D. Powell
On nonlinear optimization since 1959 1 M.J.D. Powell Abstract: This view of the development of algorithms for nonlinear optimization is based on the research that has been of particular interest to the
More informationPart 2: Linesearch methods for unconstrained optimization. Nick Gould (RAL)
Part 2: Linesearch methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective
More informationUnconstrained Multivariate Optimization
Unconstrained Multivariate Optimization Multivariate optimization means optimization of a scalar function of a several variables: and has the general form: y = () min ( ) where () is a nonlinear scalar-valued
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationSteepest descent algorithm. Conjugate gradient training algorithm. Steepest descent algorithm. Remember previous examples
Conjugate gradient training algorithm Steepest descent algorithm So far: Heuristic improvements to gradient descent (momentum Steepest descent training algorithm Can we do better? Definitions: weight vector
More informationConjugate Directions for Stochastic Gradient Descent
Conjugate Directions for Stochastic Gradient Descent Nicol N Schraudolph Thore Graepel Institute of Computational Science ETH Zürich, Switzerland {schraudo,graepel}@infethzch Abstract The method of conjugate
More informationOptimization Methods for Machine Learning
Optimization Methods for Machine Learning Sathiya Keerthi Microsoft Talks given at UC Santa Cruz February 21-23, 2017 The slides for the talks will be made available at: http://www.keerthis.com/ Introduction
More informationECS550NFB Introduction to Numerical Methods using Matlab Day 2
ECS550NFB Introduction to Numerical Methods using Matlab Day 2 Lukas Laffers lukas.laffers@umb.sk Department of Mathematics, University of Matej Bel June 9, 2015 Today Root-finding: find x that solves
More informationMath 408A: Non-Linear Optimization
February 12 Broyden Updates Given g : R n R n solve g(x) = 0. Algorithm: Broyden s Method Initialization: x 0 R n, B 0 R n n Having (x k, B k ) compute (x k+1, B x+1 ) as follows: Solve B k s k = g(x
More informationQuasi-Newton Methods. Zico Kolter (notes by Ryan Tibshirani, Javier Peña, Zico Kolter) Convex Optimization
Quasi-Newton Methods Zico Kolter (notes by Ryan Tibshirani, Javier Peña, Zico Kolter) Convex Optimization 10-725 Last time: primal-dual interior-point methods Given the problem min x f(x) subject to h(x)
More informationModification of the Armijo line search to satisfy the convergence properties of HS method
Université de Sfax Faculté des Sciences de Sfax Département de Mathématiques BP. 1171 Rte. Soukra 3000 Sfax Tunisia INTERNATIONAL CONFERENCE ON ADVANCES IN APPLIED MATHEMATICS 2014 Modification of the
More informationStep-size Estimation for Unconstrained Optimization Methods
Volume 24, N. 3, pp. 399 416, 2005 Copyright 2005 SBMAC ISSN 0101-8205 www.scielo.br/cam Step-size Estimation for Unconstrained Optimization Methods ZHEN-JUN SHI 1,2 and JIE SHEN 3 1 College of Operations
More informationNumerical Optimization
Numerical Optimization Emo Todorov Applied Mathematics and Computer Science & Engineering University of Washington Spring 2010 Emo Todorov (UW) AMATH/CSE 579, Spring 2010 Lecture 9 1 / 8 Gradient descent
More informationOptimization Methods for Circuit Design
Technische Universität München Department of Electrical Engineering and Information Technology Institute for Electronic Design Automation Optimization Methods for Circuit Design Compendium H. Graeb Version
More informationGeometry optimization
Geometry optimization Trygve Helgaker Centre for Theoretical and Computational Chemistry Department of Chemistry, University of Oslo, Norway European Summer School in Quantum Chemistry (ESQC) 211 Torre
More informationOutline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems
Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction
More informationGlobal convergence of a regularized factorized quasi-newton method for nonlinear least squares problems
Volume 29, N. 2, pp. 195 214, 2010 Copyright 2010 SBMAC ISSN 0101-8205 www.scielo.br/cam Global convergence of a regularized factorized quasi-newton method for nonlinear least squares problems WEIJUN ZHOU
More informationConjugate gradient algorithm for training neural networks
. Introduction Recall that in the steepest-descent neural network training algorithm, consecutive line-search directions are orthogonal, such that, where, gwt [ ( + ) ] denotes E[ w( t + ) ], the gradient
More informationGlobal Convergence Properties of the HS Conjugate Gradient Method
Applied Mathematical Sciences, Vol. 7, 2013, no. 142, 7077-7091 HIKARI Ltd, www.m-hikari.com http://dx.doi.org/10.12988/ams.2013.311638 Global Convergence Properties of the HS Conjugate Gradient Method
More informationQuasi-Newton Methods. Javier Peña Convex Optimization /36-725
Quasi-Newton Methods Javier Peña Convex Optimization 10-725/36-725 Last time: primal-dual interior-point methods Consider the problem min x subject to f(x) Ax = b h(x) 0 Assume f, h 1,..., h m are convex
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)
More information1 Newton s Method. Suppose we want to solve: x R. At x = x, f (x) can be approximated by:
Newton s Method Suppose we want to solve: (P:) min f (x) At x = x, f (x) can be approximated by: n x R. f (x) h(x) := f ( x)+ f ( x) T (x x)+ (x x) t H ( x)(x x), 2 which is the quadratic Taylor expansion
More information14. Nonlinear equations
L. Vandenberghe ECE133A (Winter 2018) 14. Nonlinear equations Newton method for nonlinear equations damped Newton method for unconstrained minimization Newton method for nonlinear least squares 14-1 Set
More informationPart 3: Trust-region methods for unconstrained optimization. Nick Gould (RAL)
Part 3: Trust-region methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective
More informationNumerical Optimization
Unconstrained Optimization (II) Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Unconstrained Optimization Let f : R R Unconstrained problem min x
More informationSeminal papers in nonlinear optimization
Seminal papers in nonlinear optimization Nick Gould, CSED, RAL, Chilton, OX11 0QX, England (n.gould@rl.ac.uk) December 7, 2006 The following papers are classics in the field. Although many of them cover
More informationContents. Preface. 1 Introduction Optimization view on mathematical models NLP models, black-box versus explicit expression 3
Contents Preface ix 1 Introduction 1 1.1 Optimization view on mathematical models 1 1.2 NLP models, black-box versus explicit expression 3 2 Mathematical modeling, cases 7 2.1 Introduction 7 2.2 Enclosing
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More informationSolutions and Notes to Selected Problems In: Numerical Optimzation by Jorge Nocedal and Stephen J. Wright.
Solutions and Notes to Selected Problems In: Numerical Optimzation by Jorge Nocedal and Stephen J. Wright. John L. Weatherwax July 7, 2010 wax@alum.mit.edu 1 Chapter 5 (Conjugate Gradient Methods) Notes
More information