Lecture 7 Unconstrained nonlinear programming
|
|
- Loren Whitehead
- 5 years ago
- Views:
Transcription
1 Lecture 7 Unconstrained nonlinear programming Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University, tieli@pku.edu.cn No.1 Science Building, 1575
2 Outline Application examples
3 Energy minimization: virtual drug design Virtual drug design is to find a best position of a ligand (a small protein molecule) interacting with a large target protein molecule. It is equivalent to an energy minimization problem.
4 Energy minimization: protein folding Protein folding is to find the minimal energy state of a protein molecule from its sequence structure. It is an outstanding open problem for global optimization in the molecular mechanics.
5 Energy minimization: mathematical formulation Molecular force field V total = i k ri 2 (ri ri0)2 + i k θi 2 (θi θi0)2 + i V ni 2 (1+cos(nφi γi)) + ij (( σij ) 6 ( σij ) 12) 4ɛ + r ij r ij ij q iq j ɛr ij Webpage for the explanation of the force field Energy minimization problem with respect to all the configuration of the atoms min V total (x 1,..., x N )
6 Nonlinear least squares Suppose that we have a series of experimental data (t i, y i), i = 1,..., m. We wish to find parameter x R n such that the remainder r i(x) = y i f(t i, x), i = 1,..., m minimized. Mathematically, define error function where r = (r 1,..., r m) such that φ(x) = 1 2 r(x)t r(x) min φ(x). x Because the function f is nonlinear, it is called a nonlinear least square problem.
7 Optimal control problem Classical optimal control problem: such that the constraint dx dt min = g(x, u), T 0 f(x, u)dt x(0) = x0, x(t ) = xt is satisfied. Here u(t) is the control function, x(t) is the output. It is a nonlinear optimization in function space.
8 Optimal control problem Example: Isoparametric problem. max u 1 0 x 1(t)dt dx 1 dt = u, dx 2 dt = 1 + u 2. x 1(0) = x 1(1) = 0, x 2(0) = 0, x 2(1) = π 3 x 1 dx 2 dt dx =udt 1 1 t
9 Outline Application examples
10 Iterations Iterative methods Object: construct sequence {x k } k=1, such that x k converge to a fixed vector x, and x is the solution of the linear system. General iteration idea: If we want to solve equations g(x) = 0, and the equation x = f(x) has the same solution as it, then construct x k+1 = f(x k ). If x k x, then x = f(x ), thus the root of g(x) is obtained.
11 Convergence order Suppose an iterating sequence lim x n = x, and where ɛ n is called error bound. If when x n x ɛ n lim ɛn+1 ɛ n = C, 1. 0 < C < 1, x n is called linear convergence; q, q 2, q 3,, q n,, (q < 1) 2. C = 1, x n is called sublinear convergence; 1, 1 2, 1 3,, 1 n, 3. C = 0, x n is called superlinear convergence; 1, 1 2!, 1 3!,, 1 n!,
12 Convergence order If lim ɛn+1 ɛ p = C, C > 0, p > 1 n then x n is called p-th order convergence. q, q p, q p2,, q pn, Numerical examples for different convergence orders
13 Remark on p-th order convergence If p = 1, i.e. linear convergence, the number of significant digits is increasing linearly, such as 2, 3, 4, 5,...; If p > 1, the number of significant digits is increasing exponentially (O(p n )). Suppose p = 2, then the number of significant digits is increased as 2, 4, 8, 16,...!! So a very accurate result will be obtained after 4 5 iterations;
14 Golden section method Suppose there is a triplet (a, x k, c) and f(x k ) < f(a), f(x k ) < f(c), we want to find x k+1 in (a, c) to perform a section. Suppose x k+1 is in (a, x k ). x k+1 a xk c w z 1 w If f(x k+1 ) > f(x k ), then the new search interval is (x k+1, c); If f(x k+1 ) < f(x k ), then the new search interval is (a, x k ).
15 Golden section method Define and w = x k a c a, 1 w = c x k c a z = x k x k+1. c a If we want to minimize the worst case possibility (for two cases), we must make w = z + (1 w). (w > 1 2 ) Pay attention that w is also obtained from the previous stage of applying same strategy. This scale similarity implies we have z w = 1 w 5 1 w = This is called Golden section method.
16 Golden section method Golden section method is a method to find the local minimum of a function f. Golden section method is a linear convergence method. The contraction coefficient is C = Golden section method for Example min ϕ(x) = 0.5 xe x2 where a = 0, c = 2.
17 One dimensional Newton s method Suppose we want to minimize ϕ(x) min ϕ(x) x Taylor expansion at current iteration point x 0 ϕ(x) = ϕ(x 0) + ϕ (x 0)(x x 0) ϕ (x 0)(x x 0) 2 + Local quadratic approximation ϕ(x) g(x) = ϕ(x 0) + ϕ (x 0)(x x 0) ϕ (x 0)(x x 0) 2 Minimize g(x) at g (x) = 0, then Newton s method x 1 = x 0 ϕ (x 0) ϕ (x 0) x k+1 = x k ϕ (x k ) ϕ (x k )
18 One dimensional Newton s method Graphical explanation Local parabolic approximation x k+1 x k Example where x 0 = 0.5. min ϕ(x) = 0.5 xe x2
19 One dimensional Newton s method Theorem If ϕ (x ) 0, then Newton s method converges with second order if x 0 is close to x sufficiently. Drawbacks of Newton s method: 1. one needs to compute the second order derivative which is a huge cost (especially for high dimensional case). 2. The initial state x 0 must be very close to x.
20 High dimensional Newton s method Suppose we want to minimize f(x), x R n min f(x) x Taylor expansion at current iteration point x 0 f(x) = f(x 0) + f(x 0) (x x 0) (x x0)t 2 f(x 0)(x x 0) + Local quadratic approximation f(x) g(x) = f(x 0) + f(x 0) (x x 0) (x x0)t H f (x 0)(x x 0) where H f is the Hessian matrix defined as (H f ) ij = Minimize g(x) at g(x) = 0, then 2 f x i x j. x 1 = x 0 H f (x 0) 1 f(x 0) Newton s method x k+1 = x k H f (x k ) 1 f(x k )
21 High dimensional Newton s method Example min f(x 1, x 2) = 100(x 2 x 2 1) 2 + (x 1 1) 2 Initial state x 0 = ( 1.2, 1).
22 Steepest decent method Basic idea: Find a series of decent directions p k and corresponding stepsize α k such that the iterations x k+1 = x k + α k p k and f(x k+1 ) f(x k ). The negative gradient direction f is the steepest decent direction, so choose p k := f(x k ) and choose α k such that min α f(x k + αp k )
23 Inexact line search To find α such that min α f(x k + αp k ) is equivalent to perform a one dimensional minimization. But it is enough to find an approximate α by the following inexact line search method. Inexact line search is to make the following type of the decent criterion f(x k ) f(x k+1 ) ɛ 0 is satisfied.
24 Inexact line search An example of inexact line search strategy by half increment (or decrement) method: [a 0, b 0] = [0, + ), α 0 = 1; [a 1, b 1] = [0, 1], α 1 = 1 2 [a 2, b 2] = [0, 1 2 ], α2 = 1 4 ; [a3, b3] = [ 1 4, 1 2 ], α3 = 3 8 1/4 3/8 1/2 a=0 b=1
25 Steepest decent method Steepest decent method for example Initial state x 0 = ( 1.2, 1). min f(x 1, x 2) = 100(x 2 x 2 1) 2 + (x 1 1) 2
26 Dumped Newton s method If the initial value of Newton s method is not near the minimum point, a strategy is to apply dumped Newton s method. Choose the decent direction as the Newton s direction p k := H 1 f (x k ) f(x k ) and perform the inexact line search for min α f(x k + αp k )
27 Conjugate gradient method Recalling conjugate gradient method for quadratic function ϕ(x) = 1 2 xt Ax b T x 1. Initial step: x 0, p 0 = r 0 = b Ax 0 2. Suppose we have x k, r k, p k, the CGM step 2.1 Search the optimal α k along p k ; α k = (r k) T p k (p k ) T Ap k 2.2 Update x k and gradient direction r k ; x k+1 = x k + α k p k, r k+1 = b Ax k According to the calculation before to form new search direction p k+1 β k = (r k+1) T Ap k (p k ) T Ap k, p k+1 = r k+1 + β k p k
28 Conjugate gradient method Local quadratic approximation of general nonlinear optimization f(x) f(x 0) + f(x 0) (x x 0) (x x0)t H f (x 0)(x x 0) where H f (x 0) is the Hessian of f at x 0. Apply conjugate gradient method to the quadratic function above successively. The computation of β k needs the formation of Hessian matrix H f (x 0) which is a formidable task! Equivalent transformation in the quadratic case β k = (r k+1) T Ap k = ϕ(x k+1) 2 (p k ) T Ap k ϕ(x k ) 2 This formula does NOT need the computation of Hessian matrix.
29 Conjugate gradient method for nonlinear optimization Formally generalize CGM to nonlinear optimization 1. Given initial x 0 and ɛ > 0; 2. Compute g 0 = f(x 0) and p 0 = g 0, k = 0; 3. Compute λ k from and min λ f(x k + λp k ) x k+1 = x k + λ k p k, g k+1 = f(x k+1 ) 4. If g k+1 ɛ, the iteration is over. Otherwise compute µ k+1 = g k+1 2 g k 2 p k+1 = g k+1 + µ k+1 p k Set k = k + 1, iterate until convergence.
30 Conjugate gradient method In realistic computations, because there is only n conjugate gradient directions for n dimensional problem, it often restarts from current point after n iterations.
31 Conjugate gradient method CGM for example Initial state x 0 = ( 1.2, 1). min f(x 1, x 2) = 100(x 2 x 2 1) 2 + (x 1 1) 2
32 Variable metric method A general form of iterations x k+1 = x k λ k H k f(x k ) 1. If H k = I, it is steepest decent method; 2. If H k = [ 2 f(x k )] 1, it is dumped Newton s method. In order to keep the fast convergence of Newton s method, we hope to approximate [ 2 f(x k )] 1 as H k with reduced computational efforts as H k+1 = H k + C k, where C k is a correction matrix which is easily computed.
33 Variable metric method First consider quadratic function f(x) = a + b T x xt Gx we have f(x) = b + Gx Define g(x) = f(x), g k = g(x k ), then g k+1 g k = G(x k+1 x k ). Define x k = x k+1 x k, g k = g k+1 g k we have G x k = g k.
34 Variable metric method For general nonlinear function f(x) f(x k+1 )+ f(x k+1 ) (x x k+1 )+ 1 2 (x x k+1) T H f (x k+1 )(x x k+1 ). Similar procedure as above we have [H f (x k+1 )] 1 g k = x k. As H k+1 is a approximation of [H f (x k+1 )] 1, it must satisfy H k+1 g k = x k.
35 DFP method Davidon-Fletcher-Powell method: Choose C k as rank-2 correction matrix C k = α k uu T + β k vv T where α k, β k, u, v are undetermined variables. From H k+1 = H k + C k and H k+1 g k = x k we have α k u(u T g k ) + β k v(v T g k ) = x k H k g k Take u = H k g k, v = x k and α k = 1 u T g k, β k = We obtain the famous DFP method H k+1 = H k H k g k g T k H k g T k H k g k 1 v T g k + x k x T k x T k g k
36 Remark on DFP method If f(x) is quadratic and H 0 = I, then the result will converge in n steps theoretically; If f(x) is strictly convex, the DFP method is convergent globally. If H k is SPD and g k 0, then H k+1 is SPD also.
37 DFP method DFP method for example Initial state x 0 = ( 1.2, 1). min f(x 1, x 2) = 100(x 2 x 2 1) 2 + (x 1 1) 2
38 BFGS method The most popular variable metric method is BFGS (Broyden-Fletcher-Goldfarb-Shanno) method shown as below H k+1 = H k H k g k g T k H k g T k H k g k where + x k x T k x T k g + ( g T k H k g k )v k v T k k v k = x k x T k g H k g k k g T k H k g k BFGS is more stable than DFP method; BFGS is also a rank-2 correction method for H k.
39 Nonlinear least squares Mathematically, nonlinear least squares is to minimize φ(x) = 1 2 r(x)t r(x) We have m φ(x) = J T (x)r(x), H φ (x) = J T (x)j(x) + r i(x)h ri (x) i=1 where J(x) is the Jacobian matrix of r(x). Direct Newton s method for increment s k in nonlinear least squares H φ (x k )s k = φ(x k )
40 Gauss-Newton method If make the assumption that the residual r i(x) is very small, we will drop the term m i=1 ri(x)hr i (x) in Newton s method and we obtain Gauss-Newton method (J T (x k )J(x k ))s k = φ(x k ) Gauss-Newton method is equivalent to solve a sequence of linear least squares problems to approximate the nonlinear least squares.
41 Levenberg-Marquardt method If the Jacobian J(x) is ill-conditioned, one may take the Levenberg-Marquardt method as (J T (x k )J(x k ) + µ k I)s k = φ(x k ) where µ k is a nonnegative parameter chosen by some strategy. L-M method may be viewed as a regularization method for Gauss-Newton method.
42 Homework assignment Newton s method and BFGS method for example Initial state x 0 = ( 1.2, 1). min f(x 1, x 2) = 100(x 2 x 2 1) 2 + (x 1 1) 2
43 References J.F. Bonnans et al., Numerical optimization: Theoretical and practical aspects, Universitext, Springer, Berlin, 2003.
Optimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23
Optimization: Nonlinear Optimization without Constraints Nonlinear Optimization without Constraints 1 / 23 Nonlinear optimization without constraints Unconstrained minimization min x f(x) where f(x) is
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationChapter 4. Unconstrained optimization
Chapter 4. Unconstrained optimization Version: 28-10-2012 Material: (for details see) Chapter 11 in [FKS] (pp.251-276) A reference e.g. L.11.2 refers to the corresponding Lemma in the book [FKS] PDF-file
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Quasi Newton Methods Barnabás Póczos & Ryan Tibshirani Quasi Newton Methods 2 Outline Modified Newton Method Rank one correction of the inverse Rank two correction of the
More informationNonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More information5 Quasi-Newton Methods
Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min
More information2. Quasi-Newton methods
L. Vandenberghe EE236C (Spring 2016) 2. Quasi-Newton methods variable metric methods quasi-newton methods BFGS update limited-memory quasi-newton methods 2-1 Newton method for unconstrained minimization
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationMath 273a: Optimization Netwon s methods
Math 273a: Optimization Netwon s methods Instructor: Wotao Yin Department of Mathematics, UCLA Fall 2015 some material taken from Chong-Zak, 4th Ed. Main features of Newton s method Uses both first derivatives
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More informationMethods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent
Nonlinear Optimization Steepest Descent and Niclas Börlin Department of Computing Science Umeå University niclas.borlin@cs.umu.se A disadvantage with the Newton method is that the Hessian has to be derived
More informationQuasi-Newton Methods
Newton s Method Pros and Cons Quasi-Newton Methods MA 348 Kurt Bryan Newton s method has some very nice properties: It s extremely fast, at least once it gets near the minimum, and with the simple modifications
More informationMATH 4211/6211 Optimization Quasi-Newton Method
MATH 4211/6211 Optimization Quasi-Newton Method Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 Quasi-Newton Method Motivation:
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 3. Gradient Method
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 3 Gradient Method Shiqian Ma, MAT-258A: Numerical Optimization 2 3.1. Gradient method Classical gradient method: to minimize a differentiable convex
More informationLecture 14: October 17
1-725/36-725: Convex Optimization Fall 218 Lecture 14: October 17 Lecturer: Lecturer: Ryan Tibshirani Scribes: Pengsheng Guo, Xian Zhou Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationOutline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems
Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction
More informationUnconstrained optimization
Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout
More informationNonlinearOptimization
1/35 NonlinearOptimization Pavel Kordík Department of Computer Systems Faculty of Information Technology Czech Technical University in Prague Jiří Kašpar, Pavel Tvrdík, 2011 Unconstrained nonlinear optimization,
More informationOptimization 2. CS5240 Theoretical Foundations in Multimedia. Leow Wee Kheng
Optimization 2 CS5240 Theoretical Foundations in Multimedia Leow Wee Kheng Department of Computer Science School of Computing National University of Singapore Leow Wee Kheng (NUS) Optimization 2 1 / 38
More informationStatistics 580 Optimization Methods
Statistics 580 Optimization Methods Introduction Let fx be a given real-valued function on R p. The general optimization problem is to find an x ɛ R p at which fx attain a maximum or a minimum. It is of
More informationNewton s Method. Ryan Tibshirani Convex Optimization /36-725
Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x
More information1 Numerical optimization
Contents 1 Numerical optimization 5 1.1 Optimization of single-variable functions............ 5 1.1.1 Golden Section Search................... 6 1.1. Fibonacci Search...................... 8 1. Algorithms
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationNumerical Methods I Solving Nonlinear Equations
Numerical Methods I Solving Nonlinear Equations Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 16th, 2014 A. Donev (Courant Institute)
More informationMaximum Likelihood Estimation
Maximum Likelihood Estimation Prof. C. F. Jeff Wu ISyE 8813 Section 1 Motivation What is parameter estimation? A modeler proposes a model M(θ) for explaining some observed phenomenon θ are the parameters
More information1 Numerical optimization
Contents Numerical optimization 5. Optimization of single-variable functions.............................. 5.. Golden Section Search..................................... 6.. Fibonacci Search........................................
More informationComparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming Problems
International Journal of Scientific and Research Publications, Volume 3, Issue 10, October 013 1 ISSN 50-3153 Comparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming
More informationMaria Cameron. f(x) = 1 n
Maria Cameron 1. Local algorithms for solving nonlinear equations Here we discuss local methods for nonlinear equations r(x) =. These methods are Newton, inexact Newton and quasi-newton. We will show that
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationNumerical solutions of nonlinear systems of equations
Numerical solutions of nonlinear systems of equations Tsung-Ming Huang Department of Mathematics National Taiwan Normal University, Taiwan E-mail: min@math.ntnu.edu.tw August 28, 2011 Outline 1 Fixed points
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More informationAM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α
More informationImproving the Convergence of Back-Propogation Learning with Second Order Methods
the of Back-Propogation Learning with Second Order Methods Sue Becker and Yann le Cun, Sept 1988 Kasey Bray, October 2017 Table of Contents 1 with Back-Propagation 2 the of BP 3 A Computationally Feasible
More informationOptimization Methods
Optimization Methods Decision making Examples: determining which ingredients and in what quantities to add to a mixture being made so that it will meet specifications on its composition allocating available
More informationQuasi-Newton methods for minimization
Quasi-Newton methods for minimization Lectures for PHD course on Numerical optimization Enrico Bertolazzi DIMS Universitá di Trento November 21 December 14, 2011 Quasi-Newton methods for minimization 1
More informationOptimization II: Unconstrained Multivariable
Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 1
More informationEAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science
EAD 115 Numerical Solution of Engineering and Scientific Problems David M. Rocke Department of Applied Science Multidimensional Unconstrained Optimization Suppose we have a function f() of more than one
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationOptimization Tutorial 1. Basic Gradient Descent
E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.
More informationUniversity of Houston, Department of Mathematics Numerical Analysis, Fall 2005
3 Numerical Solution of Nonlinear Equations and Systems 3.1 Fixed point iteration Reamrk 3.1 Problem Given a function F : lr n lr n, compute x lr n such that ( ) F(x ) = 0. In this chapter, we consider
More informationLecture V. Numerical Optimization
Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize
More informationGradient-Based Optimization
Multidisciplinary Design Optimization 48 Chapter 3 Gradient-Based Optimization 3. Introduction In Chapter we described methods to minimize (or at least decrease) a function of one variable. While problems
More informationMethods for Unconstrained Optimization Numerical Optimization Lectures 1-2
Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Coralia Cartis, University of Oxford INFOMM CDT: Modelling, Analysis and Computation of Continuous Real-World Problems Methods
More informationQuasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno February 6, / 25 (BFG. Limited memory BFGS (L-BFGS)
Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb Shanno (BFGS) Limited memory BFGS (L-BFGS) February 6, 2014 Quasi-Newton methods: Symmetric rank 1 (SR1) Broyden Fletcher Goldfarb
More informationAM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality
More informationLecture 18: November Review on Primal-dual interior-poit methods
10-725/36-725: Convex Optimization Fall 2016 Lecturer: Lecturer: Javier Pena Lecture 18: November 2 Scribes: Scribes: Yizhu Lin, Pan Liu Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationStatic unconstrained optimization
Static unconstrained optimization 2 In unconstrained optimization an objective function is minimized without any additional restriction on the decision variables, i.e. min f(x) x X ad (2.) with X ad R
More informationOptimization and Root Finding. Kurt Hornik
Optimization and Root Finding Kurt Hornik Basics Root finding and unconstrained smooth optimization are closely related: Solving ƒ () = 0 can be accomplished via minimizing ƒ () 2 Slide 2 Basics Root finding
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More informationOptimization II: Unconstrained Multivariable
Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Optimization II: Unconstrained
More informationConstrained optimization. Unconstrained optimization. One-dimensional. Multi-dimensional. Newton with equality constraints. Active-set method.
Optimization Unconstrained optimization One-dimensional Multi-dimensional Newton s method Basic Newton Gauss- Newton Quasi- Newton Descent methods Gradient descent Conjugate gradient Constrained optimization
More informationMatrix Derivatives and Descent Optimization Methods
Matrix Derivatives and Descent Optimization Methods 1 Qiang Ning Department of Electrical and Computer Engineering Beckman Institute for Advanced Science and Techonology University of Illinois at Urbana-Champaign
More informationNumerical Analysis of Electromagnetic Fields
Pei-bai Zhou Numerical Analysis of Electromagnetic Fields With 157 Figures Springer-Verlag Berlin Heidelberg New York London Paris Tokyo Hong Kong Barcelona Budapest Contents Part 1 Universal Concepts
More informationThe Steepest Descent Algorithm for Unconstrained Optimization
The Steepest Descent Algorithm for Unconstrained Optimization Robert M. Freund February, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 1 Steepest Descent Algorithm The problem
More informationIPAM Summer School Optimization methods for machine learning. Jorge Nocedal
IPAM Summer School 2012 Tutorial on Optimization methods for machine learning Jorge Nocedal Northwestern University Overview 1. We discuss some characteristics of optimization problems arising in deep
More informationNumerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09
Numerical Optimization 1 Working Horse in Computer Vision Variational Methods Shape Analysis Machine Learning Markov Random Fields Geometry Common denominator: optimization problems 2 Overview of Methods
More information17 Solution of Nonlinear Systems
17 Solution of Nonlinear Systems We now discuss the solution of systems of nonlinear equations. An important ingredient will be the multivariate Taylor theorem. Theorem 17.1 Let D = {x 1, x 2,..., x m
More informationProximal Newton Method. Zico Kolter (notes by Ryan Tibshirani) Convex Optimization
Proximal Newton Method Zico Kolter (notes by Ryan Tibshirani) Convex Optimization 10-725 Consider the problem Last time: quasi-newton methods min x f(x) with f convex, twice differentiable, dom(f) = R
More informationQuasi-Newton Methods. Zico Kolter (notes by Ryan Tibshirani, Javier Peña, Zico Kolter) Convex Optimization
Quasi-Newton Methods Zico Kolter (notes by Ryan Tibshirani, Javier Peña, Zico Kolter) Convex Optimization 10-725 Last time: primal-dual interior-point methods Given the problem min x f(x) subject to h(x)
More information1 Newton s Method. Suppose we want to solve: x R. At x = x, f (x) can be approximated by:
Newton s Method Suppose we want to solve: (P:) min f (x) At x = x, f (x) can be approximated by: n x R. f (x) h(x) := f ( x)+ f ( x) T (x x)+ (x x) t H ( x)(x x), 2 which is the quadratic Taylor expansion
More informationProximal Newton Method. Ryan Tibshirani Convex Optimization /36-725
Proximal Newton Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: primal-dual interior-point method Given the problem min x subject to f(x) h i (x) 0, i = 1,... m Ax = b where f, h
More informationLecture 7: Optimization methods for non linear estimation or function estimation
Lecture 7: Optimization methods for non linear estimation or function estimation Y. Favennec 1, P. Le Masson 2 and Y. Jarny 1 1 LTN UMR CNRS 6607 Polytetch Nantes 44306 Nantes France 2 LIMATB Université
More informationMultivariate Newton Minimanization
Multivariate Newton Minimanization Optymalizacja syntezy biosurfaktantu Rhamnolipid Rhamnolipids are naturally occuring glycolipid produced commercially by the Pseudomonas aeruginosa species of bacteria.
More informationj=1 r 1 x 1 x n. r m r j (x) r j r j (x) r j (x). r j x k
Maria Cameron Nonlinear Least Squares Problem The nonlinear least squares problem arises when one needs to find optimal set of parameters for a nonlinear model given a large set of data The variables x,,
More information8 Numerical methods for unconstrained problems
8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields
More informationHigher-Order Methods
Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth
More informationCS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares
CS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares Robert Bridson October 29, 2008 1 Hessian Problems in Newton Last time we fixed one of plain Newton s problems by introducing line search
More informationGRADIENT = STEEPEST DESCENT
GRADIENT METHODS GRADIENT = STEEPEST DESCENT Convex Function Iso-contours gradient 0.5 0.4 4 2 0 8 0.3 0.2 0. 0 0. negative gradient 6 0.2 4 0.3 2.5 0.5 0 0.5 0.5 0 0.5 0.4 0.5.5 0.5 0 0.5 GRADIENT DESCENT
More informationNotes on Numerical Optimization
Notes on Numerical Optimization University of Chicago, 2014 Viva Patel October 18, 2014 1 Contents Contents 2 List of Algorithms 4 I Fundamentals of Optimization 5 1 Overview of Numerical Optimization
More informationData Mining (Mineria de Dades)
Data Mining (Mineria de Dades) Lluís A. Belanche belanche@lsi.upc.edu Soft Computing Research Group Dept. de Llenguatges i Sistemes Informàtics (Software department) Universitat Politècnica de Catalunya
More information14. Nonlinear equations
L. Vandenberghe ECE133A (Winter 2018) 14. Nonlinear equations Newton method for nonlinear equations damped Newton method for unconstrained minimization Newton method for nonlinear least squares 14-1 Set
More informationOptimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30
Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationCLASS NOTES Models, Algorithms and Data: Introduction to computing 2018
CLASS NOTES Models, Algorithms and Data: Introduction to computing 2018 Petros Koumoutsakos, Jens Honore Walther (Last update: April 16, 2018) IMPORTANT DISCLAIMERS 1. REFERENCES: Much of the material
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More information13. Nonlinear least squares
L. Vandenberghe ECE133A (Fall 2018) 13. Nonlinear least squares definition and examples derivatives and optimality condition Gauss Newton method Levenberg Marquardt method 13.1 Nonlinear least squares
More informationLecture 10: September 26
0-725: Optimization Fall 202 Lecture 0: September 26 Lecturer: Barnabas Poczos/Ryan Tibshirani Scribes: Yipei Wang, Zhiguang Huo Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These
More informationOptimization Methods. Lecture 18: Optimality Conditions and. Gradient Methods. for Unconstrained Optimization
5.93 Optimization Methods Lecture 8: Optimality Conditions and Gradient Methods for Unconstrained Optimization Outline. Necessary and sucient optimality conditions Slide. Gradient m e t h o d s 3. The
More informationx 2 x n r n J(x + t(x x ))(x x )dt. For warming-up we start with methods for solving a single equation of one variable.
Maria Cameron 1. Fixed point methods for solving nonlinear equations We address the problem of solving an equation of the form (1) r(x) = 0, where F (x) : R n R n is a vector-function. Eq. (1) can be written
More information1. Nonlinear Equations. This lecture note excerpted parts from Michael Heath and Max Gunzburger. f(x) = 0
Numerical Analysis 1 1. Nonlinear Equations This lecture note excerpted parts from Michael Heath and Max Gunzburger. Given function f, we seek value x for which where f : D R n R n is nonlinear. f(x) =
More informationOptimization Methods
Optimization Methods Categorization of Optimization Problems Continuous Optimization Discrete Optimization Combinatorial Optimization Variational Optimization Common Optimization Concepts in Computer Vision
More informationNumerical solution of Least Squares Problems 1/32
Numerical solution of Least Squares Problems 1/32 Linear Least Squares Problems Suppose that we have a matrix A of the size m n and the vector b of the size m 1. The linear least square problem is to find
More informationQuadratic Programming
Quadratic Programming Outline Linearly constrained minimization Linear equality constraints Linear inequality constraints Quadratic objective function 2 SideBar: Matrix Spaces Four fundamental subspaces
More informationLecture Notes to Accompany. Scientific Computing An Introductory Survey. by Michael T. Heath. Chapter 5. Nonlinear Equations
Lecture Notes to Accompany Scientific Computing An Introductory Survey Second Edition by Michael T Heath Chapter 5 Nonlinear Equations Copyright c 2001 Reproduction permitted only for noncommercial, educational
More informationOptimization Methods. Lecture 19: Line Searches and Newton s Method
15.93 Optimization Methods Lecture 19: Line Searches and Newton s Method 1 Last Lecture Necessary Conditions for Optimality (identifies candidates) x local min f(x ) =, f(x ) PSD Slide 1 Sufficient Conditions
More informationSearch Directions for Unconstrained Optimization
8 CHAPTER 8 Search Directions for Unconstrained Optimization In this chapter we study the choice of search directions used in our basic updating scheme x +1 = x + t d. for solving P min f(x). x R n All
More informationEECS260 Optimization Lecture notes
EECS260 Optimization Lecture notes Based on Numerical Optimization (Nocedal & Wright, Springer, 2nd ed., 2006) Miguel Á. Carreira-Perpiñán EECS, University of California, Merced May 2, 2010 1 Introduction
More informationQuasi-Newton Methods. Javier Peña Convex Optimization /36-725
Quasi-Newton Methods Javier Peña Convex Optimization 10-725/36-725 Last time: primal-dual interior-point methods Consider the problem min x subject to f(x) Ax = b h(x) 0 Assume f, h 1,..., h m are convex
More information(One Dimension) Problem: for a function f(x), find x 0 such that f(x 0 ) = 0. f(x)
Solving Nonlinear Equations & Optimization One Dimension Problem: or a unction, ind 0 such that 0 = 0. 0 One Root: The Bisection Method This one s guaranteed to converge at least to a singularity, i not
More informationApplied Mathematics 205. Unit I: Data Fitting. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit I: Data Fitting Lecturer: Dr. David Knezevic Unit I: Data Fitting Chapter I.4: Nonlinear Least Squares 2 / 25 Nonlinear Least Squares So far we have looked at finding a best
More informationLinear and Nonlinear Optimization
Linear and Nonlinear Optimization German University in Cairo October 10, 2016 Outline Introduction Gradient descent method Gauss-Newton method Levenberg-Marquardt method Case study: Straight lines have
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationOn nonlinear optimization since M.J.D. Powell
On nonlinear optimization since 1959 1 M.J.D. Powell Abstract: This view of the development of algorithms for nonlinear optimization is based on the research that has been of particular interest to the
More informationmin f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;
Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many
More informationNon-linear least squares
Non-linear least squares Concept of non-linear least squares We have extensively studied linear least squares or linear regression. We see that there is a unique regression line that can be determined
More informationORIE 6326: Convex Optimization. Quasi-Newton Methods
ORIE 6326: Convex Optimization Quasi-Newton Methods Professor Udell Operations Research and Information Engineering Cornell April 10, 2017 Slides on steepest descent and analysis of Newton s method adapted
More informationWritten Examination
Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes
More informationLecture 15 Newton Method and Self-Concordance. October 23, 2008
Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications
More information