Suppose that the approximate solutions of Eq. (1) satisfy the condition (3). Then (1) if η = 0 in the algorithm Trust Region, then lim inf.
|
|
- Dulcie Fields
- 5 years ago
- Views:
Transcription
1 Maria Cameron 1. Trust Region Methods At every iteration the trust region methods generate a model m k (p), choose a trust region, and solve the constraint optimization problem of finding the minimum of m k (p) within the trust region. Typically the trust region is chosen to be a ball around x k of radius k that is updated every iteration. For poorly scaled problems ellipsoidal trust regions can be chosen. The model m k (p) is typically quadratic and given by m k (p) = f k + f T k p pt B k p, f k := f(x k ), f k := f(x k ), and B k is some symmetric matrix. When B k = f k we have a trust region Newton method. In the rest of this section, we will discuss the outline of the trust region algorithm and its convergence, and exact and approximate techniques for solving the constraint optimization problem (1) m k (p) = f k + f T k p pt B k p, p k k Outline of the algorithm and convergence. The agreement between the model m k and the objective function within the trust region is quantified by the ratio (2) ρ k := f(x k) f(x k + p k ) m k (0) m k (p k ). The numerator is called the actual reduction, and the denominator is called the predicted reduction. The predicted reduction is always nonnegative. If ρ k is close to 1, the model is quite accurate, and the trust region can be increased. If ρ k 0, the model makes a poor prediction. Then the trust region needs to be decreased and the the step needs to be rejected. The algorithm implementing these ideas is given below. Algorithm Trust Region Input: max, 0 (0, max ], η [0, 1 4 ). for k = 0, 1, 2,... Obtain p k by solving Eq. (1) exactly or approximately; Calculate ρ k from Eq. (2); if ρ k < 1 4 set k+1 = 1 4 p k else if ρ k > 3 4 and p k = k set k+1 = min{2 k, max }; else set k+1 = k ; if ρ k > η accept step: x k+1 = x k + p k ; else reject step x k+1 = x k ; end The convergence properties of this algorithm depend on the parameter η and on whether some sufficient decrease is achieved at every iteration. The sufficient decrease condition is 1
2 2 given by the inequality (3) m k (0) m k (p k ) c 1 f k min { k, f } k, c 1 (0, 1]. B k Theorem 1. Suppose B k β for some constant β, and f is continuously differentiable and bounded from below on the set {x f(x) f(x 0 )}. Suppose that the approximate solutions of Eq. (1) satisfy the condition (3). Then (1) if η = 0 in the algorithm Trust Region, then lim inf k f k := lim k ( inf m>k f m ) = 0, i.e., one can extract a subsequence from { f k } converging to zero. (2) if η (0, 1 4 ) in the algorithm Trust Region and f is in addition Lipschitz continuously differentiable in the set {x f(x) f(x 0 )}, then lim f k = 0. k 1.2. Characterization of the exact solution of the trust region problem. Theorem 2. Let p be the global solution of the trust-region problem (4) min p R n m(p) = f + gt p pt Bp, p, if and only if there is a scalar λ 0 such that the following conditions are satisfied (5) (6) (7) (B + λi)p = g, λ( p ) = 0, (B + λi) is positive semidefinite. Condition (6) shows that at least one of the following holds: λ = 0 or p =. This means that p is a global minimizer of m(p) or, if not, p =, i.e., if p is not a global minimizer of m(p), the constrained minimum is achieved on the boundary of the region. Condition (5) implies that if λ > 0, p B k p + g m(p ), i.e., p is orthogonal to the level sets of m(p). Condition (7) tells that λ 1 λ 2... λ n are the eigenvalues of B then λ [ λ 1, ). The proof of this theorem relies on the following lemma. Lemma 1. Let m be the quadratic function defined by where B is any symmetric matrix. Then m(p) = g T p pt Bp,
3 (1) m attains a minimum if and only if B is positive semidefinite and g is in the range of B; (2) m has a unique minimizer if and only if B is positive definite; (3) if B is positive semidefinite, then every p satisfying Bp = g is a global minimizer of m. Note that of g is not in the range of B then m(p) does not attain minimum. For example, let m(x, y) = y + x 2. Here B =. Proof. [ ] [ 0, while g = 1 ] is not in the range of B. Obviously, min m(x, y) = (1) ( =): Since g is in the range of B, one can find p such that Bp = g. Then for all w R n we have m(p + w) = g T (p + w) (p + w)t B(p + w) = (g T p pt Bp) + g T w + (Bp) T w wt Bw = m(p) wt Bw m(p), since B is positive semidefinite. (= ): Let p be a minimizer of m. Since m(p) = Bp + g = 0, g is in the range of B. Also, m(p) = B is positive semidefinite. (2) ( =): Since B is positive definite and hence invertible, one can find p such that Bp = g. Repeating the calculation from the previous item and taking into account that 1 2 wt Bw > 0 for all nonzero w we obtain that the minimizer is unique. (= ): Let p be a minimizer of m. From the proof of previous item B must be positive semidefinite. If B is not positive definite, one can find w 0 such that Bw = 0. Then m(p) = m(p + w), hence the minimizer is not unique, a contradiction. (3) The proof of the last item follows from the proof of the first item. Now we will proof Theorem 2 Proof. ( =): Suppose there is λ 0 such that Eqs (5)-(7) are satisfied. Lemma 1 (3) implies that p is a global minimum of the quadratic function Since ˆm(p) ˆm(p ), we have ˆm(p) = g T p pt (B + λi)p = m(p) + λ 2 pt p. m(p) m(p ) + λ 2 (p T p p T p). Since λ( p ) = 0 and therefore λ( 2 p T p ) = 0 we have m(p) m(p ) + λ 2 ( 2 p T p). 3
4 4 Since λ 0 we have that m(p) m(p ) for all p such that p. Therefore, p is a global solution of Eq. (4). (= ): Suppose p is a global solution of Eq. (4). First consider the case where p <. Then p is an unconstrained minimizer of m(p). Hence m(p ) = g + Bp = 0, m(p ) = B is positive semidefinite. Hence Eqs. (5)-(7) hold for λ = 0. Now we assume that p =. Then Eq. (6) is satisfied. Hence p is the minimum of m satisfying the constraint p =. Then the Lagrangian function has a stationary point at p satisfying L(p, λ) = m(p) + λ 2 (pt p 2 ) p L(p, λ) = Bp + g + λp = (B + λi)p + g = 0. Hence Eq. (5) holds. Since m(p) m(p ) for all p such that p = we have m(p) m(p ) + λ 2 (p T p p T p). Substituting the expression for g g = (B + λi)p into the last equation we get 1 2 (p p ) T (B + λi)(p p ) 0. Since the set of directions {w : w = ± p } p p p, p = is dense in the unit sphere, we conclude that (B + λi) is positive definite. It remains to show that λ 0. Since we have proven that (B + λi)p = g and B + λi is positive semidefinite, we have that p is a minimum of ˆm(p) = g T p pt (B + λi)p. Hence ˆm(p) ˆm(p ), i.e., m(p) m(p ) + λ 2 (p T p p T p). Now suppose that only some negative λ satisfies Eqs. (5)-(7). Then from the last equation we have that m(p) m(p ) whenever p p =. Since p minimizes m for all p with p we conclude that p is an unconstrained global minimizer of m. From Lemma 1 (1) it follows that them Bp = g and B is positive semidefinite. Hence Eqs. (5)-(7) are satisfies by λ = 0. This contradicts to the assumption that λ satisfying these conditions is negative. Thus, there exists λ 0 satisfying Eqs. (5)-(7).
5 1.3. Calculation of nearly exact solution. We start solving the trust region problem (4) with checking whether B is positive definite, and if it is, checking whether p = B 1 g satisfies p <. If B is positive semidefinite and g is in the range of B, one can find the minimum norm solution p of the underdetermined system Bp = g and check whether p. Now suppose that either B is not positive semidefinite or the global minimizer of m satisfies p >. Then we define p(λ) := (B + λi) 1 g, λ max{0, λ 1 }, where λ 1 is the smallest eigenvalue of B, and look for λ such that Let B = QΛQ T where Then Then p(λ =. Λ = diag{λ 1,..., λ n }, λ 1... λ n, Q = [q 1... q n ], p(λ) = Q(Λ + λ j I) 1 Q T = (8) p(λ) 2 = n j=1 q T j q k = δ jk n j=1 (q T j g)2 (λ j + λ) 2. q T j g λ j + λ q j. Therefore, the problem of solving (4) is reduced to the 1D root-finding problem (8). Note that if B is positive definite and B 1 g > then there is exactly one solution λ of Eq. (8) on the interval [0, ) since Read [1] for details. lim p(λ) = 0. λ 1.4. Approximate solution of the trust region problem. Three approaches for approximate solution of the trust region problem (4) are considered in [1]: the Dogleg approach, the 2D subspace approach: p span{g, B 1 g}, and Steihaug s approach good for large and sparse B = f k and based on the Conjugate Gradient method. We will consider the Dogleg approach and the 2D subspace minimization approach. We will start with the concept of the Cauchy point that is used for reference: the approximate solution must reduce the objective function f at least as much as the Cauchy point does. 5
6 Cauchy point. The Cauchy point is the minimizer of m k (p) = f k + f T k p pt B k P, p k along the steepest descent direction f k. It is readily found in the explicit form. The steepest descent direction is given by f k. A vector of length k in this direction is p s k := f k f k k. We will look for the Cauchy point in the form p c k = τ kp s k. We need to consider two cases: f T k B k f k 0 and f T k B k f k > 0. If f T k B k f k 0 the function M(τ) := m k (τp s k ) = f k τ f k k + τ 2 2 fk T B k f k f k 2 2 k, τ [ 1, 1] decreases monotonically as τ grows whenever f k. Hence we need to pick the largest admissible τ k, i.e., τ k = 1. If f T k B k f k > 0, the global minimum of M(τ) is achieved at τ min = f k 3 k f T k B k f k. Hence if τ min 1 the global minimum of M is achieved within the interval [ 1, 1]. Otherwise we need to pick the largest τ toward the minimum, i.e., τ = 1. To summarize, we have found that the Cauchy point is given by where p c k = τ f k k f k k { 1, if f T k B k f k 0, τ k = { } f min k 3 k fk T B, 1 otherwise. k f k The Cauchy point provides a sufficient reduction to the model to give global convergence. However, implementing the Cauchy point at every step we simply use the steepest descent algorithm with a particular choice of step length. It is well-known that the steepest descent performs poorly even if the optimal step length is chosen at every iteration. This consideration motivates us to fid a better approximate solution of the trust region problem than the Cauchy point.
7 The Dogleg Method. The Dogleg method method is suitable for the case where B k is positive definite. Its name is motivated by the fact that the solution of the trust region problem is looked along the path consisting of two line segments: from x k to the unconstrained minimum of m k (p) along the steepest descent direction and then to the unconstrained minimum of the quadratic model. We observe that of is small, the quadratic term in m k has little influence on the direction of the step: the direction is approximately f k, while if is large, the solution of the trust region problem is the global minimizer of the quadratic model. The unconstrained minimum of m k along the steepest descent direction is given by p U = gt g g T B k g, g := f k. The global minimizer of the quadratic model is given by p B = B 1 k g. The dogleg path p(τ), τ [0, 2] is defined by { τp U, 0 τ 1 p(τ) = p U + (τ 1)(p B p U ), 1 τ 2. The following lemma shows that the dogleg path intersects the trust region boundary at most once and the intersection point can be computed analytically! Lemma 2. Let B k be positive definite. Then (1) p(τ) is an increasing function of τ; (2) m(p(τ)) is an increasing function of τ. The proof can be found in [1]. The solution is calculated as follows. If p B k then p = p B. If p B l k while p U < k we solve the quadratic equation to find τ: If p U k we set p U + (τ 1)(p B p U ) 2 = 2 k. τ = p U Two-dimensional subspace minimization. This approach is an extension of the dogleg approach. Suppose B is positive definite. Then we solve the following constrained minimization problem (9) min p m(p) = f + g T p pt Bp, p, p span{g, B 1 g}. If B has negative eigenvalues, we look for p in another subspace defined by p span{g, (B + αi) 1 g}, α ( λ 1, 2λ 1 ), where λ 1 is the most negative eigenvalue of B. If B has zero eigenvalues but no negative eigenvalues, we use the Cauchy point as an approximate solution.
8 8 References [1] J. Nocedal, S. Wright, Numerical Optimization, Springer, 1999
Trust Region Methods. Lecturer: Pradeep Ravikumar Co-instructor: Aarti Singh. Convex Optimization /36-725
Trust Region Methods Lecturer: Pradeep Ravikumar Co-instructor: Aarti Singh Convex Optimization 10-725/36-725 Trust Region Methods min p m k (p) f(x k + p) s.t. p 2 R k Iteratively solve approximations
More informationLecture 7: CS395T Numerical Optimization for Graphics and AI Trust Region Methods
Lecture 7: CS395T Numerical Optimization for Graphics and AI Trust Region Methods Qixing Huang The University of Texas at Austin huangqx@cs.utexas.edu 1 Disclaimer This note is adapted from Section 4 of
More informationTrust Regions. Charles J. Geyer. March 27, 2013
Trust Regions Charles J. Geyer March 27, 2013 1 Trust Region Theory We follow Nocedal and Wright (1999, Chapter 4), using their notation. Fletcher (1987, Section 5.1) discusses the same algorithm, but
More informationj=1 r 1 x 1 x n. r m r j (x) r j r j (x) r j (x). r j x k
Maria Cameron Nonlinear Least Squares Problem The nonlinear least squares problem arises when one needs to find optimal set of parameters for a nonlinear model given a large set of data The variables x,,
More informationPart 3: Trust-region methods for unconstrained optimization. Nick Gould (RAL)
Part 3: Trust-region methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective
More informationIntroduction. New Nonsmooth Trust Region Method for Unconstraint Locally Lipschitz Optimization Problems
New Nonsmooth Trust Region Method for Unconstraint Locally Lipschitz Optimization Problems Z. Akbari 1, R. Yousefpour 2, M. R. Peyghami 3 1 Department of Mathematics, K.N. Toosi University of Technology,
More informationNotes on Numerical Optimization
Notes on Numerical Optimization University of Chicago, 2014 Viva Patel October 18, 2014 1 Contents Contents 2 List of Algorithms 4 I Fundamentals of Optimization 5 1 Overview of Numerical Optimization
More informationSecond Order Optimization Algorithms I
Second Order Optimization Algorithms I Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 7, 8, 9 and 10 1 The
More informationOPER 627: Nonlinear Optimization Lecture 14: Mid-term Review
OPER 627: Nonlinear Optimization Lecture 14: Mid-term Review Department of Statistical Sciences and Operations Research Virginia Commonwealth University Oct 16, 2013 (Lecture 14) Nonlinear Optimization
More informationOPER 627: Nonlinear Optimization Lecture 9: Trust-region methods
OPER 627: Nonlinear Optimization Lecture 9: Trust-region methods Department of Statistical Sciences and Operations Research Virginia Commonwealth University Sept 25, 2013 (Lecture 9) Nonlinear Optimization
More informationOn Lagrange multipliers of trust-region subproblems
On Lagrange multipliers of trust-region subproblems Ladislav Lukšan, Ctirad Matonoha, Jan Vlček Institute of Computer Science AS CR, Prague Programy a algoritmy numerické matematiky 14 1.- 6. června 2008
More informationHigher-Order Methods
Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth
More information(Here > 0 is given.) In cases where M is convex, there is a nice theory for this problem; the theory has much more general applicability too.
Convex Analysis with Applications UBC Math 604 Lecture Notes by Philip D. Loewen In trust region methods, we minimize a quadratic model function M = M(p) over the set of all p R n satisfying a constraint
More informationMaria Cameron. f(x) = 1 n
Maria Cameron 1. Local algorithms for solving nonlinear equations Here we discuss local methods for nonlinear equations r(x) =. These methods are Newton, inexact Newton and quasi-newton. We will show that
More informationGradient-Based Optimization
Multidisciplinary Design Optimization 48 Chapter 3 Gradient-Based Optimization 3. Introduction In Chapter we described methods to minimize (or at least decrease) a function of one variable. While problems
More informationInexact Newton Methods Applied to Under Determined Systems. Joseph P. Simonis. A Dissertation. Submitted to the Faculty
Inexact Newton Methods Applied to Under Determined Systems by Joseph P. Simonis A Dissertation Submitted to the Faculty of WORCESTER POLYTECHNIC INSTITUTE in Partial Fulfillment of the Requirements for
More information5 Handling Constraints
5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest
More informationTrust-region methods for rectangular systems of nonlinear equations
Trust-region methods for rectangular systems of nonlinear equations Margherita Porcelli Dipartimento di Matematica U.Dini Università degli Studi di Firenze Joint work with Maria Macconi and Benedetta Morini
More informationOn Lagrange multipliers of trust region subproblems
On Lagrange multipliers of trust region subproblems Ladislav Lukšan, Ctirad Matonoha, Jan Vlček Institute of Computer Science AS CR, Prague Applied Linear Algebra April 28-30, 2008 Novi Sad, Serbia Outline
More informationx 2 x n r n J(x + t(x x ))(x x )dt. For warming-up we start with methods for solving a single equation of one variable.
Maria Cameron 1. Fixed point methods for solving nonlinear equations We address the problem of solving an equation of the form (1) r(x) = 0, where F (x) : R n R n is a vector-function. Eq. (1) can be written
More informationUnconstrained optimization
Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More informationMethods for Unconstrained Optimization Numerical Optimization Lectures 1-2
Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Coralia Cartis, University of Oxford INFOMM CDT: Modelling, Analysis and Computation of Continuous Real-World Problems Methods
More informationGradient Descent. Dr. Xiaowei Huang
Gradient Descent Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Three machine learning algorithms: decision tree learning k-nn linear regression only optimization objectives are discussed,
More informationWritten Examination
Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes
More informationStatic unconstrained optimization
Static unconstrained optimization 2 In unconstrained optimization an objective function is minimized without any additional restriction on the decision variables, i.e. min f(x) x X ad (2.) with X ad R
More informationNonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More informationLecture 12 Unconstrained Optimization (contd.) Constrained Optimization. October 15, 2008
Lecture 12 Unconstrained Optimization (contd.) Constrained Optimization October 15, 2008 Outline Lecture 11 Gradient descent algorithm Improvement to result in Lec 11 At what rate will it converge? Constrained
More information8 Numerical methods for unconstrained problems
8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields
More informationAn Inexact Newton Method for Nonlinear Constrained Optimization
An Inexact Newton Method for Nonlinear Constrained Optimization Frank E. Curtis Numerical Analysis Seminar, January 23, 2009 Outline Motivation and background Algorithm development and theoretical results
More information1 Directional Derivatives and Differentiability
Wednesday, January 18, 2012 1 Directional Derivatives and Differentiability Let E R N, let f : E R and let x 0 E. Given a direction v R N, let L be the line through x 0 in the direction v, that is, L :=
More informationIntroduction to Nonlinear Optimization Paul J. Atzberger
Introduction to Nonlinear Optimization Paul J. Atzberger Comments should be sent to: atzberg@math.ucsb.edu Introduction We shall discuss in these notes a brief introduction to nonlinear optimization concepts,
More informationHow to Characterize the Worst-Case Performance of Algorithms for Nonconvex Optimization
How to Characterize the Worst-Case Performance of Algorithms for Nonconvex Optimization Frank E. Curtis Department of Industrial and Systems Engineering, Lehigh University Daniel P. Robinson Department
More informationPerformance Surfaces and Optimum Points
CSC 302 1.5 Neural Networks Performance Surfaces and Optimum Points 1 Entrance Performance learning is another important class of learning law. Network parameters are adjusted to optimize the performance
More informationEECS260 Optimization Lecture notes
EECS260 Optimization Lecture notes Based on Numerical Optimization (Nocedal & Wright, Springer, 2nd ed., 2006) Miguel Á. Carreira-Perpiñán EECS, University of California, Merced May 2, 2010 1 Introduction
More informationOn the iterate convergence of descent methods for convex optimization
On the iterate convergence of descent methods for convex optimization Clovis C. Gonzaga March 1, 2014 Abstract We study the iterate convergence of strong descent algorithms applied to convex functions.
More informationOptimization and Optimal Control in Banach Spaces
Optimization and Optimal Control in Banach Spaces Bernhard Schmitzer October 19, 2017 1 Convex non-smooth optimization with proximal operators Remark 1.1 (Motivation). Convex optimization: easier to solve,
More informationA trust region algorithm with a worst-case iteration complexity of O(ɛ 3/2 ) for nonconvex optimization
Math. Program., Ser. A DOI 10.1007/s10107-016-1026-2 FULL LENGTH PAPER A trust region algorithm with a worst-case iteration complexity of O(ɛ 3/2 ) for nonconvex optimization Frank E. Curtis 1 Daniel P.
More informationNOTES ON FIRST-ORDER METHODS FOR MINIMIZING SMOOTH FUNCTIONS. 1. Introduction. We consider first-order methods for smooth, unconstrained
NOTES ON FIRST-ORDER METHODS FOR MINIMIZING SMOOTH FUNCTIONS 1. Introduction. We consider first-order methods for smooth, unconstrained optimization: (1.1) minimize f(x), x R n where f : R n R. We assume
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationComplexity of gradient descent for multiobjective optimization
Complexity of gradient descent for multiobjective optimization J. Fliege A. I. F. Vaz L. N. Vicente July 18, 2018 Abstract A number of first-order methods have been proposed for smooth multiobjective optimization
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationCONSTRAINED NONLINEAR PROGRAMMING
149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach
More informationUniversità di Firenze, via C. Lombroso 6/17, Firenze, Italia,
Convergence of a Regularized Euclidean Residual Algorithm for Nonlinear Least-Squares by S. Bellavia 1, C. Cartis 2, N. I. M. Gould 3, B. Morini 1 and Ph. L. Toint 4 25 July 2008 1 Dipartimento di Energetica
More informationDerivative-Free Trust-Region methods
Derivative-Free Trust-Region methods MTH6418 S. Le Digabel, École Polytechnique de Montréal Fall 2015 (v4) MTH6418: DFTR 1/32 Plan Quadratic models Model Quality Derivative-Free Trust-Region Framework
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More information10. Unconstrained minimization
Convex Optimization Boyd & Vandenberghe 10. Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method self-concordant functions implementation
More informationw T 1 w T 2. w T n 0 if i j 1 if i = j
Lyapunov Operator Let A F n n be given, and define a linear operator L A : C n n C n n as L A (X) := A X + XA Suppose A is diagonalizable (what follows can be generalized even if this is not possible -
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2
More informationMA677 Assignment #3 Morgan Schreffler Due 09/19/12 Exercise 1 Using Hölder s inequality, prove Minkowski s inequality for f, g L p (R d ), p 1:
Exercise 1 Using Hölder s inequality, prove Minkowski s inequality for f, g L p (R d ), p 1: f + g p f p + g p. Proof. If f, g L p (R d ), then since f(x) + g(x) max {f(x), g(x)}, we have f(x) + g(x) p
More informationGlobal convergence of trust-region algorithms for constrained minimization without derivatives
Global convergence of trust-region algorithms for constrained minimization without derivatives P.D. Conejo E.W. Karas A.A. Ribeiro L.G. Pedroso M. Sachine September 27, 2012 Abstract In this work we propose
More informationMultipoint secant and interpolation methods with nonmonotone line search for solving systems of nonlinear equations
Multipoint secant and interpolation methods with nonmonotone line search for solving systems of nonlinear equations Oleg Burdakov a,, Ahmad Kamandi b a Department of Mathematics, Linköping University,
More informationIntroduction to Real Analysis Alternative Chapter 1
Christopher Heil Introduction to Real Analysis Alternative Chapter 1 A Primer on Norms and Banach Spaces Last Updated: March 10, 2018 c 2018 by Christopher Heil Chapter 1 A Primer on Norms and Banach Spaces
More informationA new ane scaling interior point algorithm for nonlinear optimization subject to linear equality and inequality constraints
Journal of Computational and Applied Mathematics 161 (003) 1 5 www.elsevier.com/locate/cam A new ane scaling interior point algorithm for nonlinear optimization subject to linear equality and inequality
More informationNumerical Optimization
Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function
More informationTHE restructuring of the power industry has lead to
GLOBALLY CONVERGENT OPTIMAL POWER FLOW USING COMPLEMENTARITY FUNCTIONS AND TRUST REGION METHODS Geraldo L. Torres Universidade Federal de Pernambuco Recife, Brazil gltorres@ieee.org Abstract - As power
More informationOn fast trust region methods for quadratic models with linear constraints. M.J.D. Powell
DAMTP 2014/NA02 On fast trust region methods for quadratic models with linear constraints M.J.D. Powell Abstract: Quadratic models Q k (x), x R n, of the objective function F (x), x R n, are used by many
More informationUnconstrained minimization of smooth functions
Unconstrained minimization of smooth functions We want to solve min x R N f(x), where f is convex. In this section, we will assume that f is differentiable (so its gradient exists at every point), and
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationConstrained optimization. Unconstrained optimization. One-dimensional. Multi-dimensional. Newton with equality constraints. Active-set method.
Optimization Unconstrained optimization One-dimensional Multi-dimensional Newton s method Basic Newton Gauss- Newton Quasi- Newton Descent methods Gradient descent Conjugate gradient Constrained optimization
More informationChapter 4. Unconstrained optimization
Chapter 4. Unconstrained optimization Version: 28-10-2012 Material: (for details see) Chapter 11 in [FKS] (pp.251-276) A reference e.g. L.11.2 refers to the corresponding Lemma in the book [FKS] PDF-file
More informationA Distributed Newton Method for Network Utility Maximization, II: Convergence
A Distributed Newton Method for Network Utility Maximization, II: Convergence Ermin Wei, Asuman Ozdaglar, and Ali Jadbabaie October 31, 2012 Abstract The existing distributed algorithms for Network Utility
More informationORIE 6326: Convex Optimization. Quasi-Newton Methods
ORIE 6326: Convex Optimization Quasi-Newton Methods Professor Udell Operations Research and Information Engineering Cornell April 10, 2017 Slides on steepest descent and analysis of Newton s method adapted
More information5.6 Penalty method and augmented Lagrangian method
5.6 Penalty method and augmented Lagrangian method Consider a generic NLP problem min f (x) s.t. c i (x) 0 i I c i (x) = 0 i E (1) x R n where f and the c i s are of class C 1 or C 2, and I and E are the
More informationExamination paper for TMA4180 Optimization I
Department of Mathematical Sciences Examination paper for TMA4180 Optimization I Academic contact during examination: Phone: Examination date: 26th May 2016 Examination time (from to): 09:00 13:00 Permitted
More informationMathematical optimization
Optimization Mathematical optimization Determine the best solutions to certain mathematically defined problems that are under constrained determine optimality criteria determine the convergence of the
More informationInterpolation-Based Trust-Region Methods for DFO
Interpolation-Based Trust-Region Methods for DFO Luis Nunes Vicente University of Coimbra (joint work with A. Bandeira, A. R. Conn, S. Gratton, and K. Scheinberg) July 27, 2010 ICCOPT, Santiago http//www.mat.uc.pt/~lnv
More informationLecture 3: Linesearch methods (continued). Steepest descent methods
Lecture 3: Linesearch methods (continued). Steepest descent methods Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lecture 3: Linesearch methods (continued).
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More information1. Introduction. We analyze a trust region version of Newton s method for the optimization problem
SIAM J. OPTIM. Vol. 9, No. 4, pp. 1100 1127 c 1999 Society for Industrial and Applied Mathematics NEWTON S METHOD FOR LARGE BOUND-CONSTRAINED OPTIMIZATION PROBLEMS CHIH-JEN LIN AND JORGE J. MORÉ To John
More informationConvex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods
More informationStep-size Estimation for Unconstrained Optimization Methods
Volume 24, N. 3, pp. 399 416, 2005 Copyright 2005 SBMAC ISSN 0101-8205 www.scielo.br/cam Step-size Estimation for Unconstrained Optimization Methods ZHEN-JUN SHI 1,2 and JIE SHEN 3 1 College of Operations
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationNonlinear equations. Norms for R n. Convergence orders for iterative methods
Nonlinear equations Norms for R n Assume that X is a vector space. A norm is a mapping X R with x such that for all x, y X, α R x = = x = αx = α x x + y x + y We define the following norms on the vector
More informationStochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions
International Journal of Control Vol. 00, No. 00, January 2007, 1 10 Stochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions I-JENG WANG and JAMES C.
More informationWorst Case Complexity of Direct Search
Worst Case Complexity of Direct Search L. N. Vicente May 3, 200 Abstract In this paper we prove that direct search of directional type shares the worst case complexity bound of steepest descent when sufficient
More informationThis manuscript is for review purposes only.
1 2 3 4 5 6 7 8 9 10 11 12 THE USE OF QUADRATIC REGULARIZATION WITH A CUBIC DESCENT CONDITION FOR UNCONSTRAINED OPTIMIZATION E. G. BIRGIN AND J. M. MARTíNEZ Abstract. Cubic-regularization and trust-region
More informationConjugate Gradient Method
Conjugate Gradient Method direct and indirect methods positive definite linear systems Krylov sequence spectral analysis of Krylov sequence preconditioning Prof. S. Boyd, EE364b, Stanford University Three
More informationPenalty and Barrier Methods General classical constrained minimization problem minimize f(x) subject to g(x) 0 h(x) =0 Penalty methods are motivated by the desire to use unconstrained optimization techniques
More informationIterative Methods for Solving A x = b
Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http
More informationAppendix A Functional Analysis
Appendix A Functional Analysis A.1 Metric Spaces, Banach Spaces, and Hilbert Spaces Definition A.1. Metric space. Let X be a set. A map d : X X R is called metric on X if for all x,y,z X it is i) d(x,y)
More informationPDE-Constrained and Nonsmooth Optimization
Frank E. Curtis October 1, 2009 Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP)
More informationAn improved convergence theorem for the Newton method under relaxed continuity assumptions
An improved convergence theorem for the Newton method under relaxed continuity assumptions Andrei Dubin ITEP, 117218, BCheremushinsaya 25, Moscow, Russia Abstract In the framewor of the majorization technique,
More informationAM 205: lecture 18. Last time: optimization methods Today: conditions for optimality
AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality Existence of Global Minimum For example: f (x, y) = x 2 + y 2 is coercive on R 2 (global min. at (0, 0)) f (x) = x 3
More informationChap 3. Linear Algebra
Chap 3. Linear Algebra Outlines 1. Introduction 2. Basis, Representation, and Orthonormalization 3. Linear Algebraic Equations 4. Similarity Transformation 5. Diagonal Form and Jordan Form 6. Functions
More information1. Introduction. We consider nonlinear optimization problems of the form. f(x) ce (x) = 0 c I (x) 0,
AN INTERIOR-POINT ALGORITHM FOR LARGE-SCALE NONLINEAR OPTIMIZATION WITH INEXACT STEP COMPUTATIONS FRANK E. CURTIS, OLAF SCHENK, AND ANDREAS WÄCHTER Abstract. We present a line-search algorithm for large-scale
More informationNewton s Method. Javier Peña Convex Optimization /36-725
Newton s Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and
More informationOptimization and Root Finding. Kurt Hornik
Optimization and Root Finding Kurt Hornik Basics Root finding and unconstrained smooth optimization are closely related: Solving ƒ () = 0 can be accomplished via minimizing ƒ () 2 Slide 2 Basics Root finding
More informationGeometry optimization
Geometry optimization Trygve Helgaker Centre for Theoretical and Computational Chemistry Department of Chemistry, University of Oslo, Norway European Summer School in Quantum Chemistry (ESQC) 211 Torre
More information6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE. Three Alternatives/Remedies for Gradient Projection
6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE Three Alternatives/Remedies for Gradient Projection Two-Metric Projection Methods Manifold Suboptimization Methods
More information10. Ellipsoid method
10. Ellipsoid method EE236C (Spring 2008-09) ellipsoid method convergence proof inequality constraints 10 1 Ellipsoid method history developed by Shor, Nemirovski, Yudin in 1970s used in 1979 by Khachian
More informationChapter 7 Iterative Techniques in Matrix Algebra
Chapter 7 Iterative Techniques in Matrix Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More informationOptimization II: Unconstrained Multivariable
Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 1
More information