Derivative-Free Trust-Region methods
|
|
- Calvin Ellis
- 6 years ago
- Views:
Transcription
1 Derivative-Free Trust-Region methods MTH6418 S. Le Digabel, École Polytechnique de Montréal Fall 2015 (v4) MTH6418: DFTR 1/32
2 Plan Quadratic models Model Quality Derivative-Free Trust-Region Framework References MTH6418: DFTR 2/32
3 Quadratic models Model Quality Derivative-Free Trust-Region Framework References MTH6418: DFTR 3/32
4 Quadratic model of f Natural basis of the space of polynomials of degree 2 in R n. It has q + 1 = (n + 1)(n + 2)/2 elements. φ(x) = (φ 0 (x) φ 1 (x)... φ q (x)) T = ( ) x 1 x 1 x 2... x 2 1 x 2 2 x n T n 2 x 1 x 2 x 1 x 3... x n 1 x n. Model of f: m f defined by α R q+1. m f (x) = α T φ(x). MTH6418: DFTR 4/32
5 Interpolation set Points at which f is known and which are used to construct the model m f. p + 1 elements of R n : Y = {y 0, y 1,..., y p }. These points are also called data points. f(y ) = ( f(y 0 ) f(y 1 )... f(y p ) ) T R p+1. The geometry of Y is important and will be studied later. How to select the data points from the cache points? One solution: Take the points around the current iterate. MTH6418: DFTR 5/32
6 Construction of the model Find α R q+1 such that y Y (f(y) m f (y)) 2 is minimal. Idea: Solve M(φ, Y )α = f(y ) with φ 0 (y 0 ) φ 1 (y 0 )... φ q (y 0 ) φ 0 (y 1 ) φ 1 (y 1 )... φ q (y 1 ) M(φ, Y ) =.... R(p+1) (q+1). φ 0 (y p ) φ 1 (y p )... φ q (y p ) Cost in O(p 3 ). 3 cases: p = q: Determined. p > q: Overdetermined. p < q: Underdetermined. MTH6418: DFTR 6/32
7 Number of necessary interpolation points n q + 1 = (n+1)(n+2) Typically in the DFO context, n 20, but: Very limited number of evaluations. Selection of the data points near the current iterate. The underdetermined case p < q is the most common. MTH6418: DFTR 7/32
8 Overdetermined & determined cases: 1/2 More data points than necessary: p > q. Use regression to solve the system in the least square sense, i.e. solve: min α R q+1 M(φ, Y )α f(y ) 2. If M(φ, Y ) has full column rank, analytic and unique solution given by α = M(φ, Y ) + f(y ) with M(φ, Y ) + = [ M(φ, Y ) T M(φ, Y ) ] 1 M(φ, Y ) T the pseudoinverse of M(φ, Y ). Works for the determined case p = q (exact interpolation). MTH6418: DFTR 8/32
9 Overdetermined & determined cases: 2/2 M(φ, Y ) can be decomposed using the Singular Value Decomposition (SVD): M(φ, Y ) = UΣV T with: U R (p+1) (p+1), U T U = I p+1. Σ R (p+1) (q+1), diagonal: Singular values 0 (sv). V R (q+1) (q+1), V V T = I q+1. M(φ, Y ) has not full rank if the smallest sv is 0. Condition number of M(φ, Y ): Largest sv / smallest sv. M(φ, Y ) + = V Σ + U T where Σ + is the pseudoinverse of Σ obtained by replacing every non-zero sv by its reciprocal and transposing the resulting matrix. Cost of the SVD: O((p + 1)(q + 1) 2 ). MTH6418: DFTR 9/32
10 Underdetermined case: Infinite number of solutions Minimum Frobenius Norm (MFN) interpolation: Choose a solution that minimizes the Frobenius norm of the Hessian of the model (the curvature). [ ] αl α = with α L R n+1, α Q R n Q, n Q = n(n + 1)/2. α Q [ M(φQ, Y )M(φ F (φ, Y ) = Q, Y ) T ] M(φ L, Y ) M(φ L, Y ) T 0 R (p+n+2) (p+n+2). [ ] [ ] µ f(y ) F (φ, Y ) = α α L 0 L R n+1 and µ R p+1. Use decomposition to solve the system, and then: α Q = M(φ Q, Y ) T µ R n Q. MTH6418: DFTR 10/32
11 Lagrange polynomials Basis of Lagrange polynomials: p + 1 polynomials l j for j = 0, 1,..., p, with: { l j (y i 1 if i = j ) = 0 if i j. Model: m f (x) = p f(y i )l i (x). i=0 Cost of constructing a model is in O(p 3 ), but cost of updating the model by one point is in O(p 2 ). MTH6418: DFTR 11/32
12 Lagrange polynomials: Example f(x, y) = x + y + 2x 2 + 3y 3 {[ ] [ ] [ Y =,, ], [ 2 0 ] [ 1, 1 ] [ 0, 2 l 0 (x, y) = x 3 2 y x y2 + xy, l 1 (x, y) = 2x x 2 xy, l 2 (x, y) = 2y y 2 xy, l 3 (x, y) = 1 2 x x2, l 4 (x, y) = xy, l 5 (x, y) = 1 2 y y2. m f (x, y) = 0 l 0 (x, y) + 3l 1 (x, y) + 4l 2 (x, y) + 10l 3 (x, y) + 7l 4 (x, y) + 26l 5 (x, y) = 2x 2 + 9y 2 + x 5y. ]} MTH6418: DFTR 12/32
13 Quadratic models Model Quality Derivative-Free Trust-Region Framework References MTH6418: DFTR 13/32
14 FL and FQ models A model m f is called Fully Linear (FL) { on B(y; ), for f C 1 and f Lipschitz f(x) mf (x) κ continuous, if f 2 f(x) m(x) κ g. Fully Quadratic (FQ) on B(y; ), for f C 2 and f 2 f(x) m f (x) κ f 3 Lipschitz continuous, if f(x) m(x) κ g 2 2 f(x) 2 m(x) κ h. For all x B(y; ) and some constants κ f, κ g, κ h. MTH6418: DFTR 14/32
15 FL and FQ class of models A set of models M = {m : R n R, m C 2 } is called a FL (FQ) class of models if: There exists a FL (FQ) model in M. There exists a model-improvement algorithm (MIA) that, in a finite number of steps, can: Determine if a given model is FL (FQ) on B(x; ). Or find a model that is FL (FQ) on B(x; ). MTH6418: DFTR 15/32
16 Well-poisedness A set Y is said poised for polynomial interpolation or regression if M(φ, Y ) is nonsingular (p = q), or if M(φ, Y ) has full rank. Well-poisedness. Good geometry of Y = well-poised set Y. Condition number of M(φ, Y ) may be a good indicator only with some bases φ and if some specific scaling is performed. Lagrange polynomials can be good indicators. Quantify the well-poisedness with the Λ-poisedness. MTH6418: DFTR 16/32
17 Λ-poisedness Let Λ > 0 and B R n. Y is Λ-poised in B if Λ Λ l = max max l i(x) where B(Y ) is the smallest ball 0 i p x B(Y ) containing Y. Or for all x B, there exists λ R p+1 such that p φ(x) = λ i φ(y i ) and max λ i Λ. 0 i p i=0 Or replacing any point in Y by any x B can increase the volume of the set φ(y ) at most by a factor Λ, with φ(y ) = {φ(y 0 ), φ(y 1 ),..., φ(y p )} and its volume defined by det(m(φ,y )) (p+1)!. MTH6418: DFTR 17/32
18 Quadratic models Model Quality Derivative-Free Trust-Region Framework References MTH6418: DFTR 18/32
19 Introduction We consider the unconstrained problem min x R n f(x). Bounds and linear constraints can be easily treated. Need more elaborate strategies to handle general constraints. See Lesson #9 on the constraints. We present a first order algorithm that ensures global convergence to first order critical points using a FL class of models. This is the general DFTR framework from [Conn et al., 2009]. We suppose f C 1 and f Lipchitz continuous. But derivatives are not available. MTH6418: DFTR 19/32
20 Notations for the DFTR framework x k : Current iterate. Model of f: m f (x) = f(x k ) + g T k (x x k) + 1/2(x x k ) T H k (x xk) with g k, H k : Gradient and Hessian of the model at iteration k. k : Trust-region radius. For candidate t: r k (t) = f(x k) f(t) m f (x k ) m f (t). m f m f {t} means: update the model with t. ε c > 0. 0 η 0 η 1 < 1, η < γ dec < 1 < γ inc. µ > 0. MTH6418: DFTR 20/32
21 First order algorithm: 1/3 Step 0 [Initialization] Choose FL class of models. Model-improvement algorithm (MIA). x 0, max, 0 ]0; max ], initialize model m f, k 0. Step 1 [Criticality test]: If g k ε c Call MIA to certify m f FL on B(x k ; k ). If ((mf not FL on B(x k ; k )) or ( k > µ g k )) [model not good enough or trust-region too large]: Construct new model. Check stopping criterion; Stop or goto [Step 1]. MTH6418: DFTR 21/32
22 First order algorithm: 2/3 Step 2 [Subproblem Optimization] Find t argmin x B(x k ; k ) m f (x). Evaluate candidate t in B(xk ; k ). Compute rk (t) = f(x k) f(t) m f (x k ) m f (t). Step 3 [Acceptance of candidate] If rk (t) > η 1 or (r k (t) > η 0 and m f is FL on B(x k ; k ) ): x k+1 t, m f m f {t}. Otherwise: x k+1 x k. Step 4 [Model Improvement] If r k (t) < η 1 and m f not FL Call MIA to certify mf FL on B(x k ; k ). MTH6418: DFTR 22/32
23 First order algorithm: 3/3 Step 5 [Trust-region radius update] k+1 [ k ; min{γ inc k, max }] if r k (t) η 1, {γ dec k } if r k (t) < η 1 and m f is FL, { k } if r k (t) < η 1 and m f is not FL. k k + 1, goto [Step 1]. MTH6418: DFTR 23/32
24 First order algorithm: Comments Successful iteration if r k (t) η 1. Then k+1 k. Acceptable iteration if η 1 > r k (t) η 0 and m f is FL. Then k+1 < k. Model-improving iteration if r k (t) < η 1 and m f not FL. Then model must be improved and x k, k are not updated. Unsuccessful iteration if r k (t) < η 0 and m f is FL. Then k+1 < k and x k is not updated. Do not reduce the trust-region radius when the model is not good. MTH6418: DFTR 24/32
25 Second order algorithm Global convergence to second order critical points using a FQ class of models. f C 2 and f Lipschitz continuous. Second order stationarity of the model: σk m = max{ g k, λ min (H k )} where λ min (H k ) denotes the smallest eigenvalue of H k. Criticality test based on σ m k instead of g k. MTH6418: DFTR 25/32
26 Definition of the subproblem Trust-region subproblem. We want to solve candidate t. min m f (x) in order to obtain a x B(x k ; k ) The trust-region constraint can be expressed with different norms. We do not need an exact resolution. MTH6418: DFTR 26/32
27 Optimization of the subproblem Some methods to solve the subproblem: Gradient projection. Moré Sorensen. Generalized Lanczos trust-region. Sequential Subspace. Gould Robinson Thorne. Rendl Wolkowicz. MTH6418: DFTR 27/32
28 Quadratic models Model Quality Derivative-Free Trust-Region Framework References MTH6418: DFTR 28/32
29 DFTR solvers BOBYQA. COBYLA. CONDOR. DFO. LINCOA. NEWUOA. ORBIT. SNOBFIT. Wedge. MTH6418: DFTR 29/32
30 References I Conn, A., Scheinberg, K., and Vicente, L. (2009). Introduction to Derivative-Free Optimization. MOS-SIAM Series on Optimization. SIAM, Philadelphia. Golub, G. and Van Loan, C. (1996). Matrix Computations, chapter The Singular Value Decomposition, pages The John Hopkins University Press, Baltimore and London, third edition. (SVD). Gould, N., Lucidi, S., and Toint, P. (1999). Solving the trust-region subproblem using the Lanczos method. SIAM Journal on Optimization, 9(2): MTH6418: DFTR 30/32
31 References II Gould, N., Robinson, D., and Thorne, H. (2010). On solving trust-region and other regularised subproblems in optimization. Mathematical Programming Computation, 2(1): Moré, J. and Sorensen, D. (1983). Computing a trust region step. SIAM Journal on Scientific Computing, 4(3): Nocedal, J. and Wright, S. (2006). Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer, Berlin, second edition. (Gradient projection). MTH6418: DFTR 31/32
32 References III Rendl, F. and Wolkowicz, H. (1997). A semidefinite framework for trust region subproblems with applications to large scale minimization. Mathematical Programming, 77(1): MTH6418: DFTR 32/32
Interpolation-Based Trust-Region Methods for DFO
Interpolation-Based Trust-Region Methods for DFO Luis Nunes Vicente University of Coimbra (joint work with A. Bandeira, A. R. Conn, S. Gratton, and K. Scheinberg) July 27, 2010 ICCOPT, Santiago http//www.mat.uc.pt/~lnv
More informationA recursive model-based trust-region method for derivative-free bound-constrained optimization.
A recursive model-based trust-region method for derivative-free bound-constrained optimization. ANKE TRÖLTZSCH [CERFACS, TOULOUSE, FRANCE] JOINT WORK WITH: SERGE GRATTON [ENSEEIHT, TOULOUSE, FRANCE] PHILIPPE
More information1101 Kitchawan Road, Yorktown Heights, New York 10598, USA
Self-correcting geometry in model-based algorithms for derivative-free unconstrained optimization by K. Scheinberg and Ph. L. Toint Report 9/6 February 9 IBM T.J. Watson Center Kitchawan Road Yorktown
More informationGEOMETRY OF INTERPOLATION SETS IN DERIVATIVE FREE OPTIMIZATION
GEOMETRY OF INTERPOLATION SETS IN DERIVATIVE FREE OPTIMIZATION ANDREW R. CONN, KATYA SCHEINBERG, AND LUíS N. VICENTE Abstract. We consider derivative free methods based on sampling approaches for nonlinear
More informationOn Lagrange multipliers of trust region subproblems
On Lagrange multipliers of trust region subproblems Ladislav Lukšan, Ctirad Matonoha, Jan Vlček Institute of Computer Science AS CR, Prague Applied Linear Algebra April 28-30, 2008 Novi Sad, Serbia Outline
More informationBilevel Derivative-Free Optimization and its Application to Robust Optimization
Bilevel Derivative-Free Optimization and its Application to Robust Optimization A. R. Conn L. N. Vicente September 15, 2010 Abstract We address bilevel programming problems when the derivatives of both
More informationGlobal convergence of trust-region algorithms for constrained minimization without derivatives
Global convergence of trust-region algorithms for constrained minimization without derivatives P.D. Conejo E.W. Karas A.A. Ribeiro L.G. Pedroso M. Sachine September 27, 2012 Abstract In this work we propose
More informationOn Lagrange multipliers of trust-region subproblems
On Lagrange multipliers of trust-region subproblems Ladislav Lukšan, Ctirad Matonoha, Jan Vlček Institute of Computer Science AS CR, Prague Programy a algoritmy numerické matematiky 14 1.- 6. června 2008
More informationA DERIVATIVE-FREE ALGORITHM FOR THE LEAST-SQUARE MINIMIZATION
A DERIVATIVE-FREE ALGORITHM FOR THE LEAST-SQUARE MINIMIZATION HONGCHAO ZHANG, ANDREW R. CONN, AND KATYA SCHEINBERG Abstract. We develop a framework for a class of derivative-free algorithms for the least-squares
More informationHigher-Order Methods
Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth
More informationA trust-region derivative-free algorithm for constrained optimization
A trust-region derivative-free algorithm for constrained optimization P.D. Conejo and E.W. Karas and L.G. Pedroso February 26, 204 Abstract We propose a trust-region algorithm for constrained optimization
More informationA derivative-free trust-funnel method for equality-constrained nonlinear optimization by Ph. R. Sampaio and Ph. L. Toint Report NAXYS-??-204 8 February 204 Log 2 Scaled Performance Profile on Subset of
More informationA trust-funnel method for nonlinear optimization problems with general nonlinear constraints and its application to derivative-free optimization
A trust-funnel method for nonlinear optimization problems with general nonlinear constraints and its application to derivative-free optimization by Ph. R. Sampaio and Ph. L. Toint Report naxys-3-25 January
More informationSURVEY OF TRUST-REGION DERIVATIVE FREE OPTIMIZATION METHODS
Manuscript submitted to AIMS Journals Volume X, Number 0X, XX 200X Website: http://aimsciences.org pp. X XX SURVEY OF TRUST-REGION DERIVATIVE FREE OPTIMIZATION METHODS Middle East Technical University
More informationUSING SIMPLEX GRADIENTS OF NONSMOOTH FUNCTIONS IN DIRECT SEARCH METHODS
Pré-Publicações do Departamento de Matemática Universidade de Coimbra Preprint Number 06 48 USING SIMPLEX GRADIENTS OF NONSMOOTH FUNCTIONS IN DIRECT SEARCH METHODS A. L. CUSTÓDIO, J. E. DENNIS JR. AND
More informationA Derivative-Free Gauss-Newton Method
A Derivative-Free Gauss-Newton Method Coralia Cartis Lindon Roberts 29th October 2017 Abstract We present, a derivative-free version of the Gauss-Newton method for solving nonlinear least-squares problems.
More informationALGORITHM XXX: SC-SR1: MATLAB SOFTWARE FOR SOLVING SHAPE-CHANGING L-SR1 TRUST-REGION SUBPROBLEMS
ALGORITHM XXX: SC-SR1: MATLAB SOFTWARE FOR SOLVING SHAPE-CHANGING L-SR1 TRUST-REGION SUBPROBLEMS JOHANNES BRUST, OLEG BURDAKOV, JENNIFER B. ERWAY, ROUMMEL F. MARCIA, AND YA-XIANG YUAN Abstract. We present
More informationSuppose that the approximate solutions of Eq. (1) satisfy the condition (3). Then (1) if η = 0 in the algorithm Trust Region, then lim inf.
Maria Cameron 1. Trust Region Methods At every iteration the trust region methods generate a model m k (p), choose a trust region, and solve the constraint optimization problem of finding the minimum of
More informationBOOSTERS: A Derivative-Free Algorithm Based on Radial Basis Functions
BOOSTERS: A Derivative-Free Algorithm Based on Radial Basis Functions R. OEUVRAY Rodrigue.Oeuvay@gmail.com M. BIERLAIRE Michel.Bierlaire@epfl.ch December 21, 2005 Abstract Derivative-free optimization
More informationUSING SIMPLEX GRADIENTS OF NONSMOOTH FUNCTIONS IN DIRECT SEARCH METHODS
USING SIMPLEX GRADIENTS OF NONSMOOTH FUNCTIONS IN DIRECT SEARCH METHODS A. L. CUSTÓDIO, J. E. DENNIS JR., AND L. N. VICENTE Abstract. It has been shown recently that the efficiency of direct search methods
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods
More informationCOMP 558 lecture 18 Nov. 15, 2010
Least squares We have seen several least squares problems thus far, and we will see more in the upcoming lectures. For this reason it is good to have a more general picture of these problems and how to
More informationDELFT UNIVERSITY OF TECHNOLOGY
DELFT UNIVERSITY OF TECHNOLOGY REPORT 10-12 Large-Scale Eigenvalue Problems in Trust-Region Calculations Marielba Rojas, Bjørn H. Fotland, and Trond Steihaug ISSN 1389-6520 Reports of the Department of
More information1. Introduction. In this paper, we design a class of derivative-free optimization algorithms for the following least-squares problem:
SIAM J. OPTIM. Vol. 20, No. 6, pp. 3555 3576 c 2010 Society for Industrial and Applied Mathematics A DERIVATIVE-FREE ALGORITHM FOR LEAST-SQUARES MINIMIZATION HONGCHAO ZHANG, ANDREW R. CONN, AND KATYA SCHEINBERG
More informationGI07/COMPM012: Mathematical Programming and Research Methods (Part 2) 2. Least Squares and Principal Components Analysis. Massimiliano Pontil
GI07/COMPM012: Mathematical Programming and Research Methods (Part 2) 2. Least Squares and Principal Components Analysis Massimiliano Pontil 1 Today s plan SVD and principal component analysis (PCA) Connection
More informationPart 3: Trust-region methods for unconstrained optimization. Nick Gould (RAL)
Part 3: Trust-region methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective
More informationA conjugate gradient-based algorithm for large-scale quadratic programming problem with one quadratic constraint
Noname manuscript No. (will be inserted by the editor) A conjugate gradient-based algorithm for large-scale quadratic programming problem with one quadratic constraint A. Taati, M. Salahi the date of receipt
More informationA Trust Funnel Algorithm for Nonconvex Equality Constrained Optimization with O(ɛ 3/2 ) Complexity
A Trust Funnel Algorithm for Nonconvex Equality Constrained Optimization with O(ɛ 3/2 ) Complexity Mohammadreza Samadi, Lehigh University joint work with Frank E. Curtis (stand-in presenter), Lehigh University
More informationGlobal and derivative-free optimization Lectures 1-4
Global and derivative-free optimization Lectures 1-4 Coralia Cartis, University of Oxford INFOMM CDT: Contemporary Numerical Techniques Global and derivative-free optimizationlectures 1-4 p. 1/46 Lectures
More informationSolving Separable Nonlinear Equations Using LU Factorization
Western Washington University Western CEDAR Mathematics College of Science and Engineering 03 Solving Separable Nonlinear Equations Using LU Factorization Yun-Qiu Shen Western Washington University, yunqiu.shen@wwu.edu
More informationThis manuscript is for review purposes only.
1 2 3 4 5 6 7 8 9 10 11 12 THE USE OF QUADRATIC REGULARIZATION WITH A CUBIC DESCENT CONDITION FOR UNCONSTRAINED OPTIMIZATION E. G. BIRGIN AND J. M. MARTíNEZ Abstract. Cubic-regularization and trust-region
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationTotal least squares. Gérard MEURANT. October, 2008
Total least squares Gérard MEURANT October, 2008 1 Introduction to total least squares 2 Approximation of the TLS secular equation 3 Numerical experiments Introduction to total least squares In least squares
More informationLSTRS 1.2: MATLAB Software for Large-Scale Trust-Regions Subproblems and Regularization
LSTRS 1.2: MATLAB Software for Large-Scale Trust-Regions Subproblems and Regularization Marielba Rojas Informatics and Mathematical Modelling Technical University of Denmark Computational Methods with
More informationOn fast trust region methods for quadratic models with linear constraints. M.J.D. Powell
DAMTP 2014/NA02 On fast trust region methods for quadratic models with linear constraints M.J.D. Powell Abstract: Quadratic models Q k (x), x R n, of the objective function F (x), x R n, are used by many
More information2. Linear algebra. matrices and vectors. linear equations. range and nullspace of matrices. function of vectors, gradient and Hessian
FE661 - Statistical Methods for Financial Engineering 2. Linear algebra Jitkomut Songsiri matrices and vectors linear equations range and nullspace of matrices function of vectors, gradient and Hessian
More informationA decoupled first/second-order steps technique for nonconvex nonlinear unconstrained optimization with improved complexity bounds
A decoupled first/second-order steps technique for nonconvex nonlinear unconstrained optimization with improved complexity bounds S. Gratton C. W. Royer L. N. Vicente August 5, 28 Abstract In order to
More informationAn interior-point trust-region polynomial algorithm for convex programming
An interior-point trust-region polynomial algorithm for convex programming Ye LU and Ya-xiang YUAN Abstract. An interior-point trust-region algorithm is proposed for minimization of a convex quadratic
More informationA New Trust Region Algorithm Using Radial Basis Function Models
A New Trust Region Algorithm Using Radial Basis Function Models Seppo Pulkkinen University of Turku Department of Mathematics July 14, 2010 Outline 1 Introduction 2 Background Taylor series approximations
More informationNewton s Method. Javier Peña Convex Optimization /36-725
Newton s Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More informationMethods for Unconstrained Optimization Numerical Optimization Lectures 1-2
Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Coralia Cartis, University of Oxford INFOMM CDT: Modelling, Analysis and Computation of Continuous Real-World Problems Methods
More informationBindel, Fall 2009 Matrix Computations (CS 6210) Week 8: Friday, Oct 17
Logistics Week 8: Friday, Oct 17 1. HW 3 errata: in Problem 1, I meant to say p i < i, not that p i is strictly ascending my apologies. You would want p i > i if you were simply forming the matrices and
More informationSobolev seminorm of quadratic functions with applications to derivative-free optimization
Noname manuscript No. (will be inserted by the editor) Sobolev seorm of quadratic functions with applications to derivative-free optimization Zaikun Zhang Received: date / Accepted: date Abstract This
More information1. Introduction Let the least value of an objective function F (x), x2r n, be required, where F (x) can be calculated for any vector of variables x2r
DAMTP 2002/NA08 Least Frobenius norm updating of quadratic models that satisfy interpolation conditions 1 M.J.D. Powell Abstract: Quadratic models of objective functions are highly useful in many optimization
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization /36-725
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: proximal gradient descent Consider the problem min g(x) + h(x) with g, h convex, g differentiable, and h simple
More informationMesures de criticalité d'ordres 1 et 2 en recherche directe
Mesures de criticalité d'ordres 1 et 2 en recherche directe From rst to second-order criticality measures in direct search Clément Royer ENSEEIHT-IRIT, Toulouse, France Co-auteurs: S. Gratton, L. N. Vicente
More information5.6 Penalty method and augmented Lagrangian method
5.6 Penalty method and augmented Lagrangian method Consider a generic NLP problem min f (x) s.t. c i (x) 0 i I c i (x) = 0 i E (1) x R n where f and the c i s are of class C 1 or C 2, and I and E are the
More informationTrust Regions. Charles J. Geyer. March 27, 2013
Trust Regions Charles J. Geyer March 27, 2013 1 Trust Region Theory We follow Nocedal and Wright (1999, Chapter 4), using their notation. Fletcher (1987, Section 5.1) discusses the same algorithm, but
More information1. Introduction. In this paper we address unconstrained local minimization
ORBIT: OPTIMIZATION BY RADIAL BASIS FUNCTION INTERPOLATION IN TRUST-REGIONS STEFAN M. WILD, ROMMEL G. REGIS, AND CHRISTINE A. SHOEMAKER Abstract. We present a new derivative-free algorithm, ORBIT, for
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationPreliminary Examination, Numerical Analysis, August 2016
Preliminary Examination, Numerical Analysis, August 2016 Instructions: This exam is closed books and notes. The time allowed is three hours and you need to work on any three out of questions 1-4 and any
More informationOn the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method
Optimization Methods and Software Vol. 00, No. 00, Month 200x, 1 11 On the Local Quadratic Convergence of the Primal-Dual Augmented Lagrangian Method ROMAN A. POLYAK Department of SEOR and Mathematical
More information5.5 Quadratic programming
5.5 Quadratic programming Minimize a quadratic function subject to linear constraints: 1 min x t Qx + c t x 2 s.t. a t i x b i i I (P a t i x = b i i E x R n, where Q is an n n matrix, I and E are the
More informationUniversity of Houston, Department of Mathematics Numerical Analysis, Fall 2005
3 Numerical Solution of Nonlinear Equations and Systems 3.1 Fixed point iteration Reamrk 3.1 Problem Given a function F : lr n lr n, compute x lr n such that ( ) F(x ) = 0. In this chapter, we consider
More informationThe Trust Region Subproblem with Non-Intersecting Linear Constraints
The Trust Region Subproblem with Non-Intersecting Linear Constraints Samuel Burer Boshi Yang February 21, 2013 Abstract This paper studies an extended trust region subproblem (etrs in which the trust region
More informationIntroduction to Nonlinear Optimization Paul J. Atzberger
Introduction to Nonlinear Optimization Paul J. Atzberger Comments should be sent to: atzberg@math.ucsb.edu Introduction We shall discuss in these notes a brief introduction to nonlinear optimization concepts,
More informationLecture 6. Numerical methods. Approximation of functions
Lecture 6 Numerical methods Approximation of functions Lecture 6 OUTLINE 1. Approximation and interpolation 2. Least-square method basis functions design matrix residual weighted least squares normal equation
More informationConditional Gradient (Frank-Wolfe) Method
Conditional Gradient (Frank-Wolfe) Method Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 1 Outline Today: Conditional gradient method Convergence analysis Properties
More informationGlobal Convergence of Radial Basis Function Trust Region Derivative-Free Algorithms
ARGONNE NATIONAL LABORATORY 9700 South Cass Avenue Argonne, Illinois 60439 Global Convergence of Radial Basis Function Trust Region Derivative-Free Algorithms Stefan M. Wild and Christine Shoemaker Mathematics
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More information1 Singular Value Decomposition and Principal Component
Singular Value Decomposition and Principal Component Analysis In these lectures we discuss the SVD and the PCA, two of the most widely used tools in machine learning. Principal Component Analysis (PCA)
More informationMax-Planck-Institut für Mathematik in den Naturwissenschaften Leipzig
Max-Planck-Institut für Mathematik in den Naturwissenschaften Leipzig Convergence analysis of Riemannian GaussNewton methods and its connection with the geometric condition number by Paul Breiding and
More information1. Introduction. We analyze a trust region version of Newton s method for the optimization problem
SIAM J. OPTIM. Vol. 9, No. 4, pp. 1100 1127 c 1999 Society for Industrial and Applied Mathematics NEWTON S METHOD FOR LARGE BOUND-CONSTRAINED OPTIMIZATION PROBLEMS CHIH-JEN LIN AND JORGE J. MORÉ To John
More informationKey words. conjugate gradients, normwise backward error, incremental norm estimation.
Proceedings of ALGORITMY 2016 pp. 323 332 ON ERROR ESTIMATION IN THE CONJUGATE GRADIENT METHOD: NORMWISE BACKWARD ERROR PETR TICHÝ Abstract. Using an idea of Duff and Vömel [BIT, 42 (2002), pp. 300 322
More informationNewton s Method. Ryan Tibshirani Convex Optimization /36-725
Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x
More information10-725/36-725: Convex Optimization Prerequisite Topics
10-725/36-725: Convex Optimization Prerequisite Topics February 3, 2015 This is meant to be a brief, informal refresher of some topics that will form building blocks in this course. The content of the
More informationConstrained optimization: direct methods (cont.)
Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a
More information1. Background: The SVD and the best basis (questions selected from Ch. 6- Can you fill in the exercises?)
Math 35 Exam Review SOLUTIONS Overview In this third of the course we focused on linear learning algorithms to model data. summarize: To. Background: The SVD and the best basis (questions selected from
More informationMaria Cameron. f(x) = 1 n
Maria Cameron 1. Local algorithms for solving nonlinear equations Here we discuss local methods for nonlinear equations r(x) =. These methods are Newton, inexact Newton and quasi-newton. We will show that
More informationKeywords: Nonlinear least-squares problems, regularized models, error bound condition, local convergence.
STRONG LOCAL CONVERGENCE PROPERTIES OF ADAPTIVE REGULARIZED METHODS FOR NONLINEAR LEAST-SQUARES S. BELLAVIA AND B. MORINI Abstract. This paper studies adaptive regularized methods for nonlinear least-squares
More informationTHE restructuring of the power industry has lead to
GLOBALLY CONVERGENT OPTIMAL POWER FLOW USING COMPLEMENTARITY FUNCTIONS AND TRUST REGION METHODS Geraldo L. Torres Universidade Federal de Pernambuco Recife, Brazil gltorres@ieee.org Abstract - As power
More informationDS-GA 1002 Lecture notes 10 November 23, Linear models
DS-GA 2 Lecture notes November 23, 2 Linear functions Linear models A linear model encodes the assumption that two quantities are linearly related. Mathematically, this is characterized using linear functions.
More informationj=1 r 1 x 1 x n. r m r j (x) r j r j (x) r j (x). r j x k
Maria Cameron Nonlinear Least Squares Problem The nonlinear least squares problem arises when one needs to find optimal set of parameters for a nonlinear model given a large set of data The variables x,,
More informationarxiv: v1 [math.oc] 1 Jul 2016
Convergence Rate of Frank-Wolfe for Non-Convex Objectives Simon Lacoste-Julien INRIA - SIERRA team ENS, Paris June 8, 016 Abstract arxiv:1607.00345v1 [math.oc] 1 Jul 016 We give a simple proof that the
More informationChap 3. Linear Algebra
Chap 3. Linear Algebra Outlines 1. Introduction 2. Basis, Representation, and Orthonormalization 3. Linear Algebraic Equations 4. Similarity Transformation 5. Diagonal Form and Jordan Form 6. Functions
More informationr=1 r=1 argmin Q Jt (20) After computing the descent direction d Jt 2 dt H t d + P (x + d) d i = 0, i / J
7 Appendix 7. Proof of Theorem Proof. There are two main difficulties in proving the convergence of our algorithm, and none of them is addressed in previous works. First, the Hessian matrix H is a block-structured
More informationApplying Bayesian Estimation to Noisy Simulation Optimization
Applying Bayesian Estimation to Noisy Simulation Optimization Geng Deng Michael C. Ferris University of Wisconsin-Madison INFORMS Annual Meeting Pittsburgh 2006 Simulation-based optimization problem Computer
More information10. Unconstrained minimization
Convex Optimization Boyd & Vandenberghe 10. Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method self-concordant functions implementation
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationMATH 350: Introduction to Computational Mathematics
MATH 350: Introduction to Computational Mathematics Chapter V: Least Squares Problems Greg Fasshauer Department of Applied Mathematics Illinois Institute of Technology Spring 2011 fasshauer@iit.edu MATH
More informationIntroduction. Chapter One
Chapter One Introduction The aim of this book is to describe and explain the beautiful mathematical relationships between matrices, moments, orthogonal polynomials, quadrature rules and the Lanczos and
More informationA Trust-region-based Sequential Quadratic Programming Algorithm
Downloaded from orbit.dtu.dk on: Oct 19, 2018 A Trust-region-based Sequential Quadratic Programming Algorithm Henriksen, Lars Christian; Poulsen, Niels Kjølstad Publication date: 2010 Document Version
More informationEIGENVALUES AND SINGULAR VALUE DECOMPOSITION
APPENDIX B EIGENVALUES AND SINGULAR VALUE DECOMPOSITION B.1 LINEAR EQUATIONS AND INVERSES Problems of linear estimation can be written in terms of a linear matrix equation whose solution provides the required
More informationLinear Least-Squares Data Fitting
CHAPTER 6 Linear Least-Squares Data Fitting 61 Introduction Recall that in chapter 3 we were discussing linear systems of equations, written in shorthand in the form Ax = b In chapter 3, we just considered
More informationMATH 5720: Unconstrained Optimization Hung Phan, UMass Lowell September 13, 2018
MATH 57: Unconstrained Optimization Hung Phan, UMass Lowell September 13, 18 1 Global and Local Optima Let a function f : S R be defined on a set S R n Definition 1 (minimizers and maximizers) (i) x S
More informationGlobally Solving the Trust Region Subproblem Using Simple First-Order Methods
Globally Solving the Trust Region Subproblem Using Simple First-Order Methods Amir Beck and Yakov Vaisbourd October 2, 2017 Abstract We consider the trust region subproblem which is given by a minimization
More informationORIE 6326: Convex Optimization. Quasi-Newton Methods
ORIE 6326: Convex Optimization Quasi-Newton Methods Professor Udell Operations Research and Information Engineering Cornell April 10, 2017 Slides on steepest descent and analysis of Newton s method adapted
More informationKey words. first order methods, trust region subproblem, optimality conditions, global optimum
GLOBALLY SOLVING THE TRUST REGION SUBPROBLEM USING SIMPLE FIRST-ORDER METHODS AMIR BECK AND YAKOV VAISBOURD Abstract. We consider the trust region subproblem which is given by a minimization of a quadratic,
More informationA PRIMAL-DUAL TRUST REGION ALGORITHM FOR NONLINEAR OPTIMIZATION
Optimization Technical Report 02-09, October 2002, UW-Madison Computer Sciences Department. E. Michael Gertz 1 Philip E. Gill 2 A PRIMAL-DUAL TRUST REGION ALGORITHM FOR NONLINEAR OPTIMIZATION 7 October
More informationAlgorithm 873: LSTRS: MATLAB Software for Large-Scale Trust-Region Subproblems and Regularization
Algorithm 873: LSTRS: MATLAB Software for Large-Scale Trust-Region Subproblems and Regularization MARIELBA ROJAS Technical University of Denmark SANDRA A. SANTOS State University of Campinas and DANNY
More informationThere are six more problems on the next two pages
Math 435 bg & bu: Topics in linear algebra Summer 25 Final exam Wed., 8/3/5. Justify all your work to receive full credit. Name:. Let A 3 2 5 Find a permutation matrix P, a lower triangular matrix L with
More informationGLOBALLY CONVERGENT GAUSS-NEWTON METHODS
GLOBALLY CONVERGENT GAUSS-NEWTON METHODS by C. Fraley TECHNICAL REPORT No. 200 March 1991 Department of Statistics, GN-22 University of Washington Seattle, Washington 98195 USA Globally Convergent Gauss-Newton
More informationA PROJECTED HESSIAN GAUSS-NEWTON ALGORITHM FOR SOLVING SYSTEMS OF NONLINEAR EQUATIONS AND INEQUALITIES
IJMMS 25:6 2001) 397 409 PII. S0161171201002290 http://ijmms.hindawi.com Hindawi Publishing Corp. A PROJECTED HESSIAN GAUSS-NEWTON ALGORITHM FOR SOLVING SYSTEMS OF NONLINEAR EQUATIONS AND INEQUALITIES
More informationSecond Order Optimization Algorithms I
Second Order Optimization Algorithms I Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 7, 8, 9 and 10 1 The
More informationAn Inexact Sequential Quadratic Optimization Method for Nonlinear Optimization
An Inexact Sequential Quadratic Optimization Method for Nonlinear Optimization Frank E. Curtis, Lehigh University involving joint work with Travis Johnson, Northwestern University Daniel P. Robinson, Johns
More informationBasic Concepts in Linear Algebra
Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear
More informationIterative Methods for Smooth Objective Functions
Optimization Iterative Methods for Smooth Objective Functions Quadratic Objective Functions Stationary Iterative Methods (first/second order) Steepest Descent Method Landweber/Projected Landweber Methods
More informationOn the use of quadratic models in unconstrained minimization without derivatives 1. M.J.D. Powell
On the use of quadratic models in unconstrained minimization without derivatives 1 M.J.D. Powell Abstract: Quadratic approximations to the objective function provide a way of estimating first and second
More information