The Newton-Raphson Algorithm
|
|
- Camron Chapman
- 6 years ago
- Views:
Transcription
1 The Newton-Raphson Algorithm David Allen University of Kentucky January 31, 2013
2 1 The Newton-Raphson Algorithm The Newton-Raphson algorithm, also called Newton s method, is a method for finding the minimum or maximum of a function of one or more variables. It is named after named after Isaac Newton and Joseph Raphson. Back 2
3 Its use in statistics Statisticians often want to find parameter values that minimize an objective function such a residual sum of squares or a negative log likelihood function. As θ is a popular symbol for a generic parameter, θ is used here to represent the argument of an objective function. Newton s algorithm is for finding the value of θ that minimizes an objective function. Back 3
4 Synopsis The basic Newton s algorithm starts with a provisional value of θ. Then it 1. constructs a quadratic function with the same value, slope, and curvature as the objective function at the provisional value; 2. finds the value of θ that minimizes the quadratic function; and 3. resets the provisional value to this minimizing value. If all goes well, these steps are repeated until the provisional value converges to the minimizing value. Back 4
5 An example with one variable The next few slides demonstrate repeated applications of the steps above for a scalar θ. Back 5
6 The First Approximation The first approximation is with θ = Back 6
7 The Second Approximation The second approximation is with θ = Back 7
8 The Third Approximation The third approximation is with θ = Back 8
9 The Final Approximation The estimate of θ is Back 9
10 In Matrix Notation Let o(θ) be the objective function to be minimized. Its vector of first derivatives, called the gradient vector, is g(θ) = d d θ o(θ) Its matrix of second derivatives, called the Hessian matrix, is H(θ) = d2 d θ d θ t o(θ) Back 10
11 The quadratic approximation The quadratic approximation of o(θ) at θ = θ 0 in terms of the gradient vector and Hessian matrix is ô(θ) = o(θ 0 ) + g t (θ 0 )(θ θ 0 ) (θ θ 0) t H(θ 0 )(θ θ 0 ) Provided H(θ 0 ) is positive definite, the approximating quadratic function is minimized by θ = θ 0 H 1 (θ 0 )g(θ 0 ) Back 11
12 Implementation There may be problems with convergence in practice, so Newton s algorithm must be implemented with controls. Excellent discussions of Newton s algorithm are given in Dennis and Schnabel [1], Fletcher [2], Nocedal and Wright [4], and Gill, Murray, and Wright [3]. Back 12
13 Minimum or Maximum? By checking second derivatives Newton s algorithm provides a definitive check that a minimum, maximum, or saddle point of the objective function is found. Back 13
14 The Rosenbrock function is Rosenbrock s function 100( )2 + (1 1 ) 2. Rosenbrock s function is a frequently used test function for numerical optimization procedures. Even though it is a simple looking function of two variables, it has some gotchas. Back 14
15 An exercise Exercise 1.1. Write an R program to apply Newton s method to the Rosenbrock function. Do not use built in R functions except for solve. Run your program using different starting values and observe the results. Back 15
16 2 Least Squares In situations where the response observations are uncorrelated with equal variances, least squares is the preferred method of estimation. Let Y represent the th response observation and η (θ) its expected value. Here θ is a vector of parameters that is functionally independent of the variance. The residual sum of squares is n s(θ) = (Y η (θ)) 2 (1) =1 where n is the number of observations. The least squares estimate of θ is the value of θ that minimizes s(θ) (assuming the minimum exists). Back 16
17 Derivatives of the residual sum of squares The vector of first derivative of s(θ), called the gradient vector, is g(θ) = 2 n (Y η (θ)) d =1 d θ η (θ) (2) The matrix of second derivatives, called the Hessian matrix, is H(θ) = 2 n =1 d d θ η (θ) d d θ t η (θ) 2 n d 2 (Y η (θ)) d θ d θ t η (θ) =1 (3) Back 17
18 The quadratic approximation The quadratic approximation of s(θ) at θ = θ 0 in terms of the gradient vector and Hessian matrix is ŝ(θ) = s(θ 0 ) + g t (θ 0 )(θ θ 0 ) (θ θ 0) t H(θ 0 )(θ θ 0 ) Newton s algorithm, with the terms in H(θ) involving second derivatives omitted, is called the Gauss-Newton algorithm. Back 18
19 The minimizing value Provided H(θ 0 ) is positive definite, the approximating quadratic function is minimized by θ = θ 0 H 1 (θ 0 )g(θ 0 ) Back 19
20 Summary In the preceding, the objective function is the residual sum of squares. The chain rule of differentiation provides formulas needed to calculate quadratic approximation of the objective function in terms of derivatives d d θ η (θ) and d 2 d θ d θ t η (θ). When the η (θ) are components of a solution of linear differential equations, the partial derivatives can be calculated by a computer. In the case of other objective functions, a similar process must be followed i.e. use the chain rule to find d expressions for g(θ) and H(θ) in terms of d θ η (θ) and d 2 η d θ d θ t (θ). Unfortunately, this is sometimes difficult. Back 20
21 References [1] J. E. Dennis, Jr. and Robert B. Schnabel. Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Prentice-Hall, Inc., Englewood Cliffs, New Jersey 07632, [2] Roger Fletcher. Practical methods of optimization, volume 1, Unconstrained optimization. John Wiley & and Sons, Ltd., [3] Philip E. Gill, Walter Murray, and Margaret H. Wright. Practical Optimization. Academic Press, Inc., [4] Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer-Verlag New York, Inc., Back 21
Optimization: an Overview
Optimization: an Overview Moritz Diehl University of Freiburg and University of Leuven (some slide material was provided by W. Bangerth and K. Mombaur) Overview of presentation Optimization: basic definitions
More informationGLOBALLY CONVERGENT GAUSS-NEWTON METHODS
GLOBALLY CONVERGENT GAUSS-NEWTON METHODS by C. Fraley TECHNICAL REPORT No. 200 March 1991 Department of Statistics, GN-22 University of Washington Seattle, Washington 98195 USA Globally Convergent Gauss-Newton
More informationStatistics 580 Optimization Methods
Statistics 580 Optimization Methods Introduction Let fx be a given real-valued function on R p. The general optimization problem is to find an x ɛ R p at which fx attain a maximum or a minimum. It is of
More informationGradient-Based Optimization
Multidisciplinary Design Optimization 48 Chapter 3 Gradient-Based Optimization 3. Introduction In Chapter we described methods to minimize (or at least decrease) a function of one variable. While problems
More informationA NOTE ON PAN S SECOND-ORDER QUASI-NEWTON UPDATES
A NOTE ON PAN S SECOND-ORDER QUASI-NEWTON UPDATES Lei-Hong Zhang, Ping-Qi Pan Department of Mathematics, Southeast University, Nanjing, 210096, P.R.China. Abstract This note, attempts to further Pan s
More informationOptimization Methods
Optimization Methods Decision making Examples: determining which ingredients and in what quantities to add to a mixture being made so that it will meet specifications on its composition allocating available
More informationNumerical Optimization: Basic Concepts and Algorithms
May 27th 2015 Numerical Optimization: Basic Concepts and Algorithms R. Duvigneau R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 1 Outline Some basic concepts in optimization Some
More informationHandout on Newton s Method for Systems
Handout on Newton s Method for Systems The following summarizes the main points of our class discussion of Newton s method for approximately solving a system of nonlinear equations F (x) = 0, F : IR n
More informationEstimation, Detection, and Identification
Estimation, Detection, and Identification Graduate Course on the CMU/Portugal ECE PhD Program Spring 2008/2009 Chapter 5 Best Linear Unbiased Estimators Instructor: Prof. Paulo Jorge Oliveira pjcro @ isr.ist.utl.pt
More informationCondensed Table of Contents for Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control by J. C.
Condensed Table of Contents for Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control by J. C. Spall John Wiley and Sons, Inc., 2003 Preface... xiii 1. Stochastic Search
More informationA NOTE ON Q-ORDER OF CONVERGENCE
BIT 0006-3835/01/4102-0422 $16.00 2001, Vol. 41, No. 2, pp. 422 429 c Swets & Zeitlinger A NOTE ON Q-ORDER OF CONVERGENCE L. O. JAY Department of Mathematics, The University of Iowa, 14 MacLean Hall Iowa
More informationj=1 r 1 x 1 x n. r m r j (x) r j r j (x) r j (x). r j x k
Maria Cameron Nonlinear Least Squares Problem The nonlinear least squares problem arises when one needs to find optimal set of parameters for a nonlinear model given a large set of data The variables x,,
More informationSingle Variable Minimization
AA222: MDO 37 Sunday 1 st April, 2012 at 19:48 Chapter 2 Single Variable Minimization 2.1 Motivation Most practical optimization problems involve many variables, so the study of single variable minimization
More informationHow do we recognize a solution?
AMSC 607 / CMSC 764 Advanced Numerical Optimization Fall 2010 UNIT 2: Unconstrained Optimization, Part 1 Dianne P. O Leary c 2008,2010 The plan: Unconstrained Optimization: Fundamentals How do we recognize
More information446 CHAP. 8 NUMERICAL OPTIMIZATION. Newton's Search for a Minimum of f(x,y) Newton s Method
446 CHAP. 8 NUMERICAL OPTIMIZATION Newton's Search for a Minimum of f(xy) Newton s Method The quadratic approximation method of Section 8.1 generated a sequence of seconddegree Lagrange polynomials. It
More informationMaximum Likelihood Estimation
Maximum Likelihood Estimation Prof. C. F. Jeff Wu ISyE 8813 Section 1 Motivation What is parameter estimation? A modeler proposes a model M(θ) for explaining some observed phenomenon θ are the parameters
More informationNumerical Optimization of Partial Differential Equations
Numerical Optimization of Partial Differential Equations Part I: basic optimization concepts in R n Bartosz Protas Department of Mathematics & Statistics McMaster University, Hamilton, Ontario, Canada
More informationThe Squared Slacks Transformation in Nonlinear Programming
Technical Report No. n + P. Armand D. Orban The Squared Slacks Transformation in Nonlinear Programming August 29, 2007 Abstract. We recall the use of squared slacks used to transform inequality constraints
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods
More informationp 1 p 0 (p 1, f(p 1 )) (p 0, f(p 0 )) The geometric construction of p 2 for the se- cant method.
80 CHAP. 2 SOLUTION OF NONLINEAR EQUATIONS f (x) = 0 y y = f(x) (p, 0) p 2 p 1 p 0 x (p 1, f(p 1 )) (p 0, f(p 0 )) The geometric construction of p 2 for the se- Figure 2.16 cant method. Secant Method The
More informationBasic concepts in Linear Algebra and Optimization
Basic concepts in Linear Algebra and Optimization Yinbin Ma GEOPHYS 211 Outline Basic Concepts on Linear Algbra vector space norm linear mapping, range, null space matrix multiplication terative Methods
More informationMath 409/509 (Spring 2011)
Math 409/509 (Spring 2011) Instructor: Emre Mengi Study Guide for Homework 2 This homework concerns the root-finding problem and line-search algorithms for unconstrained optimization. Please don t hesitate
More informationOptimization 2. CS5240 Theoretical Foundations in Multimedia. Leow Wee Kheng
Optimization 2 CS5240 Theoretical Foundations in Multimedia Leow Wee Kheng Department of Computer Science School of Computing National University of Singapore Leow Wee Kheng (NUS) Optimization 2 1 / 38
More information2. Quasi-Newton methods
L. Vandenberghe EE236C (Spring 2016) 2. Quasi-Newton methods variable metric methods quasi-newton methods BFGS update limited-memory quasi-newton methods 2-1 Newton method for unconstrained minimization
More informationGeneralization Of The Secant Method For Nonlinear Equations
Applied Mathematics E-Notes, 8(2008), 115-123 c ISSN 1607-2510 Available free at mirror sites of http://www.math.nthu.edu.tw/ amen/ Generalization Of The Secant Method For Nonlinear Equations Avram Sidi
More informationLine Search Techniques
Multidisciplinary Design Optimization 33 Chapter 2 Line Search Techniques 2.1 Introduction Most practical optimization problems involve many variables, so the study of single variable minimization may
More informationOptimization Tutorial 1. Basic Gradient Descent
E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.
More informationConstrained Optimization Theory
Constrained Optimization Theory Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Constrained Optimization Theory IMA, August
More informationImproved Damped Quasi-Newton Methods for Unconstrained Optimization
Improved Damped Quasi-Newton Methods for Unconstrained Optimization Mehiddin Al-Baali and Lucio Grandinetti August 2015 Abstract Recently, Al-Baali (2014) has extended the damped-technique in the modified
More informationOptimization Methods
Optimization Methods Categorization of Optimization Problems Continuous Optimization Discrete Optimization Combinatorial Optimization Variational Optimization Common Optimization Concepts in Computer Vision
More informationA Likelihood Ratio Test
A Likelihood Ratio Test David Allen University of Kentucky February 23, 2012 1 Introduction Earlier presentations gave a procedure for finding an estimate and its standard error of a single linear combination
More informationConstrained optimization. Unconstrained optimization. One-dimensional. Multi-dimensional. Newton with equality constraints. Active-set method.
Optimization Unconstrained optimization One-dimensional Multi-dimensional Newton s method Basic Newton Gauss- Newton Quasi- Newton Descent methods Gradient descent Conjugate gradient Constrained optimization
More informationENSIEEHT-IRIT, 2, rue Camichel, Toulouse (France) LMS SAMTECH, A Siemens Business,15-16, Lower Park Row, BS1 5BN Bristol (UK)
Quasi-Newton updates with weighted secant equations by. Gratton, V. Malmedy and Ph. L. oint Report NAXY-09-203 6 October 203 0.5 0 0.5 0.5 0 0.5 ENIEEH-IRI, 2, rue Camichel, 3000 oulouse France LM AMECH,
More informationComparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming Problems
International Journal of Scientific and Research Publications, Volume 3, Issue 10, October 013 1 ISSN 50-3153 Comparative study of Optimization methods for Unconstrained Multivariable Nonlinear Programming
More informationA SUFFICIENTLY EXACT INEXACT NEWTON STEP BASED ON REUSING MATRIX INFORMATION
A SUFFICIENTLY EXACT INEXACT NEWTON STEP BASED ON REUSING MATRIX INFORMATION Anders FORSGREN Technical Report TRITA-MAT-2009-OS7 Department of Mathematics Royal Institute of Technology November 2009 Abstract
More informationSpectral gradient projection method for solving nonlinear monotone equations
Journal of Computational and Applied Mathematics 196 (2006) 478 484 www.elsevier.com/locate/cam Spectral gradient projection method for solving nonlinear monotone equations Li Zhang, Weijun Zhou Department
More informationECON 5350 Class Notes Nonlinear Regression Models
ECON 5350 Class Notes Nonlinear Regression Models 1 Introduction In this section, we examine regression models that are nonlinear in the parameters and give a brief overview of methods to estimate such
More informationTrust Regions. Charles J. Geyer. March 27, 2013
Trust Regions Charles J. Geyer March 27, 2013 1 Trust Region Theory We follow Nocedal and Wright (1999, Chapter 4), using their notation. Fletcher (1987, Section 5.1) discusses the same algorithm, but
More informationOPTIMAL DESIGN INPUTS FOR EXPERIMENTAL CHAPTER 17. Organization of chapter in ISSO. Background. Linear models
CHAPTER 17 Slides for Introduction to Stochastic Search and Optimization (ISSO)by J. C. Spall OPTIMAL DESIGN FOR EXPERIMENTAL INPUTS Organization of chapter in ISSO Background Motivation Finite sample
More informationA DIMENSION REDUCING CONIC METHOD FOR UNCONSTRAINED OPTIMIZATION
1 A DIMENSION REDUCING CONIC METHOD FOR UNCONSTRAINED OPTIMIZATION G E MANOUSSAKIS, T N GRAPSA and C A BOTSARIS Department of Mathematics, University of Patras, GR 26110 Patras, Greece e-mail :gemini@mathupatrasgr,
More informationStochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions
International Journal of Control Vol. 00, No. 00, January 2007, 1 10 Stochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions I-JENG WANG and JAMES C.
More information"ZERO-POINT" IN THE EVALUATION OF MARTENS HARDNESS UNCERTAINTY
"ZERO-POINT" IN THE EVALUATION OF MARTENS HARDNESS UNCERTAINTY Professor Giulio Barbato, PhD Student Gabriele Brondino, Researcher Maurizio Galetto, Professor Grazia Vicario Politecnico di Torino Abstract
More informationModule 10: Finite Difference Methods for Boundary Value Problems Lecture 42: Special Boundary Value Problems. The Lecture Contains:
The Lecture Contains: Finite Difference Method Boundary conditions of the second and third kind Solution of the difference scheme: Linear case Solution of the difference scheme: nonlinear case Problems
More informationNew implicit method for analysis of problems in nonlinear structural dynamics
Applied and Computational Mechanics 5 (2011) 15 20 New implicit method for analysis of problems in nonlinear structural dynamics A. A. Gholampour a,, M. Ghassemieh a a School of Civil Engineering, University
More information1 Numerical optimization
Contents 1 Numerical optimization 5 1.1 Optimization of single-variable functions............ 5 1.1.1 Golden Section Search................... 6 1.1. Fibonacci Search...................... 8 1. Algorithms
More informationLecture 3 Optimization methods for econometrics models
Lecture 3 Optimization methods for econometrics models Cinzia Cirillo University of Maryland Department of Civil and Environmental Engineering 06/29/2016 Summer Seminar June 27-30, 2016 Zinal, CH 1 Overview
More informationLinear Mixed Models: Methodology and Algorithms
Linear Mixed Models: Methodology and Algorithms David M. Allen University of Kentucky March 6, 2017 C Topics from Calculus Maximum likelihood and REML estimation involve the minimization of a negative
More informationIntroduction to Maximum Likelihood Estimation
Introduction to Maximum Likelihood Estimation Eric Zivot July 26, 2012 The Likelihood Function Let 1 be an iid sample with pdf ( ; ) where is a ( 1) vector of parameters that characterize ( ; ) Example:
More informationOn Lagrange multipliers of trust-region subproblems
On Lagrange multipliers of trust-region subproblems Ladislav Lukšan, Ctirad Matonoha, Jan Vlček Institute of Computer Science AS CR, Prague Programy a algoritmy numerické matematiky 14 1.- 6. června 2008
More informationIntroduction to Unconstrained Optimization: Part 2
Introduction to Unconstrained Optimization: Part 2 James Allison ME 555 January 29, 2007 Overview Recap Recap selected concepts from last time (with examples) Use of quadratic functions Tests for positive
More informationDiscussion of Maximization by Parts in Likelihood Inference
Discussion of Maximization by Parts in Likelihood Inference David Ruppert School of Operations Research & Industrial Engineering, 225 Rhodes Hall, Cornell University, Ithaca, NY 4853 email: dr24@cornell.edu
More informationMaria Cameron. f(x) = 1 n
Maria Cameron 1. Local algorithms for solving nonlinear equations Here we discuss local methods for nonlinear equations r(x) =. These methods are Newton, inexact Newton and quasi-newton. We will show that
More informationLecture 16 Solving GLMs via IRWLS
Lecture 16 Solving GLMs via IRWLS 09 November 2015 Taylor B. Arnold Yale Statistics STAT 312/612 Notes problem set 5 posted; due next class problem set 6, November 18th Goals for today fixed PCA example
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationNew Inexact Line Search Method for Unconstrained Optimization 1,2
journal of optimization theory and applications: Vol. 127, No. 2, pp. 425 446, November 2005 ( 2005) DOI: 10.1007/s10957-005-6553-6 New Inexact Line Search Method for Unconstrained Optimization 1,2 Z.
More informationQuasi-Newton Methods
Quasi-Newton Methods Werner C. Rheinboldt These are excerpts of material relating to the boos [OR00 and [Rhe98 and of write-ups prepared for courses held at the University of Pittsburgh. Some further references
More informationIntroduction to gradient descent
6-1: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction to gradient descent Derivation and intuitions Hessian 6-2: Introduction to gradient descent Prof. J.C. Kao, UCLA Introduction Our
More informationDynamics and Control Preliminary Examination Topics
Dynamics and Control Preliminary Examination Topics 1. Particle and Rigid Body Dynamics Meirovitch, Leonard; Methods of Analytical Dynamics, McGraw-Hill, Inc New York, NY, 1970 Chapters 1-5 2. Atmospheric
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2
More informationScientific Computing: Optimization
Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture
More informationSeminal papers in nonlinear optimization
Seminal papers in nonlinear optimization Nick Gould, CSED, RAL, Chilton, OX11 0QX, England (n.gould@rl.ac.uk) December 7, 2006 The following papers are classics in the field. Although many of them cover
More informationChapter 10 Conjugate Direction Methods
Chapter 10 Conjugate Direction Methods An Introduction to Optimization Spring, 2012 1 Wei-Ta Chu 2012/4/13 Introduction Conjugate direction methods can be viewed as being intermediate between the method
More informationLecture 10. Neural networks and optimization. Machine Learning and Data Mining November Nando de Freitas UBC. Nonlinear Supervised Learning
Lecture 0 Neural networks and optimization Machine Learning and Data Mining November 2009 UBC Gradient Searching for a good solution can be interpreted as looking for a minimum of some error (loss) function
More informationApplied Mathematics 205. Unit I: Data Fitting. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit I: Data Fitting Lecturer: Dr. David Knezevic Unit I: Data Fitting Chapter I.4: Nonlinear Least Squares 2 / 25 Nonlinear Least Squares So far we have looked at finding a best
More informationLINEAR MODELS FOR CLASSIFICATION. J. Elder CSE 6390/PSYC 6225 Computational Modeling of Visual Perception
LINEAR MODELS FOR CLASSIFICATION Classification: Problem Statement 2 In regression, we are modeling the relationship between a continuous input variable x and a continuous target variable t. In classification,
More informationMatrix Differential Calculus with Applications in Statistics and Econometrics
Matrix Differential Calculus with Applications in Statistics and Econometrics Revised Edition JAN. R. MAGNUS CentERjor Economic Research, Tilburg University and HEINZ NEUDECKER Cesaro, Schagen JOHN WILEY
More informationIntroduction to unconstrained optimization - direct search methods
Introduction to unconstrained optimization - direct search methods Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Structure of optimization methods Typically Constraint handling converts the
More informationTime delay system identification based on optimization approaches
Time delay system identification based on optimization approaches Ahlem SASSI, Saïda BEDOUI, Kamel ABDERRAHIM Numerical Control of Industrial Processes, National School of Engineers of Gabes, University
More informationApplied Nonlinear Control Slotine Solution Manual Solesa
We have made it easy for you to find a PDF Ebooks without any digging. And by having access to our ebooks online or by storing it on your computer, you have convenient answers with applied nonlinear control
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization Instructor: Michael Saunders Spring 2017 Notes 10: MINOS Part 1: the Reduced-Gradient
More informationMathematical optimization
Optimization Mathematical optimization Determine the best solutions to certain mathematically defined problems that are under constrained determine optimality criteria determine the convergence of the
More informationA Term Project report on. State Estimation of Power Systems Using Weighted Least Squares Optimization Technique
MA5630 Numerical Optimization Course Instructor: Prof. Allan A. Struthers A Term Project report on State Estimation of Power Systems Using Weighted Least Squares Optimization Technique Submitted by Alorf,
More informationOn Application Of Modified Gradient To Extended Conjugate Gradient Method Algorithm For Solving Optimal Control Problems
IOSR Journal of Mathematics (IOSR-JM) e-issn: 2278-5728, p-issn:2319-765x. Volume 9, Issue 5 (Jan. 2014), PP 30-35 On Application Of Modified Gradient To Extended Conjugate Gradient Method Algorithm For
More informationChapter 6: Derivative-Based. optimization 1
Chapter 6: Derivative-Based Optimization Introduction (6. Descent Methods (6. he Method of Steepest Descent (6.3 Newton s Methods (NM (6.4 Step Size Determination (6.5 Nonlinear Least-Squares Problems
More informationNonlinear Coordinate Transformations for Unconstrained Optimization I. Basic Transformations
Nonlinear Coordinate Transformations for Unconstrained Optimization I. Basic Transformations TIBOR CSENDES Kalmár Laboratory, József Attila University, Szeged, Hungary and TAMÁS RAPCSÁK Computer and Automation
More informationOptimization Concepts and Applications in Engineering
Optimization Concepts and Applications in Engineering Ashok D. Belegundu, Ph.D. Department of Mechanical Engineering The Pennsylvania State University University Park, Pennsylvania Tirupathi R. Chandrupatia,
More information1 Numerical optimization
Contents Numerical optimization 5. Optimization of single-variable functions.............................. 5.. Golden Section Search..................................... 6.. Fibonacci Search........................................
More informationMath (P)refresher Lecture 8: Unconstrained Optimization
Math (P)refresher Lecture 8: Unconstrained Optimization September 2006 Today s Topics : Quadratic Forms Definiteness of Quadratic Forms Maxima and Minima in R n First Order Conditions Second Order Conditions
More informationOptimization II: Unconstrained Multivariable
Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Optimization II: Unconstrained
More informationOPER 627: Nonlinear Optimization Lecture 14: Mid-term Review
OPER 627: Nonlinear Optimization Lecture 14: Mid-term Review Department of Statistical Sciences and Operations Research Virginia Commonwealth University Oct 16, 2013 (Lecture 14) Nonlinear Optimization
More informationLOGISTIC REGRESSION Joseph M. Hilbe
LOGISTIC REGRESSION Joseph M. Hilbe Arizona State University Logistic regression is the most common method used to model binary response data. When the response is binary, it typically takes the form of
More informationLine Search Methods. Shefali Kulkarni-Thaker
1 BISECTION METHOD Line Search Methods Shefali Kulkarni-Thaker Consider the following unconstrained optimization problem min f(x) x R Any optimization algorithm starts by an initial point x 0 and performs
More informationIPAM Summer School Optimization methods for machine learning. Jorge Nocedal
IPAM Summer School 2012 Tutorial on Optimization methods for machine learning Jorge Nocedal Northwestern University Overview 1. We discuss some characteristics of optimization problems arising in deep
More informationLecture 7: CS395T Numerical Optimization for Graphics and AI Trust Region Methods
Lecture 7: CS395T Numerical Optimization for Graphics and AI Trust Region Methods Qixing Huang The University of Texas at Austin huangqx@cs.utexas.edu 1 Disclaimer This note is adapted from Section 4 of
More informationMultipoint secant and interpolation methods with nonmonotone line search for solving systems of nonlinear equations
Multipoint secant and interpolation methods with nonmonotone line search for solving systems of nonlinear equations Oleg Burdakov a,, Ahmad Kamandi b a Department of Mathematics, Linköping University,
More informationLecture 3 September 1
STAT 383C: Statistical Modeling I Fall 2016 Lecture 3 September 1 Lecturer: Purnamrita Sarkar Scribe: Giorgio Paulon, Carlos Zanini Disclaimer: These scribe notes have been slightly proofread and may have
More informationImproving L-BFGS Initialization for Trust-Region Methods in Deep Learning
Improving L-BFGS Initialization for Trust-Region Methods in Deep Learning Jacob Rafati http://rafati.net jrafatiheravi@ucmerced.edu Ph.D. Candidate, Electrical Engineering and Computer Science University
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationNumerical computation II. Reprojection error Bundle adjustment Family of Newtonʼs methods Statistical background Maximum likelihood estimation
Numerical computation II Reprojection error Bundle adjustment Family of Newtonʼs methods Statistical background Maximum likelihood estimation Reprojection error Reprojection error = Distance between the
More informationAlgorithms for nonlinear programming problems II
Algorithms for nonlinear programming problems II Martin Branda Charles University Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Computational Aspects of Optimization
More informationNear-Optimal Discretization of the Brachistochrone Problem
Near-Optimal iscretization of the Brachistochrone Problem Jon attorro.5.45.4.35.3.5..5..5.5..5..5.3.35.4.45.5 A project in partial fulfillment of the requirements for EESOR34 Optimization Algorithms ecember
More informationReduced-Hessian Methods for Constrained Optimization
Reduced-Hessian Methods for Constrained Optimization Philip E. Gill University of California, San Diego Joint work with: Michael Ferry & Elizabeth Wong 11th US & Mexico Workshop on Optimization and its
More informationSemester 3 MULTIVARIATE CALCULUS AND INTEGRAL TRANSFORMS
PC 11 Semester 3 MT03C11 MULTIVARIATE CALCULUS AND INTEGRAL TRANSFORMS Text 1: Tom APOSTOL, Mathematical Analysis, Second edition, Narosa Publishing House. Text 2: WALTER RUDIN, Principles of Mathematical
More informationSampling distribution of GLM regression coefficients
Sampling distribution of GLM regression coefficients Patrick Breheny February 5 Patrick Breheny BST 760: Advanced Regression 1/20 Introduction So far, we ve discussed the basic properties of the score,
More informationKKT Conditions for Rank-Deficient Nonlinear Least-Square Problems with Rank-Deficient Nonlinear Constraints1
JOURNAL OF OPTIMIZATION THEORY AND APPLICATIONS: Vol. 100, No. 1. pp. 145-160, JANUARY 1999 KKT Conditions for Rank-Deficient Nonlinear Least-Square Problems with Rank-Deficient Nonlinear Constraints1
More informationMark Gales October y (x) x 1. x 2 y (x) Inputs. Outputs. x d. y (x) Second Output layer layer. layer.
University of Cambridge Engineering Part IIB & EIST Part II Paper I0: Advanced Pattern Processing Handouts 4 & 5: Multi-Layer Perceptron: Introduction and Training x y (x) Inputs x 2 y (x) 2 Outputs x
More informationDepartment of Mathematics Faculty of Natural Science, Jamia Millia Islamia, New Delhi-25
Department of Mathematics Faculty of Natural Science, Jamia Millia Islamia, New Delhi-25 Course Structure of U.G. under CBCS* (Only for those students who have not taken mathematics as a core or subsidiary
More informationEECE Adaptive Control
EECE 574 - Adaptive Control Basics of System Identification Guy Dumont Department of Electrical and Computer Engineering University of British Columbia January 2010 Guy Dumont (UBC) EECE574 - Basics of
More informationSTAT Advanced Bayesian Inference
1 / 8 STAT 625 - Advanced Bayesian Inference Meng Li Department of Statistics March 5, 2018 Distributional approximations 2 / 8 Distributional approximations are useful for quick inferences, as starting
More informationA choice of norm in discrete approximation
147 A choice of norm in discrete approximation Tomislav Marošević Abstract. We consider the problem of choice of norms in discrete approximation. First, we describe properties of the standard l 1, l 2
More information