Applied Computational Economics Workshop. Part 3: Nonlinear Equations

Size: px
Start display at page:

Download "Applied Computational Economics Workshop. Part 3: Nonlinear Equations"

Transcription

1 Applied Computational Economics Workshop Part 3: Nonlinear Equations 1

2 Overview Introduction Function iteration Newton s method Quasi-Newton methods Practical example Practical issues 2

3 Introduction Nonlinear equations take one of two forms Root-finding problem Given a function : R R, compute an -vector, called a root of, such that Fixed-point problem ( )=0 Given a function : R R, compute an -vector, called a fixed-point of, such that ( )= 3

4 Introduction The two forms of nonlinear equations are equivalent A root of is a fixed-point of A fixed-point of is root of 4

5 Introduction Nonlinear equations arise naturally in economics multi-commodity market equilibrium models multi-person static game models unconstrained optimization models Nonlinear equations also arise indirectly when numerically solving economic models involving functional equations dynamic optimization models rational expectations models arbitrage pricing models 5

6 Function Iteration Function iteration is an algorithm for computing a fixed-point of a function Guess an initial value and successively form the iterates until the iterates converge 6

7 Computing Fixed-Point of Iteration Using Function 7

8 Function Iteration: Example To compute the fixed-point of ( ) = employs the iteration rule +0.2using function iteration, one = ( ) = +0.2 In MATLAB x = 0.4; for it=1:50 xold = x x = sqrt(x+0.2); if abs(x-xold)<1.e-10, break, end end After 28 iterations, x converges to

9 Function Iteration When is function iteration guaranteed to converge? By the Contraction Mapping Theorem, if for some, for all and, then possesses an unique fixed-point and function iteration will converge to it from any initial value 9

10 Newton s Method Newton s method is an algorithm for computing a root of a function Guess an initial value and successively form the iterates until the iterates converge 10

11 Newton s Method: Example To compute the root of ( ) = 2using Newton s method, one employs the iteration rule = = 2 4 In MATLAB x = 2.3; for it=1:50 s = -(x^4-2)/(4*x^3); x = x + s; if abs(s)<1.e-10, break, end end After 8 iterations, converges to

12 Newton s Method Newton s method employs a strategy of successive linearization The strategy calls for the nonlinear function to be approximated by a sequence of linear functions whose roots are easily computed and, ideally, converge to the root of In particular, the +1 iterate = ( ) is the root of the Taylor linear approximation of around the preceding iterate, viz. ( )+ ( )( ) 12

13 Computing Root of Method Using Newton s 13

14 Newton s Method When is Newton s method guaranteed to converge? In theory, Newton s method will converge if the initial value is close to a root of at which is non-singular Theory, however, provides no practical definition of close Moreover, in practice, Newton s method will fail to converge to if is ill-conditioned, i.e. nearly singular 14

15 Quasi-Newton Methods Quasi-Newton methods replace the Jacobian in Newton s method with an estimate that is easier to compute Specifically, quasi-newton methods use an iteration rule where is an estimate of the Jacobian 15

16 Secant Method The quasi-newton method for univariate root-finding problems is called the secant method The secant method replaces the derivative in Newton s method with the estimate which leads to the iteration rule, = ( ) The secant method is so called because it approximates the function using secant lines drawn through successive pairs of points on its graph 16

17 Computing Root of Using Secant Method 17

18 Broyden s Method Broyden s method is the most popular multivariate generalization of the univariate secant method Broyden s method replaces the Jacobian in Newton s method with an estimate that is updated by making the smallest possible change that is consistent with the secant condition This yields a complicated iteration rule that is omitted here, but which may found in many textbooks, including Miranda and Fackler (2002) 18

19 Quasi-Newton Methods When are quasi-newton methods guaranteed to converge? In theory, a quasi-newton (e.g. Broyden s) method will converge if the initial value is close to a root of at which the Jacobian is non-singular and if the initial Jacobian estimate is close to Theory, however, provides no practical definition of close Moreover, in practice, Broyden s method will fail to converge to if the Jacobian estimates become illconditioned, i.e. nearly singular 19

20 Numerical Examples The CompEcon Toolbox provides two utilities for computing the root of function Utility newton uses Newton s method Utility broyden uses Broyden s method 20

21 Numerical Examples The calling protocol for newton is Input [x,fval] = newton(f,x,varargin) f x varargin Output x fval function of form [fval,fjac]=f(x,varargin) where fval and fjac are the function value and the Jacobian value at x, respectively initial guess for a root of f additional arguments for f (optional) a root of f value of f at x 21

22 Numerical Examples The calling protocol for broyden is Input f x [x,fval] = broyden(f,x,varargin) function of form fval=f(x,varargin) where fval is the function value at x initial guess for a root of f varargin additional arguments for f (optional) Output x fval a root of f value of f at x 22

23 Numerical Examples Let us compute the root of a function : R R given by ( ) = exp( ) 2 To use broyden, first compose a MATLAB function function fval = f(x) fval = [x(2)*exp(x(1))-2*x(2);x(1)*x(2)-x(2)^3]; and save it as f.m Then, on the MATLAB command screen, execute x = [1.0;0.5]; optset('broyden','showiters',1); [x,fval] = broyden(@f,x) After 11 iterations, this produces x=(0.6931,0.8326) 23

24 Numerical Examples To use newton, edit f.m so that it computes the analytic Jacobian, viz. function [fval,fjac] = f(x) fval = [x(2)*exp(x(1))-2*x(2);x(1)*x(2)-x(2)^3]; fjac = [x(2)*exp(x(1)) exp(x(1)) 2; x(2) x(1)-3*x(2)^2]; On the MATLAB command screen, execute x = [1.0;0.5]; checkjac(@f,x) This checks the internal consistency of your function file by comparing the analytic derivative to a numerical derivative 24

25 Numerical Examples Then, on the MATLAB command screen, execute x = [1.0;0.5]; optset('newton','showiters',1); [x,fval] = newton(@f,x) After 5 iterations, this produces x=(0.6931,0.8326) 25

26 Convergence Path for Newton and Broyden Methods 26

27 Numerical Examples (Cournot Duopoly) Consider a market with two firms producing the same good Firm s total cost of production is a function of the quantity it produces. The market clearing price is a function of the total quantity produced by both firms 27

28 Numerical Examples (Cournot Duopoly) Firm chooses production so as to maximize its profit taking the other firm s output as given Thus, in equilibrium, for 28

29 Numerical Examples (Cournot Duopoly) Suppose,, and To compute the equilibrium using Broyden s method, open the file exampcournot.m and uncomment alpha = 0.6; beta = [ ]; P Pder f P(q)+Pder(q)*q(2)-beta(2)*q(2)]; Here, f computes the marginal profits of both firms 29

30 Numerical Examples (Cournot Duopoly) Then type the following and execute q = [0.2;0.2]; optset('broyden','showiters',1); q = broyden(f,q) After 10 iterations, converges to 30

31 Practical Issues Failure to converge Execution speed Choosing a solution method 31

32 Failure to Converge In practice, nonlinear equation algorithms can fail to converge for various reasons human error bad initial value ill-conditioning 32

33 Failure to Converge: Human Error Math errors: analyst incorrectly derives function or Jacobian Coding errors: analyst incorrectly codes function or Jacobian Coding errors are less likely with function iteration and Broyden s method because they are derivative-free 33

34 Failure to Converge: Bad Initial Value Nonlinear equation algorithms require initial values If initial value is far from desired root, algorithm can diverge or converge to a wrong root Theory provides no guidance on how to specify initial value Analyst must supply good guess from knowledge of model If algorithm diverges, try another initial value Well behaved functions are more robust to initial value Poorly behaved functions are more sensitive to initial value 34

35 Failure to Converge: Ill-Conditioning Computing iteration step in Newton s and Broyden s methods requires solution to a linear equation involving the Jacobian or its estimate If the Jacobian or estimate is ill-conditioned near solution, the iteration step cannot be accurately computed Very little can be done about this It arises more often than we like 35

36 Execution Speed Two factors determine the speed with which a properly coded and initiated algorithm will converge to a solution asymptotic rate of convergence computational effort per iteration 36

37 Execution Speed The number of iterations required for a properly coded and initiated algorithm to converge is closely tied to its theoretical asymptotic rate of convergence Function iteration converges at a linear rate relatively slow Broyden s method converges at a superlinear rate relatively fast Newton s method converges at a quadratic rate extremely fast 37

38 Rate of Convergence When Computing Fixed-Point of Using Various Methods, 38

39 Execution Speed However, algorithms differ in computations per iteration function iteration requires a function evaluation Broyden s method additionally requires a linear solve Newton s method additionally requires a Jacobian evaluation Thus, a faster rate of convergence typically can be achieved only by investing greater computational effort per iteration The optimal tradeoff between rate of convergence and computational effort per iteration varies across applications 39

40 Choosing a Solution Method Concerns about execution speed, however, are exaggerated The time that must be invested by the analyst to write and debug code typically is far more important Derivative-free methods such as function iteration and Broyden s method can be implemented faster in real time and more reliably than Newton s method Newton s method should be used only if dimension is low or derivatives are simple other methods have failed to converge, or general purpose, re-usable code is needed 40

Solving Nonlinear Equations

Solving Nonlinear Equations Solving Nonlinear Equations Jijian Fan Department of Economics University of California, Santa Cruz Oct 13 2014 Overview NUMERICALLY solving nonlinear equation Four methods Bisection Function iteration

More information

MATH 3795 Lecture 13. Numerical Solution of Nonlinear Equations in R N.

MATH 3795 Lecture 13. Numerical Solution of Nonlinear Equations in R N. MATH 3795 Lecture 13. Numerical Solution of Nonlinear Equations in R N. Dmitriy Leykekhman Fall 2008 Goals Learn about different methods for the solution of F (x) = 0, their advantages and disadvantages.

More information

Nonlinear equations can take one of two forms. In the nonlinear root nding n n

Nonlinear equations can take one of two forms. In the nonlinear root nding n n Chapter 3 Nonlinear Equations Nonlinear equations can take one of two forms. In the nonlinear root nding n n problem, a function f from < to < is given, and one must compute an n-vector x, called a root

More information

Lecture Notes to Accompany. Scientific Computing An Introductory Survey. by Michael T. Heath. Chapter 5. Nonlinear Equations

Lecture Notes to Accompany. Scientific Computing An Introductory Survey. by Michael T. Heath. Chapter 5. Nonlinear Equations Lecture Notes to Accompany Scientific Computing An Introductory Survey Second Edition by Michael T Heath Chapter 5 Nonlinear Equations Copyright c 2001 Reproduction permitted only for noncommercial, educational

More information

CS 450 Numerical Analysis. Chapter 5: Nonlinear Equations

CS 450 Numerical Analysis. Chapter 5: Nonlinear Equations Lecture slides based on the textbook Scientific Computing: An Introductory Survey by Michael T. Heath, copyright c 2018 by the Society for Industrial and Applied Mathematics. http://www.siam.org/books/cl80

More information

Key Concepts: Economic Computation, Part III

Key Concepts: Economic Computation, Part III Key Concepts: Economic Computation, Part III Brent Hickman Summer, 8 1 Using Newton s Method to Find Roots of Real- Valued Functions The intuition behind Newton s method is that finding zeros of non-linear

More information

Numerical Methods I Solving Nonlinear Equations

Numerical Methods I Solving Nonlinear Equations Numerical Methods I Solving Nonlinear Equations Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 October 16th, 2014 A. Donev (Courant Institute)

More information

Outline. Scientific Computing: An Introductory Survey. Nonlinear Equations. Nonlinear Equations. Examples: Nonlinear Equations

Outline. Scientific Computing: An Introductory Survey. Nonlinear Equations. Nonlinear Equations. Examples: Nonlinear Equations Methods for Systems of Methods for Systems of Outline Scientific Computing: An Introductory Survey Chapter 5 1 Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign

More information

Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Scientific Computing: An Introductory Survey Chapter 5 Nonlinear Equations Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction

More information

CS 323: Numerical Analysis and Computing

CS 323: Numerical Analysis and Computing CS 323: Numerical Analysis and Computing MIDTERM #2 Instructions: This is an open notes exam, i.e., you are allowed to consult any textbook, your class notes, homeworks, or any of the handouts from us.

More information

EAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science

EAD 115. Numerical Solution of Engineering and Scientific Problems. David M. Rocke Department of Applied Science EAD 115 Numerical Solution of Engineering and Scientific Problems David M. Rocke Department of Applied Science Multidimensional Unconstrained Optimization Suppose we have a function f() of more than one

More information

5 Quasi-Newton Methods

5 Quasi-Newton Methods Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min

More information

x 2 x n r n J(x + t(x x ))(x x )dt. For warming-up we start with methods for solving a single equation of one variable.

x 2 x n r n J(x + t(x x ))(x x )dt. For warming-up we start with methods for solving a single equation of one variable. Maria Cameron 1. Fixed point methods for solving nonlinear equations We address the problem of solving an equation of the form (1) r(x) = 0, where F (x) : R n R n is a vector-function. Eq. (1) can be written

More information

Line Search Methods. Shefali Kulkarni-Thaker

Line Search Methods. Shefali Kulkarni-Thaker 1 BISECTION METHOD Line Search Methods Shefali Kulkarni-Thaker Consider the following unconstrained optimization problem min f(x) x R Any optimization algorithm starts by an initial point x 0 and performs

More information

Solving non-linear systems of equations

Solving non-linear systems of equations Solving non-linear systems of equations Felix Kubler 1 1 DBF, University of Zurich and Swiss Finance Institute October 7, 2017 Felix Kubler Comp.Econ. Gerzensee, Ch2 October 7, 2017 1 / 38 The problem

More information

Numerical Methods in Informatics

Numerical Methods in Informatics Numerical Methods in Informatics Lecture 2, 30.09.2016: Nonlinear Equations in One Variable http://www.math.uzh.ch/binf4232 Tulin Kaman Institute of Mathematics, University of Zurich E-mail: tulin.kaman@math.uzh.ch

More information

Numerical solutions of nonlinear systems of equations

Numerical solutions of nonlinear systems of equations Numerical solutions of nonlinear systems of equations Tsung-Ming Huang Department of Mathematics National Taiwan Normal University, Taiwan E-mail: min@math.ntnu.edu.tw August 28, 2011 Outline 1 Fixed points

More information

Lecture 7: Numerical Tools

Lecture 7: Numerical Tools Lecture 7: Numerical Tools Fatih Guvenen January 10, 2016 Fatih Guvenen Lecture 7: Numerical Tools January 10, 2016 1 / 18 Overview Three Steps: V (k, z) =max c,k 0 apple u(c)+ Z c + k 0 =(1 + r)k + z

More information

CS 323: Numerical Analysis and Computing

CS 323: Numerical Analysis and Computing CS 323: Numerical Analysis and Computing MIDTERM #2 Instructions: This is an open notes exam, i.e., you are allowed to consult any textbook, your class notes, homeworks, or any of the handouts from us.

More information

Numerical Analysis Fall. Roots: Open Methods

Numerical Analysis Fall. Roots: Open Methods Numerical Analysis 2015 Fall Roots: Open Methods Open Methods Open methods differ from bracketing methods, in that they require only a single starting value or two starting values that do not necessarily

More information

Lecture V. Numerical Optimization

Lecture V. Numerical Optimization Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize

More information

Math 411 Preliminaries

Math 411 Preliminaries Math 411 Preliminaries Provide a list of preliminary vocabulary and concepts Preliminary Basic Netwon s method, Taylor series expansion (for single and multiple variables), Eigenvalue, Eigenvector, Vector

More information

MATH 350: Introduction to Computational Mathematics

MATH 350: Introduction to Computational Mathematics MATH 350: Introduction to Computational Mathematics Chapter IV: Locating Roots of Equations Greg Fasshauer Department of Applied Mathematics Illinois Institute of Technology Spring 2011 fasshauer@iit.edu

More information

MATH 350: Introduction to Computational Mathematics

MATH 350: Introduction to Computational Mathematics MATH 350: Introduction to Computational Mathematics Chapter IV: Locating Roots of Equations Greg Fasshauer Department of Applied Mathematics Illinois Institute of Technology Spring 2011 fasshauer@iit.edu

More information

Lecture 8. Root finding II

Lecture 8. Root finding II 1 Introduction Lecture 8 Root finding II In the previous lecture we considered the bisection root-bracketing algorithm. It requires only that the function be continuous and that we have a root bracketed

More information

ECS550NFB Introduction to Numerical Methods using Matlab Day 2

ECS550NFB Introduction to Numerical Methods using Matlab Day 2 ECS550NFB Introduction to Numerical Methods using Matlab Day 2 Lukas Laffers lukas.laffers@umb.sk Department of Mathematics, University of Matej Bel June 9, 2015 Today Root-finding: find x that solves

More information

Chapter 1. Root Finding Methods. 1.1 Bisection method

Chapter 1. Root Finding Methods. 1.1 Bisection method Chapter 1 Root Finding Methods We begin by considering numerical solutions to the problem f(x) = 0 (1.1) Although the problem above is simple to state it is not always easy to solve analytically. This

More information

Unconstrained Multivariate Optimization

Unconstrained Multivariate Optimization Unconstrained Multivariate Optimization Multivariate optimization means optimization of a scalar function of a several variables: and has the general form: y = () min ( ) where () is a nonlinear scalar-valued

More information

17 Solution of Nonlinear Systems

17 Solution of Nonlinear Systems 17 Solution of Nonlinear Systems We now discuss the solution of systems of nonlinear equations. An important ingredient will be the multivariate Taylor theorem. Theorem 17.1 Let D = {x 1, x 2,..., x m

More information

Applied Mathematics 205. Unit I: Data Fitting. Lecturer: Dr. David Knezevic

Applied Mathematics 205. Unit I: Data Fitting. Lecturer: Dr. David Knezevic Applied Mathematics 205 Unit I: Data Fitting Lecturer: Dr. David Knezevic Unit I: Data Fitting Chapter I.4: Nonlinear Least Squares 2 / 25 Nonlinear Least Squares So far we have looked at finding a best

More information

Math 409/509 (Spring 2011)

Math 409/509 (Spring 2011) Math 409/509 (Spring 2011) Instructor: Emre Mengi Study Guide for Homework 2 This homework concerns the root-finding problem and line-search algorithms for unconstrained optimization. Please don t hesitate

More information

Queens College, CUNY, Department of Computer Science Numerical Methods CSCI 361 / 761 Spring 2018 Instructor: Dr. Sateesh Mane.

Queens College, CUNY, Department of Computer Science Numerical Methods CSCI 361 / 761 Spring 2018 Instructor: Dr. Sateesh Mane. Queens College, CUNY, Department of Computer Science Numerical Methods CSCI 361 / 761 Spring 2018 Instructor: Dr. Sateesh Mane c Sateesh R. Mane 2018 3 Lecture 3 3.1 General remarks March 4, 2018 This

More information

Advanced Microeconomics

Advanced Microeconomics Advanced Microeconomics Leonardo Felli EC441: Room D.106, Z.332, D.109 Lecture 8 bis: 24 November 2004 Monopoly Consider now the pricing behavior of a profit maximizing monopolist: a firm that is the only

More information

(One Dimension) Problem: for a function f(x), find x 0 such that f(x 0 ) = 0. f(x)

(One Dimension) Problem: for a function f(x), find x 0 such that f(x 0 ) = 0. f(x) Solving Nonlinear Equations & Optimization One Dimension Problem: or a unction, ind 0 such that 0 = 0. 0 One Root: The Bisection Method This one s guaranteed to converge at least to a singularity, i not

More information

, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are

, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are Quadratic forms We consider the quadratic function f : R 2 R defined by f(x) = 2 xt Ax b T x with x = (x, x 2 ) T, () where A R 2 2 is symmetric and b R 2. We will see that, depending on the eigenvalues

More information

Line Search Algorithms

Line Search Algorithms Lab 1 Line Search Algorithms Investigate various Line-Search algorithms for numerical opti- Lab Objective: mization. Overview of Line Search Algorithms Imagine you are out hiking on a mountain, and you

More information

1 The best of all possible worlds

1 The best of all possible worlds Notes for 2017-03-18 1 The best of all possible worlds Last time, we discussed three methods of solving f(x) = 0: Newton, modified Newton, and bisection. Newton is potentially faster than bisection; bisection

More information

1. Method 1: bisection. The bisection methods starts from two points a 0 and b 0 such that

1. Method 1: bisection. The bisection methods starts from two points a 0 and b 0 such that Chapter 4 Nonlinear equations 4.1 Root finding Consider the problem of solving any nonlinear relation g(x) = h(x) in the real variable x. We rephrase this problem as one of finding the zero (root) of a

More information

Chapter 3: Root Finding. September 26, 2005

Chapter 3: Root Finding. September 26, 2005 Chapter 3: Root Finding September 26, 2005 Outline 1 Root Finding 2 3.1 The Bisection Method 3 3.2 Newton s Method: Derivation and Examples 4 3.3 How To Stop Newton s Method 5 3.4 Application: Division

More information

SOLUTION OF ALGEBRAIC AND TRANSCENDENTAL EQUATIONS BISECTION METHOD

SOLUTION OF ALGEBRAIC AND TRANSCENDENTAL EQUATIONS BISECTION METHOD BISECTION METHOD If a function f(x) is continuous between a and b, and f(a) and f(b) are of opposite signs, then there exists at least one root between a and b. It is shown graphically as, Let f a be negative

More information

THE SECANT METHOD. q(x) = a 0 + a 1 x. with

THE SECANT METHOD. q(x) = a 0 + a 1 x. with THE SECANT METHOD Newton s method was based on using the line tangent to the curve of y = f (x), with the point of tangency (x 0, f (x 0 )). When x 0 α, the graph of the tangent line is approximately the

More information

Lecture 5. September 4, 2018 Math/CS 471: Introduction to Scientific Computing University of New Mexico

Lecture 5. September 4, 2018 Math/CS 471: Introduction to Scientific Computing University of New Mexico Lecture 5 September 4, 2018 Math/CS 471: Introduction to Scientific Computing University of New Mexico 1 Review: Office hours at regularly scheduled times this week Tuesday: 9:30am-11am Wed: 2:30pm-4:00pm

More information

Non-polynomial Least-squares fitting

Non-polynomial Least-squares fitting Applied Math 205 Last time: piecewise polynomial interpolation, least-squares fitting Today: underdetermined least squares, nonlinear least squares Homework 1 (and subsequent homeworks) have several parts

More information

Optimization 2. CS5240 Theoretical Foundations in Multimedia. Leow Wee Kheng

Optimization 2. CS5240 Theoretical Foundations in Multimedia. Leow Wee Kheng Optimization 2 CS5240 Theoretical Foundations in Multimedia Leow Wee Kheng Department of Computer Science School of Computing National University of Singapore Leow Wee Kheng (NUS) Optimization 2 1 / 38

More information

MATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year

MATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year MATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2 1 Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year 2013-14 OUTLINE OF WEEK 8 topics: quadratic optimisation, least squares,

More information

Goals for This Lecture:

Goals for This Lecture: Goals for This Lecture: Learn the Newton-Raphson method for finding real roots of real functions Learn the Bisection method for finding real roots of a real function Look at efficient implementations of

More information

Solution of Nonlinear Equations

Solution of Nonlinear Equations Solution of Nonlinear Equations (Com S 477/577 Notes) Yan-Bin Jia Sep 14, 017 One of the most frequently occurring problems in scientific work is to find the roots of equations of the form f(x) = 0. (1)

More information

Optimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23

Optimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23 Optimization: Nonlinear Optimization without Constraints Nonlinear Optimization without Constraints 1 / 23 Nonlinear optimization without constraints Unconstrained minimization min x f(x) where f(x) is

More information

Maria Cameron. f(x) = 1 n

Maria Cameron. f(x) = 1 n Maria Cameron 1. Local algorithms for solving nonlinear equations Here we discuss local methods for nonlinear equations r(x) =. These methods are Newton, inexact Newton and quasi-newton. We will show that

More information

Bindel, Spring 2016 Numerical Analysis (CS 4220) Notes for

Bindel, Spring 2016 Numerical Analysis (CS 4220) Notes for Life beyond Newton Notes for 2016-04-08 Newton s method has many attractive properties, particularly when we combine it with a globalization strategy. Unfortunately, Newton steps are not cheap. At each

More information

Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 3. Gradient Method

Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 3. Gradient Method Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 3 Gradient Method Shiqian Ma, MAT-258A: Numerical Optimization 2 3.1. Gradient method Classical gradient method: to minimize a differentiable convex

More information

Motivation: We have already seen an example of a system of nonlinear equations when we studied Gaussian integration (p.8 of integration notes)

Motivation: We have already seen an example of a system of nonlinear equations when we studied Gaussian integration (p.8 of integration notes) AMSC/CMSC 460 Computational Methods, Fall 2007 UNIT 5: Nonlinear Equations Dianne P. O Leary c 2001, 2002, 2007 Solving Nonlinear Equations and Optimization Problems Read Chapter 8. Skip Section 8.1.1.

More information

Unconstrained optimization

Unconstrained optimization Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout

More information

CHEE 222: PROCESS DYNAMICS AND NUMERICAL METHODS

CHEE 222: PROCESS DYNAMICS AND NUMERICAL METHODS CHEE 222: PROCESS DYNAMICS AND NUMERICAL METHODS Winter 2017 Implementation of Numerical Methods via MATLAB Instructor: Xiang Li 1 Outline 1. Introduction - Command, script and function - MATLAB function

More information

Simple Iteration, cont d

Simple Iteration, cont d Jim Lambers MAT 772 Fall Semester 2010-11 Lecture 2 Notes These notes correspond to Section 1.2 in the text. Simple Iteration, cont d In general, nonlinear equations cannot be solved in a finite sequence

More information

Tvestlanka Karagyozova University of Connecticut

Tvestlanka Karagyozova University of Connecticut September, 005 CALCULUS REVIEW Tvestlanka Karagyozova University of Connecticut. FUNCTIONS.. Definition: A function f is a rule that associates each value of one variable with one and only one value of

More information

October 16, 2018 Notes on Cournot. 1. Teaching Cournot Equilibrium

October 16, 2018 Notes on Cournot. 1. Teaching Cournot Equilibrium October 1, 2018 Notes on Cournot 1. Teaching Cournot Equilibrium Typically Cournot equilibrium is taught with identical zero or constant-mc cost functions for the two firms, because that is simpler. I

More information

Optimization. Totally not complete this is...don't use it yet...

Optimization. Totally not complete this is...don't use it yet... Optimization Totally not complete this is...don't use it yet... Bisection? Doing a root method is akin to doing a optimization method, but bi-section would not be an effective method - can detect sign

More information

Bindel, Fall 2011 Intro to Scientific Computing (CS 3220) Week 6: Monday, Mar 7. e k+1 = 1 f (ξ k ) 2 f (x k ) e2 k.

Bindel, Fall 2011 Intro to Scientific Computing (CS 3220) Week 6: Monday, Mar 7. e k+1 = 1 f (ξ k ) 2 f (x k ) e2 k. Problem du jour Week 6: Monday, Mar 7 Show that for any initial guess x 0 > 0, Newton iteration on f(x) = x 2 a produces a decreasing sequence x 1 x 2... x n a. What is the rate of convergence if a = 0?

More information

Scientific Computing: Optimization

Scientific Computing: Optimization Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture

More information

A nonlinear equation is any equation of the form. f(x) = 0. A nonlinear equation can have any number of solutions (finite, countable, uncountable)

A nonlinear equation is any equation of the form. f(x) = 0. A nonlinear equation can have any number of solutions (finite, countable, uncountable) Nonlinear equations Definition A nonlinear equation is any equation of the form where f is a nonlinear function. Nonlinear equations x 2 + x + 1 = 0 (f : R R) f(x) = 0 (x cos y, 2y sin x) = (0, 0) (f :

More information

Math 551 Homework Assignment 3 Page 1 of 6

Math 551 Homework Assignment 3 Page 1 of 6 Math 551 Homework Assignment 3 Page 1 of 6 Name and section: ID number: E-mail: 1. Consider Newton s method for finding + α with α > 0 by finding the positive root of f(x) = x 2 α = 0. Assuming that x

More information

Quadratic function and equations Quadratic function/equations, supply, demand, market equilibrium

Quadratic function and equations Quadratic function/equations, supply, demand, market equilibrium Exercises 8 Quadratic function and equations Quadratic function/equations, supply, demand, market equilibrium Objectives - know and understand the relation between a quadratic function and a quadratic

More information

Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted

More information

Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted

More information

Chapter 4 AD AS. O. Afonso, P. B. Vasconcelos. Computational Economics: a concise introduction

Chapter 4 AD AS. O. Afonso, P. B. Vasconcelos. Computational Economics: a concise introduction Chapter 4 AD AS O. Afonso, P. B. Vasconcelos Computational Economics: a concise introduction O. Afonso, P. B. Vasconcelos Computational Economics 1 / 32 Overview 1 Introduction 2 Economic model 3 Numerical

More information

Numerical Analysis: Solving Nonlinear Equations

Numerical Analysis: Solving Nonlinear Equations Numerical Analysis: Solving Nonlinear Equations Mirko Navara http://cmp.felk.cvut.cz/ navara/ Center for Machine Perception, Department of Cybernetics, FEE, CTU Karlovo náměstí, building G, office 104a

More information

min f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;

min f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term; Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many

More information

CLASS NOTES Models, Algorithms and Data: Introduction to computing 2018

CLASS NOTES Models, Algorithms and Data: Introduction to computing 2018 CLASS NOTES Models, Algorithms and Data: Introduction to computing 2018 Petros Koumoutsakos, Jens Honore Walther (Last update: April 16, 2018) IMPORTANT DISCLAIMERS 1. REFERENCES: Much of the material

More information

Chapter 6. Nonlinear Equations. 6.1 The Problem of Nonlinear Root-finding. 6.2 Rate of Convergence

Chapter 6. Nonlinear Equations. 6.1 The Problem of Nonlinear Root-finding. 6.2 Rate of Convergence Chapter 6 Nonlinear Equations 6. The Problem of Nonlinear Root-finding In this module we consider the problem of using numerical techniques to find the roots of nonlinear equations, f () =. Initially we

More information

Investigating Limits in MATLAB

Investigating Limits in MATLAB MTH229 Investigating Limits in MATLAB Project 5 Exercises NAME: SECTION: INSTRUCTOR: Exercise 1: Use the graphical approach to find the following right limit of f(x) = x x, x > 0 lim x 0 + xx What is the

More information

Numerical Solution of f(x) = 0

Numerical Solution of f(x) = 0 Numerical Solution of f(x) = 0 Gerald W. Recktenwald Department of Mechanical Engineering Portland State University gerry@pdx.edu ME 350: Finding roots of f(x) = 0 Overview Topics covered in these slides

More information

Oligopoly Theory 2 Bertrand Market Games

Oligopoly Theory 2 Bertrand Market Games 1/10 Oligopoly Theory 2 Bertrand Market Games May 4, 2014 2/10 Outline 1 Bertrand Market Game 2 Bertrand Paradox 3 Asymmetric Firms 3/10 Bertrand Duopoly Market Game Discontinuous Payoff Functions (1 p

More information

6.254 : Game Theory with Engineering Applications Lecture 7: Supermodular Games

6.254 : Game Theory with Engineering Applications Lecture 7: Supermodular Games 6.254 : Game Theory with Engineering Applications Lecture 7: Asu Ozdaglar MIT February 25, 2010 1 Introduction Outline Uniqueness of a Pure Nash Equilibrium for Continuous Games Reading: Rosen J.B., Existence

More information

Quasi-Newton Methods

Quasi-Newton Methods Newton s Method Pros and Cons Quasi-Newton Methods MA 348 Kurt Bryan Newton s method has some very nice properties: It s extremely fast, at least once it gets near the minimum, and with the simple modifications

More information

Fixed Point Theorems

Fixed Point Theorems Fixed Point Theorems Definition: Let X be a set and let f : X X be a function that maps X into itself. (Such a function is often called an operator, a transformation, or a transform on X, and the notation

More information

Chapter 4. Unconstrained optimization

Chapter 4. Unconstrained optimization Chapter 4. Unconstrained optimization Version: 28-10-2012 Material: (for details see) Chapter 11 in [FKS] (pp.251-276) A reference e.g. L.11.2 refers to the corresponding Lemma in the book [FKS] PDF-file

More information

Lecture 15. Dynamic Stochastic General Equilibrium Model. Randall Romero Aguilar, PhD I Semestre 2017 Last updated: July 3, 2017

Lecture 15. Dynamic Stochastic General Equilibrium Model. Randall Romero Aguilar, PhD I Semestre 2017 Last updated: July 3, 2017 Lecture 15 Dynamic Stochastic General Equilibrium Model Randall Romero Aguilar, PhD I Semestre 2017 Last updated: July 3, 2017 Universidad de Costa Rica EC3201 - Teoría Macroeconómica 2 Table of contents

More information

Numerical Study of Some Iterative Methods for Solving Nonlinear Equations

Numerical Study of Some Iterative Methods for Solving Nonlinear Equations International Journal of Engineering Science Invention ISSN (Online): 2319 6734, ISSN (Print): 2319 6726 Volume 5 Issue 2 February 2016 PP.0110 Numerical Study of Some Iterative Methods for Solving Nonlinear

More information

Review for Exam 2 Ben Wang and Mark Styczynski

Review for Exam 2 Ben Wang and Mark Styczynski Review for Exam Ben Wang and Mark Styczynski This is a rough approximation of what we went over in the review session. This is actually more detailed in portions than what we went over. Also, please note

More information

Optimization II: Unconstrained Multivariable

Optimization II: Unconstrained Multivariable Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Optimization II: Unconstrained Multivariable 1

More information

Solving Dynamic Games with Newton s Method

Solving Dynamic Games with Newton s Method Michael Ferris 1 Kenneth L. Judd 2 Karl Schmedders 3 1 Department of Computer Science, University of Wisconsin at Madison 2 Hoover Institution, Stanford University 3 Institute for Operations Research,

More information

Numerical Methods Lecture 3

Numerical Methods Lecture 3 Numerical Methods Lecture 3 Nonlinear Equations by Pavel Ludvík Introduction Definition (Root or zero of a function) A root (or a zero) of a function f is a solution of an equation f (x) = 0. We learn

More information

INTRODUCTION TO NUMERICAL ANALYSIS

INTRODUCTION TO NUMERICAL ANALYSIS INTRODUCTION TO NUMERICAL ANALYSIS Cho, Hyoung Kyu Department of Nuclear Engineering Seoul National University 3. SOLVING NONLINEAR EQUATIONS 3.1 Background 3.2 Estimation of errors in numerical solutions

More information

DISCRETE-TIME DYNAMICS OF AN

DISCRETE-TIME DYNAMICS OF AN Chapter 1 DISCRETE-TIME DYNAMICS OF AN OLIGOPOLY MODEL WITH DIFFERENTIATED GOODS K. Andriopoulos, T. Bountis and S. Dimas * Department of Mathematics, University of Patras, Patras, GR-26500, Greece Abstract

More information

446 CHAP. 8 NUMERICAL OPTIMIZATION. Newton's Search for a Minimum of f(x,y) Newton s Method

446 CHAP. 8 NUMERICAL OPTIMIZATION. Newton's Search for a Minimum of f(x,y) Newton s Method 446 CHAP. 8 NUMERICAL OPTIMIZATION Newton's Search for a Minimum of f(xy) Newton s Method The quadratic approximation method of Section 8.1 generated a sequence of seconddegree Lagrange polynomials. It

More information

Math 4329: Numerical Analysis Chapter 03: Fixed Point Iteration and Ill behaving problems. Natasha S. Sharma, PhD

Math 4329: Numerical Analysis Chapter 03: Fixed Point Iteration and Ill behaving problems. Natasha S. Sharma, PhD Why another root finding technique? iteration gives us the freedom to design our own root finding algorithm. The design of such algorithms is motivated by the need to improve the speed and accuracy of

More information

MA 8019: Numerical Analysis I Solution of Nonlinear Equations

MA 8019: Numerical Analysis I Solution of Nonlinear Equations MA 8019: Numerical Analysis I Solution of Nonlinear Equations Suh-Yuh Yang ( 楊肅煜 ) Department of Mathematics, National Central University Jhongli District, Taoyuan City 32001, Taiwan syyang@math.ncu.edu.tw

More information

CHAPTER 10 Zeros of Functions

CHAPTER 10 Zeros of Functions CHAPTER 10 Zeros of Functions An important part of the maths syllabus in secondary school is equation solving. This is important for the simple reason that equations are important a wide range of problems

More information

1 Functions and Graphs

1 Functions and Graphs 1 Functions and Graphs 1.1 Functions Cartesian Coordinate System A Cartesian or rectangular coordinate system is formed by the intersection of a horizontal real number line, usually called the x axis,

More information

Outline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems

Outline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction

More information

Nonlinear Optimization for Optimal Control

Nonlinear Optimization for Optimal Control Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]

More information

Statistics 580 Optimization Methods

Statistics 580 Optimization Methods Statistics 580 Optimization Methods Introduction Let fx be a given real-valued function on R p. The general optimization problem is to find an x ɛ R p at which fx attain a maximum or a minimum. It is of

More information

AM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods

AM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality

More information

MATH 3795 Lecture 12. Numerical Solution of Nonlinear Equations.

MATH 3795 Lecture 12. Numerical Solution of Nonlinear Equations. MATH 3795 Lecture 12. Numerical Solution of Nonlinear Equations. Dmitriy Leykekhman Fall 2008 Goals Learn about different methods for the solution of f(x) = 0, their advantages and disadvantages. Convergence

More information

4 damped (modified) Newton methods

4 damped (modified) Newton methods 4 damped (modified) Newton methods 4.1 damped Newton method Exercise 4.1 Determine with the damped Newton method the unique real zero x of the real valued function of one variable f(x) = x 3 +x 2 using

More information

Nonlinear dynamics in the Cournot duopoly game with heterogeneous players

Nonlinear dynamics in the Cournot duopoly game with heterogeneous players arxiv:nlin/0210035v1 [nlin.cd] 16 Oct 2002 Nonlinear dynamics in the Cournot duopoly game with heterogeneous players H. N. Agiza and A. A. Elsadany Department of Mathematics, Faculty of Science Mansoura

More information

2.4 - Convergence of the Newton Method and Modified Newton Method

2.4 - Convergence of the Newton Method and Modified Newton Method 2.4 - Convergence of the Newton Method and Modified Newton Method Consider the problem of finding the solution of the equation: for in Assume that is continuous and for in 1. Newton's Method: Suppose that

More information

CS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares

CS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares CS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares Robert Bridson October 29, 2008 1 Hessian Problems in Newton Last time we fixed one of plain Newton s problems by introducing line search

More information

Game theory and market power

Game theory and market power Game theory and market power Josh Taylor Section 6.1.3, 6.3 in Convex Optimization of Power Systems. 1 Market weaknesses Recall Optimal power flow: minimize p,θ subject to λ i : χ ij 0 : f i (p i ) i p

More information