Lecture 5. September 4, 2018 Math/CS 471: Introduction to Scientific Computing University of New Mexico

Transcription

1 Lecture 5 September 4, 2018 Math/CS 471: Introduction to Scientific Computing University of New Mexico 1

2 Review: Office hours at regularly scheduled times this week Tuesday: 9:30am-11am Wed: 2:30pm-4:00pm Homework 1: Due at 23:59 on Tuesday, Sept. 4. (Today!) Make sure that Cairn Overturf (covert01) and Jacob Schroder (jbschroder) are added as developers to your LoboGit repository When submitting, me and Cairn the hash tag, and which LoboGit repository to use (i.e., which homework partner s repository contains the assignment.) Homework 2: Will go up later this week, so look for it. Is anyone looking for a HW partner? If so, please me. 2

3 Newton s Method for Root-Finding Given a scalar equation, where is a continuous function, we want to compute it s approximate solutions, i.e., the roots of the function 6 4 Y X 3

4 Iterative Methods Definition: An iterative method is a mathematical procedure that generates a sequence of improving approximate solutions to a problem. An iterative method starts with an initial approximation, which is often called an initial guess. Convergence: Let be the exact solution to a given root-finding problem, i.e. Let be the initial guess and consider a sequence of improving approximate solutions generated by an iterative method. Consider the absolute approximation error. The iterative method is said to be convergent for the given initial guess, if its corresponding sequence converges to, that is, if Linear convergence: An iterative method is said to be linearly convergent if Quadratic convergence: An iterative method is said to be quadratically convergent if 4

5 Iterative Methods Stopping Criterion: A stopping criterion tells us when to terminate an iterative algorithm. How close to the solution, is close enough? Let s consider three options. 1. Given a small error tolerance TOL, an obvious stopping criterion is. This will determine the number of iterations n necessary to obtain an approximate solution x n within the error tolerance. Note that in order to use this criterion we will need to know the exact root r, which is often not available. 2. In most practical root-finding problems, we do not have access to the exact root r. In such cases, we may consider the quantity, which approximates the absolute error. The corresponding stopping criterion will then be. 1. This is an acceptable criterion for halting, but TOL still has to be tuned to depend on the size of x. 3. To have a tolerance that is robust regarding the size of x, we can use a relative criteria, Note: In addition to stopping criteria, there is also usually a maximum number of allowed iterations, in case convergence fails, or there is a bug in the code. 5

6 Newton s Method Newton s method is a simple iterative method for finding roots of functions. The basic idea behind the method is to approximate the function with the tangent line and then approximate the root of the function by the root of the tangent line. Algorithm: Let x 0 be an initial guess for a root of f. For n = 0, 1, 2,... iteratively compute x n+1 in terms of the just computed x n by: x n+1 = x n f(x n )/f (x n ) Then for most functions and a reasonable initial guess, the sequence converges to a root of f. Demonstration of Newton s method converging to a root of a function 6

7 Newton s Method Newton s method is a simple iterative method for finding roots of functions. The basic idea behind the method is to approximate the function with the tangent line and then approximate the root of the function by the root of the tangent line. Algorithm: Let x 0 be an initial guess for a root of f. For n = 0, 1, 2,... iteratively compute x n+1 in terms of the just computed x n by: x n+1 = x n f(x n )/f (x n ) Then for most functions and a reasonable initial guess, the sequence converges to a root of f. Demonstration of Newton s method converging to a root of a function 7

8 Newton s Method Newton s method is a simple iterative method for finding roots of functions. The basic idea behind the method is to approximate the function with the tangent line and then approximate the root of the function by the root of the tangent line. Algorithm: Let x 0 be an initial guess for a root of f. For n = 0, 1, 2,... iteratively compute x n+1 in terms of the just computed x n by: x n+1 = x n f(x n )/f (x n ) Then for most functions and a reasonable initial guess, the sequence converges to a root of f. Demonstration of Newton s method converging to a root of a function 8

9 Newton s Method Newton s method is a simple iterative method for finding roots of functions. The basic idea behind the method is to approximate the function with the tangent line and then approximate the root of the function by the root of the tangent line. Algorithm: Let x 0 be an initial guess for a root of f. For n = 0, 1, 2,... iteratively compute x n+1 in terms of the just computed x n by: x n+1 = x n f(x n )/f (x n ) Then for most functions and a reasonable initial guess, the sequence converges to a root of f. Demonstration of Newton s method converging to a root of a function 9

10 Newton s Method Newton s method is a simple iterative method for finding roots of functions. The basic idea behind the method is to approximate the function with the tangent line and then approximate the root of the function by the root of the tangent line. Algorithm: Let x 0 be an initial guess for a root of f. For n = 0, 1, 2,... iteratively compute x n+1 in terms of the just computed x n by: x n+1 = x n f(x n )/f (x n ) Then for most functions and a reasonable initial guess, the sequence converges to a root of f. Demonstration of Newton s method converging to a root of a function 10

11 Newton s Method The derivation of Newton s Method relies simply on Taylor s theorem In Class Exercise: 1. Take the linear approximation to implied by Taylor s theorem 2. Set that approximation equal to 0, and solve for h (we are solving for the root!) 3. Use this step-size to finish deriving the Newton s method 11

12 Newton s Method Convergence: Let f be a twice differentiable function with a root r, i.e., f(r) = 0. If f (r) 0, then for a reasonable initial guess (one that is sufficiently close to r) Newton s method is quadratically converging to r, with The convergence will be quadratic if: 1. The initial guess is not too far from the root r, or equivalently the initial error is not too large. 2. The derivative of f at x n (which appears in the denominator) is not too close to 0. This means that the function f is not too flat at x n, otherwise the tangent line would be almost parallel to the x-axis and would intersect it far away. In particular, we need f (r) The second derivative of f is not too big in absolute value near the root. A large second derivative means the function is very concave or very convex, and hence turns away from the tangent line quickly. 12

13 Newton s Method Showing the quadratic convergence again uses Taylor s theorem where Re-arrange is between x n and r Note that, and apply the definition of a Newton s method update to get Take the limit to obtain expression from previous slide Remark: from this, we can see that "around" the root, the first derivative cannot equal 0 and that the second derivative must be continuous 13

14 Modified Newton s Method If r is a multiple root of the function f with multiplicity, that is, if f(r) = f (r) =... = f (m-1) (r) = 0, but f (m) 0, then Newton s method will not converge quadratically. In this case, Newton s method will be linearly convergent with If the multiplicity m of the root is known in advance, it is possible to modify Newton s method and recover quadratic convergence. The modified Newton s method is x n+1 = x n m f(x n )/f (x n ) 14

15 Multidimensional Newton and Systems Newton's method can also be interpreted as a nonlinear solver, i.e., solve the nonlinear equation(s) This is opposed to linear equations, like For systems of equations, just use the "Jacobian", which is the multidimensional equivalent to a first derivative, For instance, in two-dimensions With this, Newton's method can be used to solve complicated multi-dimensional nonlinear systems of equations! Carries over to optimization problems, as well. 15

16 Compare Quadratic and Linear Convergence Simple root (multiplicity 1) Double root (multiplicity 2) Newton's Method: Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Error is roughly squared each iteration Newton's Method: Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Approx. root e+00 Error is roughly halved each iteration 16

17 Some More Homework Guidelines This is for HW2 and beyond 17

18 Some More Homework Guidelines 1. You are NOT done with the homework when the code runs. That is just the first 33% of the homework! 2. Just because the code runs, that does not mean it is bug free. 3. Make sure that you have time to do the computations needed for the report. These computations will let you and the graders verify that the code works correctly. 4. Make sure you have time to write the report. We will grade on correctness of the code, results, and the exposition of the material. 18

19 Debugging Classic debugging includes print statements. This is simple, but effective. We will also go over PDB, which lets you debug Python in the middle of a computation. But as importantly, we will debug by examining the numerical behavior of an algorithm. If your implementation doesn't behave according to the theory, then something is wrong. Example: if your code approximates a derivative with a forward difference Then, you would expect that the error behaves linearly as: 19

20 Reporting Convergence Suppose that we want to display an error on the order of We want to show (graphically) convergence of order q >>> from scipy import * >>> from matplotlib import pyplot >>> h = 2**-linspace(0,10,1000) >>> pyplot.plot(h,h,h,h**2,h,h**3) >>> pyplot.show() This figure is not good! It does not clearly show the rate of q, or the difference between the curves, That is, no labels or title 20

21 Reporting Convergence Recall that that implies Try >>> pyplot.loglog(h,h,h,h**2,h,h**3) A bit better! The slope is the exponent (work this out for yourself) 21

22 Reporting Convergence Lastly, to make the plot acceptable for a report 1. Add a legend 2. Make the lines wider 3. Use a larger font size 4. Turn the grid on 5. Label the axes 22

23 How to organize your homework repository Your LoboGit Repo README HW1 HW2 HW3... README Report Code Files homework2.pdf code1 code2 code3... script1 file1 script

24 A typical low-level README README file for Homework 2, Math 471, Fall 2018, J. Schroder The homework report is called homework2.pdf. - The codes are located in the sub-directory /Codes and are documented in the appendix of the report. - The script files X and Y are located in the sub-directory /Files. - To reproduce the results in the report do this: - 1. bla bla bla - 2. bla bla bla

25 25