Chapter 3: Root Finding. September 26, 2005

Transcription

1 Chapter 3: Root Finding September 26, 2005

2 Outline 1 Root Finding The Bisection Method Newton s Method: Derivation and Examples How To Stop Newton s Method Application: Division Using Newton s Method The Newton Error Formula The Secant method: Derivation and Examples Fixed-Point Iteration

3 Calculating the roots of an arbitrary equation f (x) = 0 is a common problem in applied mathematics. We will explore some simple numerical methods (algorithms) for solving this equation and possible difficulties that may arise.

4 Example If we borrow L dollars at an annual interest rate of r for a period of m years, the size of monthly payment M is given by the equation L = 12M (1 (1 + r r 12 ) 12m ). Suppose we ant to to take out a mortgage of $150,000 and that we can afford a monthly payment of $1,000 and the amortization period of 20 years, what is the maximum interest rate we can afford? Then, 150, 000 = 12 1, 000 (1 (1 + r r 12 ) ) and we need to solve this equation for r. For that, we need a numerical method.

5 This is a very simple method, based on the continuity of the function and the Intermediate Value Theorem. If f (x) is continuous on [a, b] and f (a)f (b) < 0 then f (x) changes sign on [a, b] so it must be the case that f (x) = 0 for at least one x [a, b]. The simplest numerical procedure for finding a root is to repeatedly halve the interval [a, b], keeping the half on which f (x) changes sign.

6 Suppose c is the midpoint of [a, b]: c = 1 (a + b). 2 Then, we have three possibilities for the product f (a)f (c): 1 f (a)f (c) < 0. So, f changes sign between a and c and there is a root in that interval. 2 f (a)f (c) = 0. In that case, since we are assuming that f (a) 0, f (c) = 0 and we have found a root. 3 f (a)f (c) > 0, so a root must be in the other half [c, b].

7 Example Find the largest root of the equation accurate to within ɛ = x 6 x 1 = 0 4 y x 2 4 Figure: f (x) = x 6 x 1

8 So, the root we are looking for is in the interval [1, 2]. a = 1, b = 2, f (a) = 1, f (2) = 61. c = = 3 2 = , f (1.5000) = Since f (c) > 0, we know that the root is between a and c Halve the interval, by taking: Now, c = a := 1, b := c = = , f (1.2500) =

9 Since f (a)f (c) < 0, we halve the interval in the following way a := 1, b := c = Again, c = = , f (c) = Since f (a)f (c) > 0, the half of the interval we use in the next step is a := c = , b := We continue in the same fashion...

10 a b c b c f (c) We will take x = as the approximation of the root.

11 How did we know when to stop? Answer: If the half-length b c of the interval [a, b] where the approximation lies is less than ɛ, this means that c will be at the distance from the actual root of no more than ɛ, so we can assume that x c

12 Bisection Method (Rough Algorithm) Given an initial interval [a 0, b 0 ] = [a, b], set k = 0 and proceed as follows: 1 Compute 2 If f (c k+1 )f (a k ) < 0, set c k+1 = a k (b k a k ). 3 If f (c k+1 )f (b k ) < 0, set a k+1 = a k, b k+1 = c k+1 4 Update k and go to Step 1. b k+1 = b k, a k+1 = c k+1

13 Theorem (Convergence and Error) Let [a 0, b 0 ] = [a, b] be the initial interval, with f (a)f (b) < 0. Define the approximate root as x n = c n = b n 1 + a n 1. 2 Then, there exists a root α [a, b] such that α x n ( 1 2 )n (b a). Moreover, to achieve the accuracy of α x n ɛ, it suffices to take n log(b a) log ɛ. log 2

14 Obviously, b n a n = 1 2 (b n 1 a n 1 ) By induction, b n a n = ( 1 2 )n (b 0 a 0 ) = ( 1 2 )n (b a). Then, α x n 1 2 (b n 1 a n 1 ) = 1 2 (1 2 )n 1 (b 0 a 0 ) = ( 1 2 )n (b 0 a 0 )

15 Since we want to have α x n ( 1 2 )n (b a), ( 1 2 )n (b a) ɛ If we solve this latter inequality for n, we get n log(b a) log ɛ. log 2

16 Example In our previous example, a = 1, b = 2, ɛ = Then, we need the following number of steps to reach the desired accuracy: n log(2 1) log(0.001) log 2 = log(0.001) log 2 =

17 Bisection Method input a,b,eps external f fa = f(a) fb = f(b) if fa*fb > 0 then stop n = fix((log(b-a) - log(eps))/log(2)) + 1 for i =1 to n do c = a + 0.5*(b-a) fc = f(c) if fa*fc < 0 then b = c fb = fc

18 else if fa*fc > 0 then a = c fa = fc else alpha = c return endif endif endfor

19 Bisection Method is a global method: it always converges to the root, no matter how far we start from the root (assuming f (a)f (b) < 0) Disadvantages: 1 a relatively slow method (compared to the other methods we will study) 2 it cannot determine those roots where the function is tangent to the x-axis (example: the solution to x 2 = 0)

20 Newton s Method classic algorithm for solving equations of the form f (x) = 0. much faster than Bisection Method, but it has its own limitations (we will investigate them later)

21 Motivation Let us start by considering the solution to In that case, f (x) = x 2 2. x 2 2 = 0. Clearly: the solution is α = 2. We will try to construct a sequence x n of numbers such that lim x n = α = 2. n

22 x Figure: y = x 2 2 For x 0, we can take an arbitrary value (a good idea is to pick something not too far away from the actual solution) Next, find where the tangent line at (x 0, f (x 0 )) intersects the x-axis, call that point x 1.

23 If we take, e.g. x 0 = 2, then the equation of the tangent line at that point is: f (2) = y f (2) x 2 which yields 4 = y 2 x 2 To find the intersection with the x-axis, set y = 0: We get x = 3/2, so set 4 = 2 x 2. x 1 = 3 2. In the next iteration, we construct the tangent line at (x 1, f (x 1 )), and its intersection with the x-axis will be x 2, etc.

24 We want a general formula for x n+1 in terms of x n. The equation of the tangent line at (x n, f (x n ) is f (x n ) = y f (x n) x x n. To find x n+1, we set y = 0 since it is the x-intercept of that tangent line: f (x n ) = f (x n) x x n. Solving this equation for x, we get: x n+1 := x = x n f (x n) f (x n ). Therefore, the formula for Newton s Method is x n+1 = x n f (x n) f (x n )

25 It is also possible to derive this formula using Taylor s Theorem. Expand f into a linear polynomial around x n : Set f(x)=0: f (x) = f (x n ) + (x x n )f (x n ) (x x n) 2 f (ξ). 0 = f (x n ) + (x x n )f (x n ) (x x n) 2 f (ξ). Assuming that the remainder is very small when we are sufficiently close to the solution, we have: 0 f (x n ) + (x x n )f (x n ), and when we solve for x, we get x = x n f (x n) f (x n )

26 Example Using Newton s Method, solve the equation x 6 x 1 = 0. First, f (x) = 6x 5 1. We use the recursive formula for Newton s Method to generate the first few approximations, starting with x 0 = 1.5: x n+1 = x n x 6 n x n 1 6x 5 n 1 For example, and x 1 = (1.5) 5 1 = x 2 = ( ) ( ) 5 1 =

27 We get the table n x n f (x n ) x n x n 1 α x n E E E E E E E E E E E E E E E E E E E-9 The actual solution is α = and x 6 equals α to nine significant digits Notice that Newton s Method may converge slowly at first, but as the iterates come closer to the root, the convergence accelerates significantly.

28 Newton s Method is not a global method; there are examples for which the convergence will be poor, or ven those for which we will not be able to achieve convergence. Sometimes, this can be rectified by changing the initial approximation x 0, but we may need to take a value for x 0 to be very close to the actual solution α.

29 Example Consider the function f (x) = 20x 1. 19x 2 y x 1 2 Figure: y = 20x 1 19x which has a root at α = 1/20 = 0.05.

30 Except for initial values x 0 very close to α, the first step will move away from the actual solution, instead of convergence. Here are the values of x 1 for a few initial choices x 0 : x 0 x Therefore, we need to use x 0 = 1/16 in order to get x 1 > 0.

31 Example If f (x) = arctan x, then arctan x = 0 has α = 0 as its only root. Suppose x 0 = Then, and x 1 = x 2 = which means that x 0 = x 2, and the values start oscillating and not converging to α.

32 General Fact: (which will be explained later) If f, f, f are all continuous near the root, and if f does not equal zero at the root, then Newton s Method will converge whenever the initial approximation is sufficiently close to the root. Moreover, this convergence is very fast, with the number of correct digits roughly doubling in each iteration.

33 For the Bisection Method, we were able to determine, before we started the algorithm, how many iterations were necessary to achieve a given desired accuracy ɛ. For Newton s Method, this is generally not possible. Question: When do we stop the iteration?

34 We want the error α x n to be less than ɛ. Problem: Generally, we do not know the exact value of the root α. If f is not zero near the root, we will be able to control the size of the error.

35 Error Estimate for Newton s Method By the Mean Value Theorem: f (α) f (x n ) = f (c n )(α x n ), where c n is some value between α and x n. Solve for α x n to get: α x n = f (α) f (x n) f (c n ) = f (x n) f (c n ) = f (x n 1) f (x n ) f (x n 1 ) f (x n 1 ) = (x n x n 1 )( f (x n) f (x n 1 ) f (x n 1 ) f (c n ) f (x n 1 ) f (c n ) Here, in the last step, we have used the formula for x n in terms of x n 1.

36 Now, α x n = C n (x n x n 1 ) for C n = f (x n )/f (x n 1 ) f (x n 1 )/f (c n ) So, the error α x n is a multiple of a computable quantity x n x n 1 Assuming that the method converges and that f (α) 0, then lim n C n = 1 The last limit is true, since both lim f (x n ) n f (x n 1 ) = lim f (x n 1 ) n f (c n ) = 1 Finally, lim n α x n x n x n 1 = 1.

37 Then, we can use a computable quantity x n x n 1 to measure convergence. In practice, this means that we can stop the algorithm when some relatively small multiple of this quantity is smaller than ɛ. For instance, when 3 x n x n 1 ɛ Another possibility is to stop the algorithm when f (x N ) becomes sufficiently small (close to zero). This is explained in Exercise 2 in 3.3.

38 Division Problem Consider the function Obviously, if f (α) = 0, then f (x) = a 1 x. α = 1 a. Therefore, if we want to divide, say c, by a, we can use c a = c 1 a, where we have computed 1/a from f (x). By changing the value of a (a 0), we are able to divide any two numbers.

39 If we apply Newton s Method to f (x), we get x n+1 = x n ( a x n 1 xn 2 ) = x n (2 ax n ). Question: When does the iteration converge and how fast? What initial guesses x 0 will work?

40 We have a = b 2 t p, where b [1/2, 1]. Therefore, we only need to compute reciprocals for the numbers in [1/2, 1]. We will define the residual by Then, the error is r n = 1 ax n α x n = e n = r n a = 1 a x n

41 In that case, we can get a recursive formula for r n : r n+1 = 1 ax n+1 = 1 a[x n (2 ax n )] = 1 a[x n (1 + r n )] = 1 (1 r n )(1 + r n ) = r 2 n From this, by induction: r n = r 2n 0. If r 0 < 1, then r n will rapidly approach zero. The error e n in the n-th step will satisfy e n+1 = ae 2 n = ae 2 n and the relative error will satisfy so that we get e n+1 α = (e n α )2 e n α = (e 0 α )2n = (r 0 ) 2n

42 So, we get the convergence as long as which is equivalent to r 0 < 1 0 < x 0 < 2 a = 2α

43 Question: How do we get this initial approximation x 0? One way to do that is to use linear interpolation: approximate the function y = 1/x with the straight line connecting the endpoints of the curve on [1/2, 1]: (1/2, 2) and (1, 1). The equation of the line passing through these two points is: y = p 1 (x) = 3 2x We can assume that It can be shown that x 0 = p 1 (a) = 3 2a 1 a p 1(a) 1 2.

44 So, Then, x 0 α 1 2. r 0 = a e 0 = a In terms of the relative error, we will have e 1 0 α 2 α 1 2. Therefore, only six iterations will produce relative accuracy of

45 Example We will test this method on a = 0.8, i.e.we are trying to approximate (0.8) 1 = The initial guess is Newton s Method produces x 0 = p 1 (0.8) = 3 2(0.8) = 1.4 x 1 = x 0 (2 (0.8)x 0 ) = x 2 = x 1 (2 (0.8)x 1 ) = x 3 = x 2 (2 (0.8)x 2 ) = x 4 = x 3 (2 (0.8)x 3 ) = 1.25

46 Example Instead of a linear, we can use some quadratic interpolant to generate the initial approximation x 0. For example, we can use p 2 (x) = 1 3 (8x 2 18x + 13) This parabola still passes through the endpoints (1/2, 2) and (1, 1) and is one approximation of the curve y = 1/x on [1/2, 1].

47 How can we estimate the error if we use this p 2 (x) in place of p 1 (x)? Define the error Then, E(x) = 1 x p 2(x) E (x) = 1 x 2 1 (16x 18) 3 We want to maximize E(x), so let s find the critical points: 0 = 1 x 2 1 (16x 18) 3 Therefore, 16x 3 18x = 0

48 The easiest way to solve this cubic equation is to use a computer algebra system (e.g. Maple); the solutions are: x 1 = 0.886, x 2 = 0.595, x 3 = Only x 1 and x 2 are in the interval [1/2, 1], so we only need to check those in addition to the endpoints: E(x 1 ) = E(x 2 ) = E(1/2) = 0 E(1) = 0

49 Therefore, E(x) is the upper bound for the error on [1/2, 1] Thus, the initial error is e 0 = 1 a p 2(a) which is better than generating the initial approximation using a linear interpolant. We get the accuracy of in only four iterations.

50 We saw that the error at each step of Newton s Method is related to the square of the previous error. In other words, the number of correct significant digits roughly doubles at each step. Theorem (Newton Error Formula) If f has continuous derivatives f and f on some interval I and f (α) = 0 for some α I, and x n+1 = x n f (x n) f (x n ), then there exists a point ξ n between α and x n such that (α x n+1 ) = 1 2 (α x n) 2 f (ξ n ) f (x n ).

51 Proof. Expand f in a Taylor linear polynomial around x n : f (x) = f (x n ) + (x x n )f (x n ) (x x n) 2 f (ξ n ), where ξ n is between x n and α. Substitute x = α into the expansion: 0 = f (x n ) + (α x n )f (x n ) (α x n) 2 f (ξ n ). If we divide both sides by f (x n ) and rewrite the equality, we get: (x n α) f (x n) f (x n ) = 1 2 (α x n) 2 f (ξ n ) f (x n ). Since the left-hand side is equal to x n+1 α, we get x n+1 α = 1 2 (α x n) 2 f (ξ n ) f (x n ).

52 Example If f is such that f (x) 3 for all x and f (x) 1 for all x and if the initial error in Newton s method is less than 1 2, what is an upper bound on the error at each of the first three steps? Solution: We are given that and that Then, for every n, α x 0 < 1 2, f (x) 3, f (x) 1. f (ξ n ) f (x n ) 3 1 = 3.

53 Now, α x 1 = 1 2 α x 0 2 f (ξ 0 ) f (x 0 ) < 1 2 (1 2 )2 3 = 3 8 = Next, we estimate the error of the second iteration α x 2 = 1 2 α x 1 2 f (ξ 1 ) f (x 1 ) < 1 2 (3 8 )2 3 = =

54 Finally, we give an upper bound for the error of the third iteration: α x 3 = 1 2 α x 2 2 f (ξ 2 ) f (x 2 ) < 1 2 ( )2 3 = =

55 The formula for the error estimate shows that the error in each step decreases quadratically. If our method is convergent then we can assume that lim x n = α, n f (x n ) f (α) f (x n ) f (α). Then, (α x n+1 ) C(α x n ) 2, where C = 1 f (α) 2 f (α). This last approximation can help us decide how we can choose the initial approximation x 0 to guarantee that Newton s Method will converge to α.

56 A fact from 3.6 which will not be proved (but whose justification involves the facts from the previous slide) states: Theorem Suppose f has continuous derivatives f and f and f (α) = 0. Define the ratio M = max x R f (x) 2 min x R f (x) (< ). Then, for any initial approximation x 0 such that α x 0 < 1 M Newton s Method converges to α.

57 Example It is easy to show (by graphing the function, say) that the equation 4x cos x = 0 has a solution in the interval [ 2, 2]. How should we choose the initial approximation relative to the actual solution α in order to have convergence of the Newton s method? We can consider the first two derivatives of f (x) = 4x cos x: Then, f (x) = 4 + sin x, f (x) = cos x. max f (x) = max cos x = 1 min f (x) = min 4 + sin x = 4 + ( 1) = 3

58 Therefore, M = 1 3, and in order to have convergence, we must have α x 0 < = 3.

59 We have seen that where (α x n+1 ) C(α x n ) 2, C = 1 f (α) 2 f (α). If we do have convergence, then α x n+1 lim n (α x n ) 2 = f (α) 2f (α) = C. which shows that the errors decrease quadratically

60 There is a more precise way to state this mathematically: Definition (Order of Convergence for Sequences) If x n is a sequence converging to α, so that α x n+1 lim n (α x n ) p = C for some nonzero constant C and some p, then p is called the order of convergence of the sequence.

61 Secant Method Newton s Method was based on approximating the graph of y = f (x) with a tangent line and then using the root of this line to approximate the root of f (x). Secant Method is a variation of this method. Instead of starting with one initial approximation x 0, and constructing the tangent line at that point, we start with two initial approximations x 0 and x 1 and construct a line through the points (x 0, f (x 0 )) and (x 1, f (x 1 )), which is called a secant line The intersection of this line with the x-axis will give us the next approximation x 2.

62 The equation of the line through (x 0, f (x 0 )), (x 1, f (x 1 )) is y = f (x 1 ) + (x x 1 ) f (x 1) f (x 0 ) x 1 x 0. The intersection of this line with the x-axis is: x 2 := x = x 1 f (x 1 ) x 1 x 0 f (x 1 ) f (x 0 ). Now, construct a secant line through (x 1, f (x 1 )) and (x 2, f (x 2 )), and its intersection with the x-axis will be the next iteration x 3, etc.

63 Repeating this process, we get the general formula for teh Secant Method, which is a recursive formula for x n+1 in terms of x n and x n 1 : x n+1 = x n f (x n ) x n x n 1 f (x n ) f (x n 1 )

64 There is an alternative way of deriving this formula, by modifying the formula for Newton s Method. The formula for Newton s Method is: x n+1 = x n f (x n) f (x n ). We can approximate the derivative in the denominator using the backward difference formula, where x h = x n 1 : Then, we get: f (x n ) = f (x n) f (x n 1 ) x n x n 1. x n+1 = x n f (x n ) x n x n 1 f (x n ) f (x n 1 )

65 Example Solve the equation using the Secant Method. x 6 x 1 = 0, The formula for the Secant method applied to the function is f (x) = x 6 x 1, x n+1 = x n (xn 6 x n x n 1 x 1) xn 6 x n xn x. n 1 We can choose the two initial approximations to be, e.g. x 0 = 2, x 1 = 1.

66 n x n f (x n ) x n x n 1 α x n E E E E E E E E E E E E E E E E E E E E E-7 As with the Newton s Method, the initial iterates do not converge rapidly, but as the iterates become closer to α, the speed of convergence increases.

67 Error Estimate Using techniques from calculus, one can show that α x n+1 = (α x n )(α x n 1 )[ f (ξ n ) 2f (ζ n ) ], where ζ n is between x n and x n 1, and ξ n is between the largest and the smallest of the numbers α, x n and x n 1. One can also show that: α x n+1 lim n α x n r = f (α) 2f (α) r 1 = C, where r = So, Secant method is slower than Newton s Method since its convergence is of order 1.62 (it is superlinear), while Newton s Method converges quadratically.

68 Secant Method: Algorithm input x0, x1, tol, n external f f0 = f(x0) f1 = f(x1) for i = 1 to n do x = x1 - f1*(x1 - x0)/(f1 - f0) fx = f(x) x0 = x1 x1 = x f0 = f1 f1 = fx if abs(x1 - x0) < tol then root = x1 stop endif endfor

69 Fixed-Point Iteration General method for one-point iteration formulas (such as Newton s method, for example) Example As our motivating example, consider the equation which has the root x 2 5 = 0 α = 5 =

70 Let s consider these four iteration methods for solving this equation: 1 x n+1 = 5 + x n x 2 n 2 x n+1 = 5 x n 3 x n+1 = 1 + x n 1 5 x 2 n 4 x n+1 = 1 2 (x n + 5 x n ) All four iterations have the property that if the sequence x n converges, it converges to α.

71 n x n (I1) x n (I2) x n (I3) x n (I4) In all four iterative methods (I1)-(I4), we used the same initial approximation x 0 = 2.5 So, we see that (I1) and (I2) do not converge, while the methods (I3) and (I4) converge to α.

72 All four iteration methods have the form x n+1 = g(x n ), for some continuous function g(x). If the iterates converge to the root α, then lim x n+1 = lim g(x n ) n n α = g(α) Thus, α is a solution of the equation x = g(x) and is called a fixed point of the function g(x)

73 Theorem (Fixed point existence and convergence) Suppose g is a continuous function on [a, b] such that a g(x) b, for all x [a, b]. Then: 1 g has at least one fixed point α on [a, b] 2 If there is a constant γ < 1 such that max x [a,b] g (x) = γ for all x, y [a, b], then α is unique and the iteration method x n+1 = g(x n ) converges to α for any initial approximation x 0 in [a, b].

74 Theorem (continued...) Also: The error estimate is: α x n γn 1 γ x 1 x 0. The convergence is at least linear: α x n+1 lim n α x n = g (α).

75 Sketch of proof: 1 Define the function It is continuous on [a, b] and f (x) = x g(x). f (a) 0, f (b) 0, so it must have a root α [a, b], by the Intermediate Value Theorem. 2 Suppose there are two solutions, α and β, in [a, b]: g(α) g(β) = g (ξ)(α β) for some ξ between α and β, and since g (x) < 1: g(α) g(β) < α β. On the other hand, since α and β are solutions which is a contradiction. α β = g(α) g(β),

76 The derivation of the error estimate in (3) is slightly more involved so we will omit it. Problem: In general, this theorem is rarely used directly. Reason: It is difficult to find an interval [a, b] on which the values of g are in the same interval. Theorem (Local Convergence for Fixed-Point methods) If g is continuous around α and g (α) < 1, then the fixed-point method will converge with all the conclusions (1)-(3) from the previous theorem on some interval [a, b] around α. In other words, x 0 has to be chosen sufficiently close to α in order to get the convergence of the fixed-point method.

77 Let us go back to the four methods we looked at at the beginning of this section and see why some of them converged, while the others did not. (I1) g(x) = 5 + x x 2, g (x) = 1 2x, g (α) = < 1 So, this method will not converge to 5. (I2) g(x) = 5/x, g (x) = 5/x 2, g (α) = 1 The theorem will not tell us what happens in this case, but from the table of results, we saw that the method did not converge (oscillates).

78 (I3) g(x) = 1 + x 1 5 x 2, g (x) = x, g (α) = = The iteration will converge and when x n is close to α. α x n α x n (I4) g(x) = 1 2 (x + 5/x), g (x) = 1 2 (1 5/x 2 ), g (α) = 0 Thus, the conditions for convergence are met. This is, in fact, Newton s method for this equation.

79 General Problem: Given an equation how should we rewrite it as f (x) = 0 g(x) = x in order to get a convergent fixed-point method. Answer: Try to set up g(x) in such a way that its derivative is smaller than 1 in a neighbourhood of the expected root.

80 x Example Given the function g(x) = 1 + e x find an interval [a, b] such that g([a, b]) [a, b]. Use this to solve the fixed point problem Solution: x = 1 + e x 4 y x 2 4

81 So, one such interval is [0, 2], for example. The derivative is g (x) = e x. Obviously, on [0, 2], so that 1 g (x) e 2 g (x) 1 on that interval. Therefore, the fixed-point problem will have a solution in [1, 2]. We can take as the initial approximation x 0 = 1.5 (for example).

82 x n+1 = 1 + e xn, x 0 = 1.5 n x n And we can continue calculating successive iterates, until we reach a satisfactory accuracy...

83 We see that various single-point iteration methods, including the Newton s Method, are special cases of this general fixed-point method. We mentioned that the fixed-point methods are at least linear in convergence, while we had seen that Newton s Method is quadratically convergent. Theorem Consider the fixed-point iteration x n+1 = g(x n ), where g has p continuous derivatives, and g(α) = α. If g (α) = g (α) =... = g (p 1) (α) = 0, but g (p) (α) 0, then the fixed-point method converges with order p for x 0 sufficiently close to α.