ROOT FINDING REVIEW MICHELLE FENG 1.1. Bisection Method. 1. Root Finding Methods (1) Very naive approach based on the Intermediate Value Theorem (2) You need to be looking in an interval with only one zero, where the function achieves values of opposite sign at the endpoints of the interval to guarantee that you find the correct zero (3) Loosely, the algorithm repeatedly cuts the interval in half, identifies which half the zero is in, and repeats the search in that half of the interval. Eventually, when the interval is small enough, we can say we are close enough to the zero and stop iteration. (4) Converges linearly 1.2. Fixed Point Iteration. p 0 given, p n g(p n 1 ) (1) Can convert a root finding problem to a fixed point problem (by solving for x) (2) Any function that maps [a, b] to itself ( x [a, b], g(x) [a, b]) will have at least one fixed point (existence) (3) Any function that maps [a, b] to itself, and which is also not too steep ( g?(x) K < 1 for all x?[a, b]) has a unique fixed point (uniqueness) (4) Fixed point iteration will work for any f meeting the previous conditions (5) Fixed point iteration works faster the lower that K is (K very small gives fast convergence, K close to 1 gives very slow convergence) (6) Proof of convergence follows from the existence and uniqueness conditions. To prove existence, we use the existence condition: Take h(x) = x?g(x). Notice if h(x) = 0 for some x, that x is a fixed point of g. If h(a) = 0 or h(b) = 0, we re done. Now suppose 1
2 MICHELLE FENG not. Then g(a) [a, b] a, so g(a) > a, h(a) < 0. Similarly, g(b) < b h(b) > 0. Then by the intermediate value theorem, there exists x [a, b] s.t. h(x) = 0, and we re finished. Now, we prove uniqueness. Suppose we have two distinct points, p, q which are fixed by g, and we have the uniqueness condition g?(x) K < 1 x [a, b]. Then p q = g(p) g(q) = g (ξ) p q K p q But since K < 1, this can only be true of p q = 0, done. 1.3. Newton s Method. p 0 given, p n = p n 1 f(p n 1) f (p n 1 ) (1) Based on Taylor approximation (specifically linear approximation) (2) Converges quadratically in many cases, fails to converge occasionally (3) Take a point, look at f at that point, draw a tangent line, and find the intersection of the tangent line with zero. Most likely, this will put you closer to the real zero, repeat. (4) Proof of convergence based on fixed point convergence? we only have con- vergence in an interval around p? this interval can be very very small! So if you start too far away, you might find that Newton?s method doesn?t converge at all. (5) Derivation: From Taylor s theorem: Since f(p) = 0 f(p) f(p n ) + f (p n )(p p n ) 0 f(p n ) + f (p n )(p p n ) p p n f(p n) f (p n ) Use the right quantity as our guess p n+1. Alternatively, we can consider Newton?s method graphically? given a point p n, we find the tangent line at p n, and look for it s x-intercept. Done this way, we note that the tangent line at p n goes through the point (p n, f(p n )), and has slope f?(p n ). Writing out the equation of this line, we have f(x) f(p n ) = f (p n )(x p n )
ROOT FINDING REVIEW 3 Solve for the x-intercept by plugging in (p n+1, 0) for (x, f(x)), then solve for p n+1 1.4. Secant Method. p 0, p 1 given, p n = p n 1 f(p n 1)(p n 1 p n 2 ) f(p n 1 ) f(p n 2 ) (1) An adaptation of Newton?s method, useful when you do not know what f? looks like (2) To derive the Secant method, we note that f (x) f(a) f(x) a x if x is close to a. Then replace f (p n 1 ) from Newton s method with the approximation f(p n 1) f(p n 2 ) p n 1 p n 2, and we re done. Alternatively, given p n 1, p n 2, draw a line between them. Let p n be the x-intercept of this line. (3) Converges superlinearly 1.5. Method of False Position. (1) Combines secant and bisection method (2) Like Secant method, requires two start points. Like bisection method, these start points should have opposite sign when f is applied. (3) Formula is the same as for secant method, except that instead of using p n 1, p n 2, we use p n 1 and either p n 2 or p n 3 (whichever one gives us opposite sign from f(p n 1 ) when f is applied). (4) Converges slower than secant method, but has better convergence guarantees (similar to bisection method) 1.6. Laguerre s Method. x 0 given, G = p (x k ) p(x k ), H = G2 p (x k ) p(x k ), a = (1) Used specifically for polynomials n G ± (n 1)(nH G 2 ), x k+1 = x k a (2) Derivation: recall a polynomial p(x) of degree k can be written p(x) = C(x x 1 )(x x 2 ) (x x n) where the x i are roots. Let G = d dx ln p(x) = p (x) p(x)
4 MICHELLE FENG H = d2 dx 2 ln p(x) = p (x) 2 p (x)p(x) (p(x)) 2 = G 2 p (x) p(x) Notice that using the form written before, we can write n 1 G = x x i H = i=1 n i=1 1 (x x i ) 2 Now, suppose that we guess x, and suppose that x x i = a for exactly one root x i, and x x j = b for all other roots x j. Then G = 1 a + n 1, H = 1 b a 2 + n 1 b 2 Then solving for a, we get n a = G ± (n 1)(nH G 2 ) and we can guess that x i = x a. (3) Converges super fast (cubically) for most polynomials, and almost always converges regardless of initial point (4) However, requires you to be using a polynomial, and to compute 2 derivatives. 1.7. Horner s Method. (1) Used for polynomials (2) Since Newton s method only finds one root, Horner s method gives you a way of finding other roots (3) Horner s method allows us to do polynomial long division (see examples on Wikipedia) (4) Use Newton s method to find the first root. Then divide the polynomial using Horner s method by (x x 1 ), where x 1 is the first root. Then use Newton s method to find the second root. Repeat until no roots remain. (5) Efficient for evaluating polynomials, but as a root finding method, it s constrained by the speed of Newton s (which to be fair is usually fast). (6) Additionally, small errors at each step can become magnified, so it isn t very accurate as the number of roots goes up.
ROOT FINDING REVIEW 5 1.8. Multiple Roots. (1) When a function f has multiple roots, Newton?s method converges very slowly (linearly in fact! We?ll prove this in a bit). (2) Loosely, linear convergence happens because if f has a multiple root p, f?(p) = 0, so that as we approach p, we?re looking at tangent lines that are nearly flat, and tangent lines that aren?t very good approximations of f. (3) Why does convergence depend on whether a root is multiple or simple? Let?s take a look at the function that we?re finding a fixed point of with Newton?s method: g(x) = x f(x) f (x) We can write out a Taylor expansion for g(x) around p: g(x) = g(p) + g (p)(x p) + g (ξ) (x p) 2 Now, let s examine g (p). Suppose f has a simple root at p, that is, f (p) 0. Then g (x) = 1 f (x)f (x) f(x)f (x) f (x)f (x) = f(x)f (x) f (x)f (x) But f(p) = 0, and f (p) 0, since p a simple root, so that g (p) = 0. Then g(x) = g(p) + g (ξ) (x p) 2 But then we have g(x) g(p) = g (ξ) x p 2 Plugging in p n = x and recalling that p is a fixed point of g, since p is a root, then g(p n ) g(p) = g (ξ) p n p 2 p n+1 p = g (ξ) p n p 2 p n+1 p lim n p n p 2 = g (p) 2 which gives precisely the formula for quadratic convergence. So what goes wrong when we have a multiple root? If we have a multiple root at p, we have f?(p) = 0. This means that when we take g?(p), we?re not going to get a convenient zero (you can compute on your own that g (p) 0. So now, when you look at the Taylor expansion, we can?t just ignore the linear g?(p)(x?p) term! Instead, we have g(x) = g(p) + g (ξ)(x p)
6 MICHELLE FENG and using the same argument as above, we have p n+1 p lim = g (p) n p n p so we have only linear convergence at best (in fact, g (p) < 1 will be true, so we ll have linear convergence precisely). (4) To fix the problem of linear convergence with multiple roots, we?re going to modify f into a function that Newton?s method will converge quadratically with. To do this, we have to pick a function that has only simple roots, and that also still preserves all the roots of f. To do this, we take µ(x) = f(x) f (x) You can prove that this has the characteristics we want by using the fundamental theorem of algebra. Now we can use Newton s method on µ(x) to get quadratic convergence, and the same roots.