CES739 Introduction to Iterative Methods in Computational Science, a.k.a. Newton s method

Transcription

1 CES739 Introduction to Iterative Methods in Computational Science, a.k.a. Newton s method Y. Zinchenko March 27, 2008 Abstract These notes contain the abridged material of CES739. Proceed with caution. 1

2 CES739 Introduction to Iterative Methods in Computational Science Instructor Yuriy Zinchenko, ITB221 Course content Newton s method is a fundamental iterative tool in many scientific and engineering applications, and is often the method of choice for solving F (x) = 0, where F (x) : R n R n. In particular, Newton s method is of paramount importance in many optimization problems. This course is dedicated to theoretical analysis of Newton s method, with focus on its computational complexity. We will present three distinct perspectives on this iterative procedure: the classical Kantorovich analysis, self-concordant setting, and the Smale s analysis based on asymptotic behavior of the function s higher-order derivatives. The course will culminate with the constructive proof of Bezout s theorem. Meeting times Tentatively, the class is scheduled to meet on Thursdays, 10:00-13:00, starting February 28, at ITB AB105. Grading policy The final grade will be decided based on several assignments distributed throughout the course. Course layout Tentative course material distribution is as follows: introduction, Newton s method in one dimension, function s behavior in a neighborhood of a critical point and Kantorovich analysis, introducing more geometry the self-concordant setting, function s behavior at a single point and Smale s analysis, insuring global convergence, Bezout s theorem. 2

3 1 Lecture one Goal: understand Newton s method, namely, its computational complexity, or in rough terms when does it work and how many flop s does it take. Newton s method is the method of choice for solving f(x) = 0 (1) where f : R n R n or C n C n, f sufficiently smooth. Examples: Consider f : R R, in the simplest case we have f(x) = ax + b, a 0, then f( x) = 0 iff x = b a, f(x) = ax 2 + bx + c, have explicit formula for x s.t. f( x) = 0, f(x) = a m x m + a m 1 x m a 0, no formulas exist for m > 4 (more precisely, cannot be solved algebraically in terms of a finite number of additions, subtractions, multiplications, divisions, and root extractions). How do we go about solving f(x) = 0 in this case? For f C, one way to find a root of f(x) = 0 is to use a bisection algorithm: suppose we have α < β so that f(α) < 0, f(β) > 0, by continuity of f its root satisfies x [α, β]. Consider x = α+β and set α = x if 2 f(x) < 0, and β = x if f(x) > 0 (if neither is the case, x is the root). Note that x approximates the root x, we still have x [α, β], and after 1 k iterations β α is reduced by a factor of. This is not too bad (2 k 2 k grows quite fast), but can we do better, namely, can we have a factor of 1 (note that the polynomials above are C )? 2 2k Figure 1: Bisection for root finding. Recall: Using Taylor s expansion of f C m we can write f(x + x) = f(x) + f (x) x + f (x) x2 2! f (ξ) xm m!

4 where ξ [x, x + x]. Idea: Since f(x+ x) f(x)+f (x) x (and even more so as 0), and we are very comfortable solving linear equations, replace f(x + x) with its linearization, solve f(x) + f (x) x = 0 for x, set x := x + x and repeat the whole procedure again, until x is close enough to the root of f(x) = 0 (say, f(x) 0 or x remains almost the same for a few consecutive iterations). The above procedure is called Newton s method. To set the notation, let N (x) = x (f (x)) 1 f(x) (2) be called Newton s map. Then the Newton s method consists of iteratively applying N starting at some point x 0 to generate a sequence x 0, x 1 = N (x 0 ), x 2 = N (x 1 ) = N 2 (x 0 ),, x k = N k (x 0 ),, and we say Newton s method converges when the sequence above does. 1.1 Complex dynamics of (simple) Newton s method Consider f(z) = z 3 1 over C, f(z) = 0 has three roots z 1,2,3 = 1, 1 i ±. We want to understand how does Newton s method behave depending on the starting point z 0 C. For our particular choice of f(z) = z 3 1, N (z) = 2z z 2 Figure 2: Roots of f(z) = 0 on C. 4

5 First hypothesis: f(z) is analytic (and so is C ), thus δ > 0 may be chosen small enough to make a linearization of f(z + z), f(z) + f (z) z, arbitrary close to f(z + z) in a small enough neighborhood of a root z i, say z z i < δ, for all z, z + z < δ. In other words, the neighborhood of z i may be chosen small enough so that the linearization of f(z) is an extremely good surrogate for f(z) within this neighborhood. Note that f ( z i ) 0, so N (z) is well defined for all z : z z i < δ for δ small enough. Then it is natural to assume that Newton s method converges to the root z i in such a neighborhood (in fact, we will prove this in a bit). Such a neighborhood will be referred to as a basin of attraction of z i. A hypothetical question: If C is partitioned into only three such basins (one for each root), how would the picture look like? A natural simple guess about these basins would be, on the first pass, to assume the basins are connected components of C. Then noting the rotational symmetry of roots of f(z) (i.e., we can turn the picture clockwise by 2π without changing it), it is natural to guess that the partitioning 3 into the basins would look as follows: Figure 3: Guessing the partitioning of C: blue z 1, green z 2, yellow z 3. In fact, this is not a bad guess and is almost correct (if you look at the picture from far enough). So, what could we have missed? Observe that N (0) is not defined. Are there any more pathological points? Consider a pre-image of 0 under N, N 1 (0), giving rise to three { points 1 3 1, 2 ( 1 ± i )}, so N ( ) is not defined at either of these three points (after one iteration Newton s method stalls). By induction, it is conceivable to assume that the pre-image of 0 under N k has 3 k distinct points, etc. The moral of the story is that C is populated by many such 5

6 pathological points (assuming we would run Newton s methods until we either find a root or it breaks down), and the true dynamics of Newton s method is far more complicated as compared to what it first seems. The true basins look as follows (from hiddendimension.com website), corresponding to a fractal, which, in rough-terms, is a self-similar figure Figure 4: Partitioning of C into three basins of attraction. which resembles itself when magnified. We can also try to visualize the speed of convergence of Newton s method for finding the roots of f(z) = 0. One well-accepted procedure for doing so goes as follows. We will pick a starting point z 0 C and assign its color based on how well Newton s method progressed after K 1 iterations; starting with initial color s 0 = 0 ( we will update ) its value after 1 every iteration of N as s k+1 = s k + exp z k+1 z k, thus measuring the progress made by Newton s iterates. Note that if iterates are close together, meaning that the method probably start to converge, every new color increment would stay small due to the negative exponent. After the method makes K iterations, we would place a point z 0 on the graph coloring it with the color s K. The procedure can be used to color the part of the complex plain we are interested in. So, why Newton s method? If nothing else comes to mind, look at pretty pictures. Two MATLAB code fragments below illustrate Newton s method for f(z) = z 3 1 and compute approximate basin of attraction for the root z 1 = 1 and visualize the speed of convergence of N : fragment 1 clear all 6

7 Figure 5: Speed of convergence of Newton s method as a fractal. itermax=20; z=2*(cos(pi/3)+sin(pi/3)*i); figure, hold on, grid on, xlim([-2,2]); ylim([-2,2]); w=[0:.01*pi:2*pi]; plot(cos(w),sin(w), c-- ); plot(cos(2*pi/3),sin(2*pi/3), m* ); plot(cos(-2*pi/3),sin(-2*pi/3), m* ); plot(1,0, m* ); pause plot(real(z),imag(z), r* ); pause; plot(real(z),imag(z), b* ); for iter=1:itermax z = (2*z.^3 + 1)./(3*z.^2); plot(real(z),imag(z), r* ); pause plot(real(z),imag(z), b* ); end hold off fragment 2 clear all dx=.00001; xl=dx; xr=.005; itermax=100; eps=.01; [zr,zi] = meshgrid(xl:dx:xr,xl:dx:xr); z=complex(zr,zi); zcol=zeros(size(abs(z))); for iter=1:itermax zn = (2*z.^3 + 1)./(3*z.^2); zcol = zcol+exp(-abs(1./(zn-z))); z=zn; 7

8 end figure; mesh(zr,zi,zcol); view(2) indz=find((abs(z-1)<eps)); zr=zr(indz); zi=zi(indz); figure; plot(zr(:),zi(:), b. ) end 1.2 Few applications and basic local analysis Newton s method application examples: pictures as above, optimization; consider finding min x f(x) over R n, we assume f C 3 and convex; a necessary and sufficient condition for x to be a minimizer of f is f ( x) = 0, so we can apply Newton s method to search for a root of f (x) = 0, path-following, e.g., homotopy methods for finding roots of a polynomial; consider finding all roots x i, i = 1,..., m of a univariate polynomial p(x) of degree m, note that we can write p(x) = a m i (x x i); start with some other polynomial of degree m with all the roots known, e.g., q(x) = i (x i) and consider h(α; x) = αp(x) + (1 α)q(x), α [0, 1]; for a fixed α, h(α; x) is a polynomial of degree m and thus have m roots x i (α) and h(0; x) = q(x), h(1; x) = p(x); assuming that the roots of a polynomial are continuous functions of its coefficients, we may attempt to follow each of the curve x i(α) as α goes from 0 to 1 with Newton s method to ultimately compute the (approximate) roots x i, i = 1,..., m (consider incrementing α by a small perturbation ɛ > 0, if ɛ is small enough, the roots of h(α; x) = 0 must give a good approximation to the roots of h(α + ɛ; x) = 0, and thus each x i(α) would provide a good starting point for Newton s method to find an arbitrary-close approximation to x i (α + ɛ); now keep on incrementing α till you reach 1). Now it is time for our first take at local convergence analysis of the method. Theorem 1.1. Let f : R R, f C 2, and let x be such that f( x) = 0, f ( x) 0. Then δ > 0 such that x : x x < δ we have for some K(δ) > 0, and x N (x) K(δ) x x 2 N k (x) [ x δ, x + δ], k > 0. Proof. Note that since f ( x) 0 and f C 2, there exists δ > 0 such that f (x) 0 for x x < δ, and for some K(δ) > 0 we have f (x) < 2K(δ), f (x) 2δK(δ) < 1. With such δ in mind, at least N (x) is well-defined for any x : x x < δ. For compactness we omit the arguments x and x in subsequent notation when evaluating f, f, f simply writing f for f(x), f for f( x), etc. 8

9 Consider x N (x) = x f 1 f (x f 1 f) = ( x x) f 1 ( f f) = f 1 (f ( x x) ( f f)) and denoting f t : t f( x + t( x x)), with f t = f ( x + t( x x)) ( x x) and f t = f ( x + t( x x)) ( x x) 2, we further write 1 (f f 1 ( x x) f t(ξ)dξ) 0 ( 1 = f 1 (f t(0) f t(ξ))dξ) max χ [x, x] f (χ) 2f ( x x) 2 K(δ)( x x) 2. Finally, note that with the choice of δ as above, N (x) [ x δ, x + δ]. 0 Remark 1.1. Asymptotically we have K(δ) f ( x) as δ 0. 2f ( x) Remark 1.2. The theorem implies local quadratic convergence of Newton s method in a small neighborhood of the root; in rough terms, the number of correct digits in the approximation of x doubles every iteration. Remark 1.3. The assumption of f C 2 may be relaxed into f being Lipschitz continuous. 1.3 Assignments Due next week in class. Scaling-invariance of Newton s method: consider f : R n R n, f C 2 with non-zero derivative, A, B two invertible n n matrices. Show that, Newton s iterates for function f starting at a point x 0 are exactly the same as the Newton s iterates for a function Af(Bx) starting at a point B 1 x 0. Let f : R R be C m. We will consider the behavior of Newton s method around a root x of f when f ( x) = 0. x is called a root of multiplicity m if f (k) ( x) = 0, k = 0,..., m 1 and f (k) ( x) 0. Show that if f has a root x of multiplicity m, we can write f(x) = (x x) m h(x) where h is continuous and h( x) 0. Next, show that in the small enough neighborhood of x Newton s method will converge and asymptotically x x k+1 x x k m 1 (hint: follow the convergence m proof from the class). 1.4 Additional reading and references Fractals and Newton s method Newton s basics nice overview paper by T.B. Polyak, Newton Kantorovich method and its global convergence (google for it) 9

10 2 Lecture Kantorovich s analysis of Newton s method I Recall real vector space, metric and norm, Cauchy sequence and complete metric space, k th derivative of f : R n R n as a k-linear function, induced norm of f (k) as f (k) = an important consequence is that sup f (k) (x 1, x 2,..., x k ) ; x i =1,i=1,...,k f (k) (x 1, x 2,..., x k ) f (k) x i. Aside: If you are uncomfortable with f : R n R n and its (multivariate) derivatives, for now think of all the functions being univariate real functions (simply replace norms with absolute values); you can digest the proofs in R n on your own at a later point. A few general comments: Does the method always converge? No, even if f : R R, f C 2, f 0 and is subject to further nice restrictions as below, f is monotone, e.g., f(x) = arctan(x), far from x = 0 (in red), moreover, the method may exhibit cycling behavior, i Figure 6: Divergence of N k (x 0 ) for f(x) = arctan(x). f is convex, but no guaranteed x s.t. f( x) = 0, e.g., f(x) = x 3 x + 3 starting at x 0 = 0 (in blue), what about convex monotone f? Think about it. 10

11 Figure 7: Cycling of N k (x 0 ) for f(x) = x 3 x + 3. One of the strongest potential criticisms of Theorem 1.1 is that we need to know x, that is, we cannot say if the method actually started to converge quadratically or how close we are to x without known x upfront. The first comment above is usually addresses by introducing the so-called regularized Newton s method, while the second comment is addressed, in particular, with Kantorovich s approach this is what we attempt to understand next. The end result we are going to develop over the next two lectures is: Theorem 2.1. Let f be C 2 on {x : x x 0 < R}, R > r > 0. Assume If and there exists (f (x 0 )) 1 =: f 0 1, f 0 1 f(x 0 ) η, f 0 1 f (x 0 ) K on {x : x x 0 r}. h = Kη < 1 2 r 1 1 2h η, h then (1) has a root x so that N k (x 0 ) x, the Newton s iterates satisfy x N k (x 0 ) 1 2 k (2h)2k η h, k 0, and the root is unique in the ball {x : x x 0 r} if r < h h We start building towards Theorem 2.1 by first observing for f : R n R n, f C 2, η. x = N (x) (3) 11

12 implies f(x) = 0. In what follows we first analyze the equation of the form x = S(x) (4) and conditions for the convergence of a sequential approximation procedure generating x 0, x 1 = S(x 0 ), x 2 = S(x 1 ),..., x k+1 = S(x k ),..., (5) to generate an approximate to the root of the equation (4). Together with (4) we consider a univariate function Φ(t) and a corresponding equation t = Φ(t). (6) Let S : R n R n be C 1 defined on {x : x x 0 < R} and let Φ be differentiable on [t 0, t ] (t = t 0 + r < t 0 + R). We say (6) majorizes (4) or Φ majorizes S if S(x 0 ) x 0 Φ(t 0 ) t 0, (7) S (x) Φ (t), for x x 0 t t 0 r. (8) The function Φ will be used to measure the progress of the sequential approximation scheme (5); you can think of Φ as assigning a potential to an iterate x k that allows measure the proximity between x k and x k 1 in terms od Φ. We establish the following Theorem 2.2. Let S, Φ be as above with Φ majorizing S. Furthermore, suppose (6) has a solution in [t 0, t ]. Then (5) converges to a root x of (4), and x x 0 t t 0, where t is the smallest root of (6) in [t 0, t ]. Proof. We start by considering a sequential approximation scheme t k+1 = Φ(t k ), k = 0, 1, 2,... applied to (6). We will show that the sequence {t k } converges to the smallest root t of (6). Later in the proof we will use this convergent sequence to establish a convergence of (5) by exhibiting its majorizing Cauchy sequence. W.l.o.g. Φ(t 0 ) > t 0, since if t 0 = Φ(t 0 ) then (7) implies x 0 is a root of (4). Observe that (8) and Mean-Value Theorem imply Φ is nondecreasing and thus t k+1 t k, k. Also, if t is a root of (6), then t k < t: t 0 < t and t t k = Φ( t) Φ(t k 1 ) = t Φ (τ)dτ 0. So, t t k t where t k 1 is the smallest root of (4) in [t 0, t ], and, in particular, t k form a Cauchy sequence. Now, consider (4) together with (5). By (7) we have x 1 x 0 t 1 t 0 and x 1, x 0 {x : x x 0 r}. We show that a similar relationship 12

13 t 0 t 1 t t Figure 8: Sequential approximating sequence for t = Φ(t). holds for all k by induction. Assume x i {x : x x 0 r} and x i x i 1 t i t i 1, for i = 1,..., k 1. Consider x k x k 1 = S(x k ) S(x k 1 ) = xk 1 x k 2 S (ξ)dξ (where the integral is along a line segment [x k 2, x k 1 ]) = = 1 0 (S(x k 2 + τ(x k 1 x k 2 ))) τ dτ S (x k 2 + τ(x k 1 x k 2 ))(x k 1 x k 2 )dτ 0 Φ (t k 2 + τ(t k 1 t k 2 ))(t k 1 t k 2 )dτ (by (8) and observing x k 2 + τ(x k 1 x k 2 ) x 0 τ(x k 1 x k 2 ) + x k 2 x k 3 + x k 3 + x 0 τ(t k 2 t k 1 ) + t k 2 t k 3 + t 0 = τ(t k 2 t k 1 ) + t k 2 t 0 r = 1 0 by inductive hypothesis) (Φ(t k 2 + τ(t k 1 t k 2 ))) τ dτ = Φ(t k 1) Φ(t k 2 ) = t k t k 1, so indeed, x k x k 1 t k t k 1. In addition, x k x 0 = x k x k 1 + x k 1 x 0 t k t k 1 + t k 1 t 0 = t k t 0 < r. Finally, we show that x k form a Cauchy sequence that converges to 13

14 a root of (4): consider k 1, n 0, x k+n x k = x k+n x k+n 1 + x k+n 1 x k t n+k t k, so x k converges, say to x. By continuity of S, taking a limit in (5) we get so indeed, x is a root. x = S( x), A variant of Theorem 2.2 will be used applied to (3) to show the convergence of Newton s method with no a priori knowledge about the location of the root of (1). 2.2 Assignments show for a fixed k 1, (k-linear functions) f (k) form a vector space, verify that the induced norm is indeed a norm, i.e., satisfies norm axioms, for a square n n matrix A, viewed as a linear operator from R n to R n, x Ax, find the expression for induced norms where the norm used on R n corresponds to Euclidean norm, x 2 = i x2 i, l 1 -norm, x 1 = i x i, l -norm, x = max i x i, show that if f : R R is convex, monotone decreasing and has a root, then the Newton s method converges to this root. 2.3 Additional reading and references you may find the complete analysis and much more in the book by Kantorovich and Akilov, Functional Analysis 14

15 3 Lecture Kantorovich s analysis of Newton s method II Corollary 3.1. If the conditions of Theorem 2.2 are satisfied and, in addition, t [t 0, t ] is the unique stationary point for (6) with Φ( t) t, then the solution to (4) is also unique on {x : x x 0 r}. Proof. To be filled in. Next we apply the result of Theorem 2.2 to the analysis of Newton s method. In particular, we establish two existential-type results for the so-called modified Newton s method N 0(x) = x (f (x 0)) 1 f(x), and original Newton s method, N (x) = x (f (x)) 1 f(x); the results are existential in a sense that we still have to exhibit a majorizing function. We start by analyzing modified method first, and then use some part of the analysis to understand Newton s method itself. Remark 3.1. One may view modified Newton s method not only as an artificial ingredient introduced only to simply the analysis. In some situation the inversion of f is a very hard task by itself (in fact, as an example, the dominating work in the so-called interior-point methods in convex optimization falls precisely on inverting such a derivative), therefore it might be advantageous to do such an operation only once, if it is possible, at the cost of slower convergence as the main theorem of this section will illustrate. Together with equation (1) we consider Ψ(t) = 0 (9) where Ψ : R R is sufficiently smooth. Namely, as before, we assume f C 2 on {x : x x 0 r} and is defined on {x : x x 0 R}, R > r, and let Ψ be C 2 for t [t 0, t ], t = t 0 + r. In a way, the first result simply paraphrases Theorem 2.2 for the modified Newton s method, noting that x = f(x) is equivalent to x = N x. Theorem 3.1. If the following conditions hold true Γ 0 = f (x 0 ) 1, (10) c 0 = 1 > 0, Ψ (t 0) (11) Γ 0 f(x 0) c 0Ψ(t 0), (12) Γ 0 f (x) c o Ψ (t), for x x 0 t t 0 r, (13) equation (9) has a root in [t 0, t ], (14) then the modified Newton s method for (1) and (9), started at x 0 and t 0 respectively, converges to the roots x, t of both equations, and x x 0 t t 0, where t is the smallest root in the interval [t 0, t ]. 15

16 Proof. To be filled in. Analogously, we can obtain a convergence result for Newton s method itself as the following Theorem 3.2. Let f, Ψ satisfy the conditions of Theorem 3.1. Then the sequence of Newton s iterates N k (x 0 ) x where x is the root of (1). To both Theorems 3.1 and 3.2 we can easily establish a corollary regarding the uniqueness of the solution Corollary 3.2. If in addition to satisfying the conditions of Theorem 3.1 we have that Ψ(t ) 0 and t [t 0, t ] is the unique root of (9) in [t 0, t ], then the root x is the unique root of (1) in the ball {x : x x 0 r}. The corollary is easily established using Corollary 3.1 and observing that Ψ(t ) 0 is equivalent to Φ(t ) t. Remark 3.2. The conditions of Corollary 3.2 are trivially met if we let t = t, the smallest root of (9) in [t 0, t ], thus guaranteeing a unique solution in the ball {x : x x 0 t t 0 }. Finally, we are ready to establish the main result Theorem 3.3. Let, as before, f be defined on {x : x x 0 R} and be twice continuously differentiable on {x : x x 0 r}, r < R. In addition, assume If and Γ 0 = f (x 0 ) 1, (15) Γ 0 f(x 0 ) η, (16) Γ 0f (x) K, x {x : x x 0 r},. (17) h = Kη 1 2 r r 0 = 1 1 2h η, h then (1) has a solution x so that N k (x 0 ) x and N0 k (x 0 ) x as k, with x x 0 r 0. Moreover, if or h < 1/2, r < r 1 = h h h = 1/2, r r 1, then x is the unique solution in the ball of radius r around x 0. The speed of convergence of N k (x 0 ) is characterized by x N k (x 0) 1 η 2 k (2h)2k, k = 0, 1, 2,..., (18) h and for N k 0 (x 0 ) with h < 1/2 we can write η, x N k 0 (x 0 ) η h (1 1 2h) k+1, k = 0, 1, 2,

17 Proof. On [0, r] consider Ψ(t) = Kt 2 2t + 2η = Kt 2 2t + 2h K = h η t2 2t + 2η. Clearly, Ψ and f satisfy the conditions (10) (13) of Theorem 3.1; also, (14) is met, since the roots of Ψ(t) = 0 are and r 0 = 1 1 2h h r 1 = h η h are both real under the assumption h 1/2, while r r 0 guarantees the existence of the root in [0, r] with r 0 being the smallest root, and r < r 1 guarantees its uniqueness, implying the uniqueness of x in the ball around x 0 of radius r (by Corollary 3.2). Since t = r 0, x x 0 r 0 follows immediately. To establish the estimates bounding the speed of convergence, it is enough to consider only the sequences of corresponding values of t k that result from applying the original and the modified Newton s method to solving (9). (Note that any pair (x k, t k ) may be considered as an initial approximate ( x 0, t 0 ) to the roots ( x, t) in the sequential approximation scheme, and use Theorem 3.1 or 3.2.) First consider N k (x 0). Denote t k+1 = t k Ψ(t k) Ψ (t k, k = 0, 1,..., ) η c k = 1 Ψ (t k ), η k = c k Ψ(t k ) = t k+1 t k, K k = c k Ψ (t k ) = 2Kc k, h k = K k η k, and express this quantities in terms of the same quantities but with one index less. By Taylor s formula η k = c k Ψ(t k ) = c k Ψ(t k 1 + η k 1 ) ] Noting we get = c k [Ψ(t k 1 ) + Ψ (t k 1 )η k Ψ (t k 1 )ηk 1 2 [ ηk 1 = c k + η k ] c k 1 c k 1 2 Kk 1 c k 1 η2 k 1 = c k 2c k 1 K k 1 η 2 k 1. c k 1 = Ψ (t k ) c k Ψ (t k 1 ) = Ψ (t k 1 ) + Ψ (t k 1 )η k 1 Ψ (t k 1 ) = 1 K k 1 η k 1 = 1 h k 1 η k = 1 2 K k ηk 1 2 = η k 1 h k 1. 1 h k h k 1 17

18 Similarly, Now, from the above c k K k 2c k K = 2Kc k 1 c k 1 = K k 1. 1 h k 1 ( hk 1 ) 2. h k = K k η k = h k 1 From the last expressions for η k and h k, accounting for h k 1/2, we get and, consequently, together with η k h k 1 η k 1, h k 2 (2h k 1 ) 2, h k 1 2 (2h0)2k = 1 2 (2h)2k η k h k 1 η k 1 h k 1 h k 2 η k 2 h k 1 h 0 η k (2h)2k 1 η. Finally, according to t k+1 t k = η k, we find t t k = (t k+1 t k ) + (t k+2 t k+1 ) + = η k + η k ( 1 2 k (2h)2k η (2h)2k + 1 ) (2h)2k k (2h)2k 1 η = 1 2 k (2h)2k η h. And thus, we get the estimate (18). Considering the error estimate for the modified method, assume h < 1/2. Using the same Φ is in the proof of Theorem 3.1, we can write Assignments Check that the conditions of Theorem 3.1 in the proof of Theorem 3.3 are indeed satisfied. 3.3 Additional reading and references There seems to be an excellent (abridged) approach to Newton s method analysis by Ortega, found at JSTOR (chances are you would need to login with McMaster library credentials to get access to it). 18