CONVERGENCE BEHAVIOUR OF INEXACT NEWTON METHODS

MATHEMATICS OF COMPUTATION Volume 68, Number 228, Pages 165 1613 S 25-5718(99)1135-7 Article electronically published on March 1, 1999 CONVERGENCE BEHAVIOUR OF INEXACT NEWTON METHODS BENEDETTA MORINI Abstract. In this paper we investigate local convergence properties of inexact Newton and Newton-lie methods for systems of nonlinear equations. Processes with modified relative residual control are considered, and new sufficient conditions for linear convergence in an arbitrary vector norm are provided. For a special case the results are affine invariant. 1. Introduction We consider the system of nonlinear equations (1) F (x) =, where F is a given function from R N to R N, and we let F (x) denote the Jacobian of F at x. Locally convergent iterative procedures commonly used to solve (1) have the general form: For =step 1 until convergence do Find the step which satisfies (2) B = F (x ) Set x +1 = x + where x is a given initial guess and B is a N N nonsingular matrix. The process is Newton s method if B = F (x ), and it represents a Newton-lie method if B = B(x ) is an approximation to F (x ) ([1], [2]). In [3] and [4] iterative processes of the following general form were formulated: For =step 1 until convergence do Find some step s which satisfies r (3) B s = F (x )+r, where F (x ) η Set x +1 = x + s. Here {η } is a sequence of forcing terms such that η < 1. These iterative processes are called inexact methods. In particular, we obtain inexact Newton methods taing B = F (x ) and inexact Newton-lie methods if B approximates the Jacobian F (x ). We remar that inexact methods include the class of Newton iterative methods ([1]), where an iterative method is used to approximate the solution of the linear systems (2). Received by the editor January 23, 1997 and, in revised form, January 6, 1998. 1991 Mathematics Subject Classification. Primary 65H1. Key words and phrases. Systems of nonlinear equations, inexact methods, affine invariant conditions. 165 c 1999 American Mathematical Society

166 BENEDETTA MORINI For inexact Newton methods, local and rate of convergence properties can be characterized in terms of the forcing sequence {η }. Let denote any vector norm on R N and the matrix subordinate norm on R N N. In [3] it is shown that, if the usual assumptions for Newton s method hold and {η } is uniformly less than one, we can define a sequence {x } linearly convergent to a solution x of (1) in the norm y = F (x )y. Recently, several authors (see e.g. [5], [6], [7], [8]) have proposed applications of inexact methods in different fields of numerical analysis and pointed out difficulties in applying linear convergence results of [3]. In fact, such results are normdependent and is not computable. Then, they focused on the analysis of the stopping relative residual control r / F (x ) η and its effect on convergence properties. In this paper we consider inexact methods where a scaled relative residual control is performed at each iteration. Further, we determine conditions that assure linear convergence of inexact methods in terms of a sequence of forcing terms uniformly less than one and for an arbitrary norm on R N. Such schemes include as a special case inexact methods (3). The results obtained are valid under the assumption of widely used hypotheses on F and merge into the theory of Newton s and Newtonlie methods in the limiting case of vanishing residuals, i.e. η =foreach. Further, for a special case, such conditions of convergence are affine invariant and in agreement with the theory of [9]. 2. Preliminaries In this section we analyze local convergence results given in [3]. The following definition of rate of convergence can be found in [1]. Definition 2.1. Let {x } R N be any convergent sequence with limit x.then x x with Q-order at least p, p 1, if x +1 x = O( x x p ),. (4) Moreover, consider the quantity Q p = Q p ({x },x ), called the Q-factor: Q p ({x },x ) = lim sup x +1 x x x p. In [1] it is noted that if Q p <, then for any ɛ> there exists a such that (5) x +1 x α x x p,, where α = Q p + ɛ. With regard to p = 1 and for a given norm, the process has Q-linear convergence if <Q 1 <1, Q-sublinear convergence if Q 1 1. If, for p = 1, there exist α<1and such that (5) holds, convergence is Q-linear in that norm ([1]). In [3] the following result is proved: Theorem 2.1 ([3], Th. 2.3)). Let F : R N R N be a nonlinear mapping with the following properties: (i) there exists an x R N with F (x )=; (ii) F is continuously differentiable in a neighborhood of x,andf (x )is nonsingular.

CONVERGENCE BEHAVIOUR OF INEXACT NEWTON METHODS 167 Assume that η η max <t<1. Then there exists ɛ>such that, if x x ɛ, the sequence {x } of inexact Newton methods converges to x. Moreover, the convergence is linear in the sense that x +1 x t x x,. This thesis holds for inexact Newton-lie methods too, if we assume that B is a good approximation to F (x ), i.e. B F (x ) γ and B 1 F (x ) 1 γ, where γ is a suitable small quantity. Theorem 2.1 assures that inexact Newton methods converge Q-linearly in norm and (6) lim sup x +1 x x x t. Convergence conditions in norm can be easily derived with slight changes in the proof of Theorem 2.1. In particular, since y F (x ) 1 y F (x ) y, we obtain (7) Q 1 ({x },x ) = lim sup x +1 x x x cond(f (x )) t, where cond(f (x )) = F (x ) F (x ) 1 is the condition number of F (x ). This observation points out an interesting relation between the Q 1 -factor in norm and the number cond(f (x )) t, and shows that convergence in norm may be sublinear if cond(f (x )) t 1. Since (7) employs x, it does not allow us to evaluate the upper bound t on {η } such that convergence is assured to be Q-linear in norm. On the other hand, in the following section we will show that, for a well nown class of functions F and for an arbitrary norm on R N, new linear convergence conditions leading to computable restrictions on the forcing terms can be derived. As in Theorem 2.1, the forcing sequence is uniformly less than one. 3. Inexact methods with scaled residual control In this section we assume that F belongs to the class F(ω, Λ ) of nonlinear mappings that satisfy the following properties ([6], [9]-[13]): (i) F : D(F ) R N, D(F ) an open set, is continuously differentiable in D(F ); (ii) there exists an x D(F ) such that F (x )=andf (x ) is nonsingular; (iii) S(x,ω)={x x R N x x ω} D(F), and x is the only solution of F (x) =ins(x,ω); (iv) for x, y S(x,ω), F (x ) 1 (F (y) F (x)) Λ y x. From [1], [2] and [11] we have the following lemmata: Lemma 3.1 ([1]). Let F F(ω, Λ ). Then there exists σ<min{ω, 1/Λ } such that F (x) is invertible in S(x,σ) and for all x, y S(x,σ) we have (8) E(y, x) =F (x) 1 (F (y) F (x)), E(y, x) Λ σ y x, where Λ σ =Λ /(1 Λ σ).

168 BENEDETTA MORINI Lemma 3.2 ([2], Lemma 4.1.9). Let F : R N R N be continuously differentiable in an open convex set D R N. For any x, x + z D, F (x + z) F (x) = F (x+tz)z dt. Lemma 3.3 ([11], Lemma 3.1). Let x, y S(x,σ).Then E(x +t(x x ),y) dt Λ σ ( x x (9) + m), 2 where m =min{ x y, x y }. In the following, for brevity, we write Λ = Λ σ, S = S(x,σ). Now let us consider inexact methods of the form: For =step 1 until convergence do Find some step ŝ which satisfies (1) ˆB ŝ = F (ˆx )+ˆr, where P ˆr P F (ˆx ) θ Set ˆx +1 =ˆx +ŝ where ˆx is a given initial guess and P is an invertible matrix for each. IfP =I for each, (1) reduces to (3). It is worth noting that residuals of this form are used in iterative Newton methods if preconditioning is applied, and that P changes with index if B does. First we examine inexact Newton and modified inexact Newton methods that correspond to ˆB = F (ŷ )withŷ =ˆx and ŷ =ˆx,foreach, respectively. Theorem 3.1. Assume ˆB = F (ŷ ),, in (1). Let ˆx S, ˆx x δ, ν =θ cond(p F (ŷ )) with ν ν<ν,andµ= Λ 2δ(1 + ν). Then the iterates of (1) are well-defined and {ˆx } converges to x if (11) for inexact Newton methods and (12) α = µ(µ + ν)+ν<1, α = µ(µ + ν)+2µ+ν<1, for modified inexact Newton methods. Further, for sufficiently large we have (13) (14) ˆx +1 x ν ˆx x, ˆx +1 x (2µ + ν) ˆx x, for inexact Newton and modified inexact Newton methods, respectively. Proof. First, we point out that (11) or (12) implies ν<1, (µ + ν) < 1, and (12) yields (2µ + ν) < 1. Assume that ˆx and ŷ are in S, and determine an upper bound for ˆx +1 x. From (1) and Lemma 3.2 with ξ = x + t(ˆx x )weobtain ˆx +1 x =ˆx x F (ŷ ) 1 (F(ˆx ) F(x )) + F (ŷ ) 1ˆr = E(ξ,ŷ )dt (ˆx x )+F (ŷ ) 1 P 1 P ˆr.

CONVERGENCE BEHAVIOUR OF INEXACT NEWTON METHODS 169 Further, ˆx +1 x E(ξ,ŷ ) dt ˆx x + θ (P F (ŷ )) 1 P F (ˆx ) E(ξ,ŷ ) dt ˆx x + θ (P F (ŷ )) 1 P F (ŷ ) E(ξ,ŷ ) dt ˆx x + θ cond(p F (ŷ )) [(1 + ν ) and from Lemma 3.3 we derive (15) F (ŷ ) 1 F (ξ )dt(ˆx x ) (I + E(ξ, ŷ ))dt(ˆx x ) E(ξ, ŷ ) dt + ν ] ˆx x, ˆx +1 x [Λ(1 + ν )( 1 2 ˆx x + m )+ν ] ˆx x, where m =min{ ˆx ŷ, x ŷ }. Concerning inexact Newton methods, we observe that m =foreach,andif ˆx S(x,δ) from (15) we have ˆx +1 x [ Λ 2 (1 + ν )δ + ν ] ˆx x < (µ + ν) ˆx x. Then, from (11) it follows that ˆx +1 x < ˆx x. Moreover it can be shown that (16) ˆx +1 x <α (µ+ν)δ,. In fact, for =wehave ( ) Λ ˆx 1 x 2 (1 + ν )δ + ν ˆx x < (µ + ν)δ, and (16) holds. Suppose now ˆx x <α 1 (µ+ν)δfor a given 1. If α<1, then ˆx S(x,δ), and from (15) we obtain ˆx +1 x < [Λ(1 + ν )( 1 2 α 1 (µ + ν)δ)+ν ]α 1 (µ+ν)δ <(µ(µ+ν)+ν)α 1 (µ+ν)δ=α (µ+ν)δ, which proves (16). To prove convergence for modified inexact Newton methods we show that (16) holds with α given in (12). In particular, m =,m =min{ ˆx ˆx, ˆx x } δ for >, and (15) yields ˆx 1 x ( 1 2 Λ(1 + ν )δ + ν ) ˆx x < (µ + ν)δ. Then ˆx 1 S(x,δ), and (16) holds for =. If as inductive hypothesis we assume ˆx S(x,δ)and ˆx x <α 1 (µ+ν)δ, then from (15) we have ˆx +1 x <α ˆx x, and (16) follows. Finally, if we consider (15) and if is large enough so that Λ (17) 2 (1 + ν ) ˆx x ν ν, (13) and (14) easily follow.

161 BENEDETTA MORINI It is worth noting that, for a given ν<1, at each iteration (11) and (12) yield the upper bound ν/cond(p F (ŷ )) for θ. This theorem also gives an estimate of the radius of convergence δ σ. In particular, from α<1wehave 2(1 ν) (ν +2)+ ν2 +8 δ<, δ <, Λ(1 + ν) Λ(1 + ν) for inexact Newton and modified inexact Newton methods, respectively, and we can conclude that δ decreases while ν approaches 1. From Theorem 3.1, stated with Λ = Λ σ, we have that the sequence {ˆx } belongs to S(x,δ). Moreover, from [1] we now that, for x, y S(x,δ)andδ σ,(8) holds with Λ δ =Λ /(1 Λ δ). Then we can conclude that the thesis of Theorem 3.1 is still valid if we replace Λ with Λ δ. Further, if (11) and (12) are formulated in terms of Λ δ,thenforν= δ< 2 3Λ, δ < 2 2 2 (2 2 1)Λ, given in [12] for Newton s method and modified Newton s method, respectively. In particular, the estimate for the radius of convergence of Newton s method is nown to be sharp ([12]). Then, we can conclude that for vanishing residuals, Theorem 3.1 merges into the theory of Newton s method. A result analogous to Theorem 3.1 can be proven also for inexact Newton-lie methods where ˆB = ˆB(ˆx ) approximates F (ˆx ). Theorem 3.2. Let ˆB(x) be an approximation to the Jacobian F (x) for x D(F ), satisfying for x S the following properties: ˆB(x) is invertible; ˆB(x) 1 F (x) I τ 1 ; ˆB(x) 1 F (x) τ 2. Let ˆx S, ˆx x δ,ν =θ cond(p ˆB ) and ν ν<ν. Then the sequence {ˆx } of (1) is well-defined, and {ˆx } converges to x if (18) α = ρ(ρ + τ 1 + ντ 2 )+τ 1 +ντ 2 <1, where ρ = Λ 2 δ(1 + ν)τ 2. Moreover, for sufficiently large, (19) ˆx +1 x (τ 1 +ντ 2 ) ˆx x. Proof. To begin, note that (18) implies (τ 1 + ντ 2 )<1and(ρ+τ 1 +ντ 2 )<1. We assume that ˆx S and determine an upper bound for ˆx +1 x. From (1) and Lemma 3.2 with ξ = x + t(ˆx x )weobtain ˆx +1 x =ˆx x ˆB 1 (F(ˆx ) F(x )) + =ˆx x = 1 ˆB ˆr ˆB 1 F (ξ )dt(ˆx x )+ ˆB 1 F (ˆx )E(ξ, ˆx ))dt(ˆx x ) 1 ˆB (F (ˆx ) ˆB )(ˆx x )+ 1 ˆB ˆr 1 ˆB P 1 P ˆr.

CONVERGENCE BEHAVIOUR OF INEXACT NEWTON METHODS 1611 Therefore, applying Lemma 3.3, we derive ˆx +1 x (τ 2 E(ξ,ˆx ) dt + τ 1 ) ˆx x + θ ˆB 1 ( 1 2 Λτ 2 ˆx x + τ 1 ) ˆx x + ν τ 2 F (ˆx ) 1 F (ξ ) dt ˆx x ( 1 2 Λτ 2 ˆx x + τ 1 ) ˆx x + ν τ 2 (I + E(ξ, ˆx )) dt ˆx x, P 1 and [ ] Λ (2) ˆx +1 x 2 (1 + ν )τ 2 ˆx x + τ 1 + ν τ 2 ˆx x. To prove convergence, note that ˆx 1 x (ρ+τ 1 +ν τ 2 ) ˆx x <(ρ+τ 1 +ντ 2 )δ, and, by induction on >1, suppose that (21) ˆx x <α 1 (ρ+τ 1 +ντ 2 )δ, P F (ˆx ) with α defined in (18). We can conclude that ˆx S(x,δ) and, from (2), [ ] Λ ˆx +1 x < 2 (1 + ν )τ 2 α 1 (ρ + τ 1 + ντ 2 )δ+τ 1 +ν τ 2 ˆx x <[ ρ(ρ+τ 1 +ντ 2 )+τ 1 +ντ 2 ] ˆx x =α ˆx x, which proves the induction hypothesis. Further, for sufficiently large, (17) holds and from (2) we derive (19). The results we proved state inverse proportionality between each forcing term θ and cond(p ˆB ). Such conditions are sufficient for convergence, and may be overly restrictive for the upper bounds on {θ } if P ˆB are bad conditioned matrices. We now consider the sequence {ˆx } given by inexact methods (1) with P I, and the sequence {x } obtained for P = I,, and matrices B = ˆB(x ) in (3). From Theorem 3.2 it easily follows that if x = ˆx and the sequences {θ cond(p ˆB )} and {η cond(b )} are bounded by the same quantity ν, then {x }and {ˆx } converge to x satisfying (19). Further, consider the -th equation (3) B s = F (x )+r, and assume that both residual controls r F (x ) η, P r P F (x ) θ, are applied. If we suppose η cond(b )=θ cond(p B ), i.e. θ = cond(b ) cond(p B ) η = γη, and if P is such that 1 cond(p B ) cond(b ), we obtain γ [1, cond(b )] and θ [η,ν ]. We point out that θ does not depend on cond(p ) but only on the conditioning of P B. Further, it follows that the choice of a suitable scaling

1612 BENEDETTA MORINI matrix P such that cond(p B ) < cond(b ) leads to a relaxation on the forcing terms, and while cond(p B ) decreases, θ approaches the maximum ν. For P = B 1, called natural scaling ([13]), we have the residual control B 1 r (22) B 1 F (x ) θ, and the extremal properties cond(p B )=1andθ =ν, hold. Moreover, regarding inexact Newton and modified inexact Newton methods with control (22), from Theorem 3.1 it is easy to derive Corollary 3.1. Let the hypotheses of Theorem 3.1 hold. Then, given a sequence {θ } uniformly less than 1, there exists a ˆδ such that for ˆx S(x,δ), δ<ˆδ,the sequence {ˆx } converges Q-linearly. Proof. Since ν = θ < 1 and {θ } is bounded away from 1, for a suitable ν<1 we have ν ν < ν. Further, there exists ˆδ > such that (11) holds. Then, from Theorem 3.1 the thesis follows for inexact Newton methods. In the same way, if ˆδ is such that (12) is satisfied, the thesis holds for modified inexact Newton methods. We remar that θ < 1 is the only condition needed to define the process correctly ([3]), and that condition (22) is affine invariant, as it is insensitive with respect to transformations of the mapping F (x) of the form: F (x) AF (x), A an invertible matrix, as long as the same affine transformation is also valid for B(x) (see e.g. inexact Newton, modified inexact Newton and inexact finite difference methods). Since Newton s iterates are affine invariant, in [13] convergence conditions were determined in affine invariant terms. Ypma provided an affine invariant study of local convergence also for inexact Newton methods, although they are formally no longer affine invariant unless the residuals vanish. Specifically, in [9] it was noted that even if the method F (x )s = F (x )+r, =,1,..., x +1 = x + s, is affine invariant, the condition r / F (x ) is not. As a consequence the alternative form (22) with B = F (x ) 1 was proposed, and affine convergence theory was provided for the resulting process. We point out that for P = B 1 and for methods such that (22) is affine invariant, Theorems 3.1 and 3.2 represent an affine convergence analysis of inexact Newtonlie methods, and in the case of inexact Newton methods Corollary 3.1 agrees with the main theorem of [9] (see [9], Theorem 3.1). Acnowledgments The author would lie to than Prof. M.G. Gasparo, M. Macconi, and the referee for detailed and helpful suggestions. References [1] J.M. Ortega, W.C. Rheinboldt, Iterative solution of nonlinear equation in several variables, Academic Press, New Yor, 197. MR 42:8686 [2] J.E. Dennis, R.B. Schnabel, Numerical methods for unconstrained optimization and nonlinear equations, Prentice Hall, Englewood Cliff, NJ, 1983. MR 85j:651

CONVERGENCE BEHAVIOUR OF INEXACT NEWTON METHODS 1613 [3] R.S. Dembo, S.C. Eisenstat, T. Steihaug, Inexact Newton methods, SIAM J. Numer. Anal.,19, 1982, pp. 4-48. MR 83b:6556 [4] T. Steihaug, Quasi-Newton methods for large scale nonlinear problems, Ph.D.Thesis,School of Organization and Management, Yale University, 1981. [5] J.M. Martinez, L.Qi, Inexact Newton methods for solving nonsmooth equations, J.Comput. Appl. Math., 6, 1995, pp. 127-145. MR 96h:6576 [6] P. Deuflhard, Global inexact Newton methods for very large scale nonlinear problems, Impact Comput. Sci. and Engrg., 3, 1991, pp. 366-393. MR 92i:6593 [7] P.N. Brown, A.C. Hindmarsh, Reduced storage matrix methods in stiff ODE systems, Appl. Math. Comput., 31, 1989, pp. 4-91. MR 9f:6514 [8] K.R. Jacson, The numerical solution of large systems of stiff IVPs for ODEs, Appl. Numer. Math., 2, 1996, pp. 5-2. MR 97a:6561 [9] T.J.Ypma, Local convergence of inexact Newton methods, SIAM, J. Numer. Anal., 21, 1984, pp. 583-59. MR 85:6543 [1] T.J.Ypma, Local convergence of difference Newton-lie methods, Math. Comp. 41, 1983, pp. 527-536. MR 85f:6553 [11] J.L.M. van Dorsselaer, M.N. Spijer, The error committed by stopping the Newton iteration in the numerical solution of stiff initial value problems, IMA J. of Numer. Anal., 14, 1994, pp 183-29. MR 95c:6597 [12] T.J.Ypma, Affine invariant convergence for Newton s methods, BIT, 22, 1982, pp. 18-118. MR 84a:5818 [13] P.Deuflhard, G. Heindl, Affine invariant convergence theorem for Newton methods and extension to related methods, SIAM J. Numer. Anal., 16, 1979, pp. 1-1. MR 8i:6568 Dipartimento di Energetica Sergio Stecco, via C. Lombroso 6/17, 5134 Firenze, Italia E-mail address: morini@riscmat.de.unifi.it