YURI LEVIN AND ADI BEN-ISRAEL

Transcription

1 Pp in Progress in Analysis, Vol. Heinrich G W Begehr. Robert P Gilbert and Man Wah Wong, Editors, World Scientiic, Singapore, 003, ISBN AN INVERSE-FREE DIRECTIONAL NEWTON METHOD FOR SOLVING SYSTEMS OF NONLINEAR EQUATIONS YURI LEVIN AND ADI BEN-ISRAEL Abstract. A directional Newton method is proposed or solving systems o m equations in n unknowns. The method does not use the inverse, or generalized inverse, o the Jacobian, and applies to systems o arbitrary m, n. Quadratic convergence is established under typical assumptions irst derivative not too small, second derivative not too large. The method is stable under singularities in the Jacobian. 1. Introduction Consider a system o m equations in n unknowns: x = 0, or i x 1, x,..., x n = 0, i 1, m. 1 I m = n the Newton method or solving 1 uses the iterations, see e.g. [14],[10], x k+1 := x k J x k 1 x k, k = 0, 1,..., where the Jacobian matrix i J x := x j is assumed nonsingular. I J x is singular, or i m n, a suitable generalized inverse o J x can be used in e.g. [1],[4] without losing quadratic convergence, e.g. [9]. In particular, the Moore-Penrose inverse in gives the method x k+1 := x k J x k x k, k = 0, 1,..., 3 see e.g. [8] where quadratic convergence was established under typical assumptions. This method is applicable to least squares problems, because every limit point x o 3 is a stationary point o the sum o squares i x, i x = 0. Both methods and 3 require matrix inversions. We present here a Newton method or solving systems o equations that does not use any matrix inversion. This requires two steps Date: November 5, Mathematics Subject Classiication. Primary 65H05, 65H10; Secondary 49M15. Key words and phrases. Directional Newton method, Inverse-ree method, Single equations, Systems o equations. 1

2 YURI LEVIN AND ADI BEN-ISRAEL Step 1: Transorm 1 to a single equation in n unknowns F x = 0, or F x 1, x,..., x n = 0, 4 such that 1 and 4 have the same solutions. Step : Solve 4 by a directional Newton method, see e.g [7], In Step 1 a natural transormation is x k+1 := x k F xk F x k F xk, k = 0, 1,... 5 F x := i x = 0, 6 and several authors e.g. [, Vol II, p. 165], [1, p. 36] suggested solving 6 by minimizing the sum o squares i x using a suitable method, such as the steepest descent method. However, i one solves 6 using a directional Newton method, quadratic convergence is lost because the gradient o F F x = i x i x,...,m approaches the zero vector as the values i x tend to zero. To prevent this, we propose an alternative transormation F x := i x + θ i θ i = 0, where θ i 0, i 1, m, 7 with gradient existing i all i x 0 i x F x = i x, 8 i x + θi whose behavior, as i x 0, can be controlled by adjusting the parameters θ i. In [8] we established the quadratic convergence o The directional Newton method 5, and the more general directional method x k+1 := x k F x k F x k d k dk, k = 0, 1,..., 9 under typical assumptions on the unction F around the initial point x 0, and the successive directions {d k }. However, the convergence proos in [8] are not applicable to the special case o F given by 7. The main results o this paper are Theorems 1 and. Theorem 1 gives conditions or the quadratic convergence o the directional Newton method 5, conditions that are natural in the special case o F x given by 7. Theorem then establishes the quadratic convergence o 5 when applied to the equivalent equation F x = 0.

3 DIRECTIONAL NEWTON METHOD FOR SYSTEMS OF EQUATIONS 3 We call the method {7, 5} an inverse ree Newton method. This method is adapted to least squares solutions and optimization problems in 4 5. The method is illustrated by numerical examples in 6. Since the inverse ree Newton method does not require inversion, it is well suited or dealing with singularities in the Jacobian, along the iterations {x k } or in their limits x.. Convergence o the Directional Newton Method 5 In this section we give a new proo o the convergence o the directional Newton method 5 that apply naturally to F o 7. The main tool is the majorizing sequence, due to Kantorovich and Akilov [6], see also [10, Chapter 1.4]. Deinition 1. A sequence { y k}, y k 0, y k R or which x k+1 x k y k+1 y k, k = 0, 1,... is called a majorizing sequence or { x k}. Note that any majorizing sequence is necessarily monotonically increasing. The ollowing two lemmas are used below. Lemma 1. I F : R n R is twice dierentiable in an open set S then or any x, y S, F y F x y x sup F x + t y x. 0 t 1 Lemma. I { y k}, y k 0, y k R is a majorizing sequence or { x k}, x k R and lim k yk = y <, then there exists x = lim x k and x x k y y k, k = 0, 1,... k Lemma 1 is consequence o the Mean Value Theorem, and Lemma is proved in [10, Chapter 1.4, Lemma 1.4.1]. To prove the convergence o 5, we write it as x k+1 := x k F x k v k, 10a where v k := F x k F x k. 10b Theorem 1. Let F : R n R be a twice dierentiable unction, x 0 R n, and assume that where X 0 is deined as sup x = M, 11 x X 0 X 0 := { x : x x 0 R }, 1

4 4 YURI LEVIN AND ADI BEN-ISRAEL or R given in terms o constants M, B, C that are assumed to satisy F x 0 C, 13a F x 0 1 B, 13b MB C < 1, 13c R := 1 1 MB C. 13d MB Then: a All the points x k+1 := x k F x k v k, k = 0, 1,... lie in X 0. b x = lim x k exists, x X 0, and x = 0. k c The order o covergence o the directional Newton method 5 is quadratic. Proo. We construct a majorizing sequence or {x k } in terms o the auxiliary unction ϕ y = M y 1 B y + C. 14 By 13c, the quadratic equation ϕy = 0 has two roots r 1 = 1 1 MB C = R and MB r = 1+ 1 MB C. Also, ϕ y = My 1 and MB B ϕ y = M. Starting rom y 0 = 0, apply the scalar Newton iteration to the unction ϕ y to get y k+1 = y k ϕ y k, k = 0, 1,, ϕ y k M = y k y k 1 B yk + C M y k 1, k = 0, 1,,... B M y k C = My k 1, k = 0, 1,,... B Next we prove that the sequences { x k} and { y k}, generated by 5 and 15 respectively, satisy or k = 0, 1,... F x k ϕ y k, 16a v k 1 ϕ y k, 16b x k+1 x k y k+1 y k. 16c Statement 16c says that { y k} is a majorizing sequence or { x k}. The proo is by induction. Veriication or k = 0 : F x 0 C = ϕ y 0 = ϕ 0,

5 DIRECTIONAL NEWTON METHOD FOR SYSTEMS OF EQUATIONS 5 v 0 1 B =, ϕ y 0 x 1 x 0 = F x 0 v 0 F x 0 v 0 ϕ y0 ϕ y 0 = y1 y 0, showing that equations 16a-16c hold or k = 0. Suppose 16a-16c hold or k n. Proo o 16a or n + 1 : x n+1 x 0 n = x k+1 x k k=0 x n+1 X 0. = y n+1 y 0 = y n+1 R. n y k+1 y k k=0 Let a n x := x x n F x n + F x n F x n. a n x = x x n F x n. Let p n x := F x n F x n a nx. Note that p n x n+1 = 0. On the other hand, p n x n+1 can be represented as p n x n+1 = F x n + F x n x n+1 x n. 0 = F x n + F x n x n+1 x n. F x n+1 = F x n+1 F x n F x n x n+1 x n. So, by induction F x n+1 M x n+1 x n M y n+1 y n = ϕ y n+1, since ϕ y n+1 = M y n+1 y n. Proo o 16b or n + 1 : v n+1 1 = F x n+1 = 1 F x 0 F x 0 F x n+1 1 F x 0 F x 0 F x n+1 1 = F x 0 1 F x0 F x n+1, by Lemma 1, F x 0 B 1 MBy n+1 = 1 1 B Myn+1 = 1 ϕ y n+1.

6 6 YURI LEVIN AND ADI BEN-ISRAEL Proo o 16c or n + 1 : x n+ x n+1 = F x n+1 v n+1 F x n+1 v n+1 ϕ yn+1 ϕ y n+1 = y n+ y n+1., by 16a and 16b, Consequently, 16a-16c hold or all n 0. Since the sequence { x k} is majorized by the sequence { y k} it ollows rom Lemma that lim k x k = x and x X 0. The scalar Newton method has a quadratic rate o convergence, and y k+1 y k β θk, 1 θ k where θ < 1, see [13]. Thereore x k+1 x k β θ k, and the sequence { x k} has at least 1 θ k quadratic rate o convergence. 3. Application to the solution o a system o equations Recall the system 1 and the equivalent equation 7. The ollowing theorem gives conditions or the the convergence o the directional Newton method 5 when applied to 7. Theorem. Let i : R n R be a twice dierentiable unctions, i 1, m, let x 0 R n, and assume that i x M m, x X 0, i 1, m, 17 i x M m θ i, x X 0, i 1, m, 18 where X 0 is deined by X 0 := { x : x x 0 R }, 19 or R given in terms o constants M, B, C that are assumed to satisy i x0 + θi θ i C, 0a i x 0 i x 0 1 i x 0 + θi B, 0b MB C < 1, 0c R := 1 1 MB C. 0d MB Then the unction F x := m i x + θ i θ i, where θ i 0, i 1, m, satisies all conditions o Theorem 1 with corresponding M, B, C, R, X 0.

7 DIRECTIONAL NEWTON METHOD FOR SYSTEMS OF EQUATIONS 7 Proo. We show a unction F x := m i x + θ i θ i, where θ i 0, i 1, m, satisies all conditions o the Theorem 1 with corresponding M, B, C, R, X 0. F x 0 m = i x0 + θi θ i C, by 0a. So, 13a holds. So, 13b holds. F x = F x 0 m = F x = i x i x, i x + θi F x = i x i x i x + θi, and i x 0 i x 0 1, by 0b. i x 0 + θi B i x i x + θi i x + θ i i x i x T i x + θ i, 3 F x = i x + θ i i x i x + θ i 3 M m + M, by 17 and 18. m i x i x + θi i x + θ i i x i x T i x + θ i 3 i x + i x θ i So, 11 holds, where X 0 := {x : x x 0 R}, R := 1 1 MB C and MB C < 1. MB The unction F x := m i x + θ i θ i, where θ i 0, i 1, m, thus satisies all conditions o the Theorem 1 with corresponding M, B, C, R, X Application to least squares problems The system 1 was replaced above by an equivalent single equation F x = 0, with 7 F x := m i x + θ i θ i = 0, where θ i 0, i = 1,..., m,

8 8 YURI LEVIN AND ADI BEN-ISRAEL and then solved by the directional Newton method 5 x k+1 := x k F xk F x k F xk, k = 0, 1,... where 8 F x = m i x i x. i x + θi In practice it is oten advantageous to use the ollowing modiication o 5 x k+1 := x k F xk F x k i x k i x k, k = 0, 1,... 1 where the last occurrence o F x in 5 is replaced by i x i x, that is the gradient o i x. Experience shows similar iteration counts or 1 and 5, giving an advantage to 1 because o less work per iteration. Another advantage o 1 stems rom : all limit points o 1 are stationary points o the sum o squares i x. The modiied method 1 can thereore be used to ind least squares solutions, like the method 3 but without matrix inversion. In contrast, the method 5 converges only i the system 1 has a solution. Optimization problems oten call or solving 5. Application to optimization x = 0, 3 where is the objective unction. This is a system o nonlinear equations x x i = 0, i 1, n. that can be solved by 5 with F and F replaced by n x F x := + θi θ i x i x n F x = x i x x x + θ i. i x i

9 DIRECTIONAL NEWTON METHOD FOR SYSTEMS OF EQUATIONS 9 6. Numerical Examples In this section we illustrate the inverse-ree Newton method or our numerical examples. In Example 4 we also compare this method with method 3. Example 5 compares the modiied inverse-ree method 1 with 3 or least-squares problems. Example 1. Consider the system [5, p. 186] x = x + y 3 + z 5 y = x 3 + y 5 + z 7 z = x 5 + y 7 + z 11. The Newton method converges or x 0 =.8,.5,.3 to the root z 1 = , , , but diverges or other initial points between the roots z 1 and z = 0, 0, 0, or example, x 0 =.4,.3,.. The inverse-ree method 1 converges or x 0, giving ater 7 iterations suiciently close to the root z. Example. Consider the system , , x + y = 0, x + y = 0. The Newton method cannot be used with initial points on the y axis, where the Jacobian is singular. Both the inverse ree method 5 and the generalized inverse Newton method 3 converge to the solution 0, 0 in one iteration rom any point on the y axis. Example 3. Consider the system see [3, p. 470] x 3 + xy = 0 y + y = 0. The Jacobian o this system is singular at any point x, 0.5, and the Newton method is inapplicable there. The inverse-ree methods 5 and 1 are not aected by the singularity o the Jacobian, and converge also or initial points on the line {x, 0.5 : x R}. Example 4. Consider a system o 10 equations in 10 unknowns [8, Example ], k x := 10 x k i 10 = 0, k 1, 10, 4 with solution x = 1, 1,, 1. Problem 4 was solved using method 3 and the inverse-ree method 5, with x 0 =,,,. The Newton method cannot be used with x 0 because the Jacobians J x are singular have rank 1 at all successive iterations.

10 10 YURI LEVIN AND ADI BEN-ISRAEL Ater 10 iterations, both methods gave the solution 1, 1,, 1 o system 4. Table 1 shows ast decreases in the sum o squares errors SSE, k kx or both methods. Note, that 5 does not compute the inverse o the Jacobian, whereas 3 does. Table 1. Comparison o sum o squares errors in Example 4 or Methods 3 and 5, with initial solution x 0 =,,, Method 3 Method 5 Iteration SSE SSE Example 5. Consider a system o 10 equations in 10 unknowns [8, Example ], k x := 10 x k i 5 = 0, k 1, 10, 5 which has no solution, and a least squares solution is sought. Problem 5 was solved using method 3 and the modiied inverse-ree method 1, with x 0 =,,,. Both methods 1 ater 7 iterations and 3 ater 10 gave the least squares solution.888,.888,,.888 o system 5. Table shows ast decreases in the sum o squares errors SSE, k kx or both methods. Acknowledgement We thank Mikhail Nediak or his valuable comments and suggestions. Reerences [1] A. Ben-Israel, A Newton-Rapshon method or the solution o sytems o equations, J. Math. Anal. Appl , 43 5 [] I.S. Berezin and N.P. Zhidkov, Computing Methods, Pergamon Press, 1965 [3] D. Decker and C. Kelley, Newton s method at singular points, SIAM J. Numer. Anal , [4] P. Deulhard and G. Heindl, Aine invariant convergence theorems or Newton s method and extensions to related methods, SIAM J. Numer. Anal , 1 10

11 DIRECTIONAL NEWTON METHOD FOR SYSTEMS OF EQUATIONS 11 Table. Comparison o sum o squares errors in Example 5 or Methods 3 and 1, with initial solution x 0 =,,, Method 3 Method 1 Iteration SSE SSE [5] C.-E. Fröberg, Mumerical Mathematics: Theory and Computer Applications, Benjamin, 1985 [6] L.V. Kantorovich and G.P. Akilov, Functional Analysis in Normed Spaces, Macmillan, New York, 1964 [7] Y. Levin and A. Ben-Israel, Directional Newton methods in n variables, Math. o Computation to appear [8] Y. Levin and A. Ben-Israel, A Newton method or systems o m equations in n unknowns, Nonlinear Analyses to appear [9] M.Z. Nashed and X. Chen, Convergence o Newton-like methods or singular operator equations using outer inverses, Numer. Math , [10] J.M. Ortega and W.C. Rheinboldt, Iterative Solution o Nonlinear Equations in Several Variables, Academic Press, 1970 [11] A.M. Ostrowski, Solution o Equations in Euclidean and Banach Spaces, 3rd Edition, Academic Press, 1973 [1] A. Ralston and P. Rabinowitz, A First Course in Numerical Analysis, nd edition, McGraw-Hill, 1978 [13] J. Stoer and K. Bulirsch, Introduction to Numerical Analysis, Springer-Verlag, 1976 [14] J.F. Traub, Iterative Methods or the Solution o Equations, Prentice Hall, 1964 Yuri Levin, RUTCOR Rutgers Center or Operations Research, Rutgers University, 640 Bartholomew Rd, Piscataway, NJ , USA address: ylevin@rutcor.rutgers.edu Adi Ben-Israel, RUTCOR Rutgers Center or Operations Research, Rutgers University, 640 Bartholomew Rd, Piscataway, NJ , USA address: bisrael@rutcor.rutgers.edu