An Alternative Three-Term Conjugate Gradient Algorithm for Systems of Nonlinear Equations

Transcription

1 International Journal of Mathematical Modelling & Computations Vol. 07, No. 02, Spring 2017, An Alternative Three-Term Conjugate Gradient Algorithm for Systems of Nonlinear Equations L. Muhammad 1, and M. Y. Waziri 2 1 Department of Mathematics, Faculty of Science, Northwest University, Kano, 2 Department of Mathematical Sciences Faculty of Science, Bayero University Kano, Kano, Nigeria. Abstract. This paper presents an alternative three-term conjugate gradient algorithm for solving large-scale systems of nonlinear equations. The proposed method is the modification of memoryless BFGS quasi-newton update for which the direction is descent using projection based technique together with Powel restarting criteria. Moreover, we proved the global convergence of the proposed method with a derivative free line search under suitable assumptions. The numerical results show that the proposed method is promising. Received: 27 October 2016, Revised: 14 August 2017, Accepted: 04 November Keywords: BFGS update, Systems of onlinear equations, Conjugate gradient, Derivative free line search. Index to information contained in this paper 1 Introduction 2 Algorithm 3 Convergence Result 4 Numerical Results 5 Conclusion 1. Introduction Consider the system of nonlinear equations F (x) = 0, (1) where F : R n R n is a nonlinear mapping. Often, the mapping, F is assumed to satisfying the following assumptions: A1. There exists an x R n s.t F (x ) = 0; Corresponding author. lawalmuhd.nuw.maths@gmail.com c 2017 IAUCTB

2 146 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) A2. F is montone and continuously differentiable mapping. The general conjugate gradient conjugate gradient method for solving (1) generate iterative points {x k } from initial given point {x 0 } as follows: x k+1 = x k + α k d k, (2) where α k > 0 is attained using line search produre, and the direction d k is defined by d k+1 = F (x k+1 ) + β k d k, k 1 d 0 = F (x 0 ), k = 0. (3) The β k is a scalar termed as conjugate gradient parameter. Several methods have been developed for solving (1), most of these methods fall within the framework of Newton and quasi-newton strategy [5, 13, 17], and are particularly accepted because of their rapid convergence property from a sufficiently good initial guess. Notwithstanding, they are usually not powerful for large-scale nonlinear systems of equations because it requires compution and storage of the Jacobian matrix and its inverse as well as solving n linear system of equations in each iteration and the convergence may even be lost when the Jacobian is singular. The conjugate gradient methods are also developed in order to eliminate some of the shortcomings of Newton s and Quasi-Newton methods for unconstrained optimization. An extention of its application to handle nonlinear system of equations was considered possible by some reserchers see [6, 13, 14, 18, 21]. In this paper we are particularly interested in three term conjugate gradient algorithm for solving large scale systems of nonlinear equations which is obtained by modifying the update of the memoryless BFGS inverse approximation of the inverse Jacobian restarted as a θ multiple of an identity matrix at every iteration. The method posseed low memory requirement and, global convergence properties and simple implementation procedure. The main contribution of this paper is to construct a fast and efficient three-term conjugate gradient method for solving (1), the proposed method is based on the recent three-term conjugate gradient method [2] for unconstrained optimization. In other words our algorithm can be thought as an application to three-term conjugate gradient method to a general systems of nonlinear equations. We present extensive numerical results and performance comparism with a DF-SDCG algorithm for large-scale nonlinear equations [9] which illustrated that the proposed algorithm is efficient and promising. The rest of the paper is organized as follows: In section 2, we present the details of the proposed method. Subsequently Convergence results are presented in section 3. Some numerical results are reported in section 4. Finally, conclusions are made in section 5. Throughout this paper,. denote the euclidean norm of a vector. For simplicity, we abbreviate F (x k ) as F k in the context. 2. Algorithm This section, presents a three term conjugate gradient algorithm for solving largescale systems of nonlinear equations.

3 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) It is well known that if the Jacobian matrix J k of F is positive definite, then the most efficient search direction at x k is the Newton direction Thus, for the Newton direction d k+1 holds true that where s k = x k+1 x k, and from secant condition that, and y k = F k+1 F k. (5) can be approximated by d k+1 = (J k+1 ) 1 F k+1, (4) s T k J k+1d k+1 = s T k F k+1, (5) J k+1 s k = y k (6) y T k d k+1 = s T k F k+1. (7) Therefore any a search direction d k+1 that is required to satisfy (7), then it can be regarded as an approximate Newton direction. Recall that, the BFGS update of the Jacobian inverse approximation is represented as B k+1 = B k s ky T k B k + B k y k s T k y T k s k By setting B k = θ k I (8) can be transformed into ( yt k B ) ky k sk s T k yk T s k yk T s k (8) Q k+1 = θ k I s ky T k θ k θ k y k s T k y T k s k ( yt k θ ) ky k sk s T k yk T s k yk T s k (9) Where, θ k = st k s k s T k y k (10) For the θ k see Raydan [16] for details. It can be noted that, the matrix Q k in (9) is a modification of (8) in the sense that it is restarted with the multiple of an identity matrix at every step (B k = θ k I) and modifying the sign infront of y k s T k in the second term of (9) to get the decsent property. Multiplying both sides of (9) by F (x k+1 ) we obtained Q k+1 F k+1 = θ k F k+1 ( sk y T k θ k θ k y k s T k y T k s k Observe that the direction d k+1 from (11) can be written as ) ( F k yt k θ ) ky k sk s T k yk T s k yk T s F k+1 (11) k d k+1 = Q k F k+1 (12)

4 148 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) hence our new direction is; d k+1 = θ k F k+1 δ k s k η k y k (13) where δ k = ( 1 + yt k θ ) ky k s T k F k+1 yk T s k yk T s k yt k θ kf k+1 yk T s, (14) k η k = θ ks T k F k+1 yk T s, (15) k Solodov and Svaiter [17] introduced projection based algorithm for optimization problems which requires a hyperplane that separate the current iterate x k from zero of the system of the equation, and this can easely be done by setting x k+1 = x k F (z k) T (x k z k ) F (z k ) 2 F (z k ). (16) Therefore incoporating this projection based algorithm together with the proposed search direction and the derivative freee line search of Li and Li [14] we find α k = max{s, ρs, ρ 2 s,...} such that F (x k + α k d k ) T d k σα k F (x k + α k d k ) d k 2. (17) Where σ, s > 0 and ρ (0, 1). we describe our proposed algorithm as follows; Algorithm 2.1 (ATTCG) Step 1 : Given x 0 σ (0, 1), and compute d 0 = F 0, set k = 0. Step 2 : If F k < ϵ. then stop; otherwise continue with Step 3. Step 3 : Determine the stepsize α k by using conditions in (17), Step 4 : Let the next iterate be z = x k + α k d k. Step 5 : If F (z k ) < ϵ then stop; otherwise compute x k+1 by (16) Step 6 : Find the search direction by (13). Step 7 : Restart criterion. If F T k+1 F k 2 > 0.2 F k+1 2, then set d k+1 = F k+1. Step 8 : Consider k = k + 1 and go to step Convergence Result In this Section, we will establish the global convergence for the ATTCG algorithm. Assumption A (i) The level set S = {x R n : F (x) F (x 0 ) } is bounded, i.e there exists positive constant B > 0 such that, for all x S, x B. (ii) The function F is lipschitz continuous on R n i.e there exist a positive constant L > 0 s.t x, y R n

5 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) F (x) F (y) L x y (18) Assumption A(ii) implies that there exists a constant λ > 0 such that F k λ. (19) The following lemma is in Solodov and Svaiter [17], from Theorem 2.1 which only state the proof can be in similar way as in [17]. Lemma 1 Suppose that Assumption A holds and {x k } is generated by TTCG Algorithm let x be a solution of problem (1) with F (x ) = 0 Then x k+1 x 2 x k x 2 x k+1 x k 2 (20) holds and the sequence {x k } is bounded. Furthermore, either the sequence {x k } is finite and the last iteration is a solution or the sequence is infinite and x k+1 x k 2 < (21) K 0 Lemma 2 Suppose that the line search satifies the condition (17) and F is uniformly convex then d k+1 given by (13) and (14)-(15) is a descent direction, and F k+1 d k+1 Proof. Since F is uniformly convex it follows that yk T s k computation, we have ( Fk+1 T d k+1 = F k y k 2 ) θ k (s T k F k+1 ) 2 yk T s k yk T s k > 0. Now, by direct 0. (22) F k+1 F T k+1 d k+1 F k+1 T d k+1 F k+1 d k+1 (23) Lemma 3 Suppose that assumptions A hold, and consider the ATTCG algorithm and (13) where d k is a descent direction and α k is computed by the condition (17). Suppose that F is uniformly convex function on S, i.e there exists a constant µ > 0 such that (F (x) F (y)) T (x y) µ x y 2

6 150 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) for all x, y S; then Proof. d k+1 κ + κ µ L2 (2 + L + ). (24) µ From Lipchitz continuity, we know that y k L s k. On the other hand by uniform convexity, it yields y T k s k µ s k 2. (25) Thus, using the Cauchy Shwartz inequality, assumptions A and the above inequalities, we have δ k st k F k+1 y T k s k + y k 2 θ k s T k F k+1 yk T s k 2 + yt k θ kf k+1 yk T s k (26) κ µ s k + L2 κ µ 2 s k + Lκ µ s k (27) = κ L2 (1 + L + µ µ ) 1 s k. (28) Since η k = θ ks T k F k+1 y T k s k s k F k+1 µ s k 2 κ µ s k, (29) Finally d k+1 θ k F k+1 + δ k s k + η k y k κ + κ µ Hence, d k+1 is bounded. L2 (2 + L + ). (30) µ The next lemma shows that the line search in step 3 of ATTCG algorithm is reasonable, then the presented algorithm is well defined. Lemma 4 Let the Assumption A hold. Then ATTCG algorithm produces an iterate of x k+1 = x k + α k d k, in a finite number of backtraking steps. Proof: We suppose that F k 0 does not hold, or the algorithm is stoped. Then there exists a constant ϵ 0 > 0 such that F k ϵ 0, k 0 (31)

7 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) up to subsequnce The aim is to guarantee that the linesearch strategy (17) is always terminated in a finite number of steps with a positive steplength α k. we will get this by contradiction. Suppose that for some iterate indexes such as k the condition (17) is not true. Then by letting αk m = ρ m s, it can be concluded that F (x k + α m k d k ) T d k < σα m k F (x k + α m k d k ) d k 2, m 0. combining the above inequality with (22), we have F (x k ) 2 = d T k F k ( 1 + y k 2 θ k y T k s k ) (s T k F k ) 2 y T k s k = [F (x k + α m k d k ) F (x K )] T d k F (x k + α m k d k ) T d k By (23) and (31) < [L + σ F (x k + α m k d k )]α m k d k 2, m 0. F (x k + α m k d k ) F (x k + α m k d k ) F k + F k Lα m k d k Thus, we obtain Ls(κ + κ L2 (2 + L + ν µ )). α m k > F k 2 [L + σ F (x k + α k d k ) ] d k 2 > L + Ls(κ + κ µ ϵ 2 0 L2 (2 + L + µ )) > 0 m 0. Thus, it contradicts with the defination of α m k. consequently, the line search procedure (17) can attain a positive steplength α k in a finite number of backtracking steps. Hence the proof is complete. Theorem 3.1 Let assumption A hold and {α k, d k, x k+1, F k+1 } be generated by ATTCG algorithm. Then

8 152 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) lim inf k F k = 0 (32) Proof: We prove this theorem by contradiction, namely the relation (31) holds. By (23) we have By lemma 2 α k d k 0, k It follows that from (33) By linesearch technique (17) we have d k F k ϵ 0 k 0 (33) lim α k = 0 (34) k F (x k + α k d k ) T d k < σα k F (x k + α k d k ) d k 2 (35) where α k = α k ρ.. By the boundedness of {x k} in lemma 3 we can deduce that there exist an accumution point x and a subsequence {x kj } of {x k } such that lim x k j = x. j The sequence {d k } is bounded. So d R n and a subsequence {d kj } (we assume that without loss of generality the subsequences {x kj }and {d kj } have the same indices) such that lim d k j = d, k thus, taking the limit as k up to subsequence in both sides of (35) generates F ( x) T d > 0. According to the other hand, by taking the limit as k up to subsequence in both sides of (17) we get F ( x) T d 0. Then a contradiction is generated. So the proof is complete.

9 4. Numerical Results L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) In this section, we tested ATTCG algorithm and compare it s performance with a DF-SDCG Algorithm [9] : The test functions are listed as follows. Examples 1 and 5 are from [19], problem 6 and 8 are from [22] and problem 9 and 10 are from [21] where as the rest are arbitrarily constructed by us. Example 1: The strictly convex function: F i (x) = e x i 1 i = 1, 2,..., n. Example 2: System of n nonlinear equations F i (x) = x i 3x i ( sinx i ) + 2 i = 2, 3,..., n. Example 3:System of n nonlinear equations F i (x) = cos x x 1 + 8e x2, F i (x) = cos x i 9 + 3x i + 8e xi 1, i = 1, 2,..., n Example 4:System of n nonlinear equations F i (x) = (0.5 x i ) 2 + (n + 1 i) x i 1, F n (x) = n 10 1 e x2 n, i = 1, 2,..., n 1. Example 5: System of n nonlinear equations F i (x) = 4x i + x i+1 2x i x i+1, 3 F n (x) = 4x n + x n 1 2x n x n+1, 3 i = 1, 2,..., n 1. Example 6: System of n nonlinear equations F i (x) = x 2 i 4, Example 7:System of n nonlinear equations F 1 (x) = sin(x 1 x 2 ) 4e 2 x2 + 2x 1, F i (x) = sin(2 x i ) 4e x i 2 + 2x i + cos(2 x i ) e 2 x i, i = 2, 3,..., n. Example 8:System of n nonlinear equations F i (x) = n i=1 x ix i 1 + e xi 1 1, i = 2, 3,..., n Example 9:System of n nonlinear equations F i (x) = x i n i=1 x 2 i n 2 + n i=1 x i n, i = 1, 2,..., n Example 10:System of n nonlinear equations F i (x) = 5x 2 i 2x i 3, i = 1, 2,..., n

10 154 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) Table 1. Numerical results based on dimension of problem (n), Number of iterations and CPU Time (in seconds) of problems 1-6 ATTCG Algorithm DF-SDCG Algorithm P Dim Iter F k CPU time Iter F k CPU time e e e e e e e e e e e e e e e e e e e e e e E e e e e e e e e e e e e e e e e e e Table 2. Numerical results based on dimension of problem (n), Number of iterations and CPU Time (in seconds) of problems 7-10 ATTCG Algorithm TPRP Algorithm P Dim Iter F k CPU time Iter F k CPU time e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e

11 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) y ATTCG DF SDCG x Figure 1. Performance profile of ATTCG and DF-SDCG methods with respect to number of iterations for Examples 1-10 y ATTCG DF SDCG x Figure 2. Performance profile of ATTCG and DF-SDCG methods with respect to CPU time in seconds for Examples 1-10 In the experiments, we compare the performance of the method introduced in this work with that of DF-SDCG conjugate gradient methods for large scale nonlinear systems of equations in order to check it s effectiveness. Numerical computations have been performed in MATLAB R2013a on a PC with Intel CELERON(R) processor with 4.00GB of RAM and CPU 1.80GHz. We used 10 test problems with dimensions 100, 1000, 5000 and to test the performance of the proposed method in terms of the number of iterations (NI) and the CPU time (in seconds). We defined a termination of the method whenever F (x k ) < (36) The table list the numerical results, where Iter and Time stand for the total number of all iterations and the CPU time in seconds, respectively; F k is the norm of the residual at the stopping point. We also used - to represent failure during iteration process due one of the following: 1. The number of iteration and/or the CPU time in second reaches 1000; 2. Failure on code execution due to insufficient memory; 3. If F k is not a number (NaN).

12 156 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) In Table 1 and 2 we listed numerical results.the numerical results indicated that the proposed method, ATTCG algorithm compared to a DF-SDCG Algorithm [9] for large-scale nonlinear equations has minimum number of iterations and CPU time. Figures 1 and 2 are performance profiles derived by Dolan and More [11] which show that our claim is justified, that is, less CPU time and number of iterations for each test problem. Furthermore, on average, our F k is small which signifies that the solution obtained is true approximation of the exact solution compared to the DF-SDCG Algorithm [9]. 5. Conclusion In this paper a new three-term conjugate gradient algorithm as modification of memoryless BFGS quasi-newton update with descent direction, has been presented. The convergence of this algorithm was proved using a derivative free linesearch. Intensive numerical experiments on some benchmark nonlinear system of equations of different characteristics proved that the suggested algorithm is faster and more efficient compared to a DF-SDCG Algorithm for large-scale nonlinear equations [9]. References [1] N. Andrei, A modified Polak-Ribi ere-polyak conjugate gradient algorithm for unconstrained optimization, Journal of Computational and Applied Mathematics, 60 (2011) [2] N. Andrei, A simple three-term conjugate gradient algorithm for unconstrained optimization, Journal of Computational and Applied Mathematics, 241 (2013) [3] N. Andrei, On three-term conjugate gradient algorithms for unconstrained optimization, Applied Mathematics and Computation, 219 (2013) [4] A. Bouaricha, R. B. Schnabel, Tensor methods for large sparse systems of nonlinear equations, Math. Program, 82 (1998) [5] S. Buhmiler, N. Kreji and Z. Luanin, Practical quasi-newton algorithms for singular nonlinear systems, Numer. Algorithms, 55 (2010) [6] C. G. Broyden, A class of methods for solving nonlinear simultaneous equations. Math. Comput, (1965) [7] C. S. Byeong, et al A comparison of the Newton-Krylov method with high order Newton-like methods to solve nonlinear systems. Appl. Math. Comput, 217 (2010) [8] W. Cheng, A PRP type method for systems of monotone equations,journal of Computational and Applied Mathematics, 50 (2009) [9] W. Cheng et.al, A family of derivative-free conjugate gradient methods for large-scale nonlinear systems of equations, Journal of Computational and Applied Mathematics, 222 (2009) [10] Y. H. Dai and Y. Yuan, A nonlinear conjugate gradient method with a strong global convergence property, SIAM J. Optim, 10 (1999) , Comput., 19 (1965) [11] E. D. Dolan and J. J. More, Benchmarking optimization software with performance profiles, Mathematical Programming, 91 (2) (2002) [12] C. T. Kelley, Iterative Methods for Linear and Nonlinear Equations, PA: SIAM, Philadelphia, (1995). [13] W. Leong, M. A. Hassan and M. Y. Waziri, A matrix-free quasi-newton method for solving nonlinear systems. Computers and Mathematics with Applications, 62 (2011) [14] Q. Li and D. Hui L, A class fo derivative free- methods for large-scale nonlinear monotone equations journal of Numerical Analysis, 31 (2011) [15] Y. Narushima and H. Yabe, Conjugate gradient methods based on secant conditions that generate descent search directions for unconstrained optimization, Journal of Computational and Applied Mathematics, 236 (2012) [16] M. Raydan, The Barzilaiai and Borwein gradient method forthe large scale unconstrained minimization problem, SIAM Journal on Optimization, 7 (1997) [17] M. V. Solodov and B. F. Svaiter, A globally convergent inexact Newton method for systems of monotone equations, in: M. Fukushima, L. Qi (Eds.) [18] G. Yuan and M. Zhang, A three-terms Polak-Ribiere-Polyak conjugate gradient algorithm for largescale nonlinear equations Journal of Computational and Applied Mathematics, 286 (2015) [19] La Cruz. W, Martinez. J. M and Raydan. M, spetral residual method without gradient information for solving large-scale nonlinear systems of equations: Theory and experiments, optimization, 76 (79)

13 L. Muhammad & M. Y. Waziri / IJM 2 C, (2017) [20] L. Zhang, W. Zhou and D. H. Li, A descent modified Polak-Ribi ere-polyak conjugate gradient method and its global convergnece, IMA J. Numer.Anal, 26 (2006) [21] M. Y. Waziri and J. Sabi u, A deivative-free conjugate gradient method and its global convergence for solving symmetric nonlinear equations, International Journal of Mathematics and mathematical Science, (2015). [22] M. Y. Waziri, H. A. Aisha and M. Mamat, A structured Broyden s-like method for solving systems of nonlinear equations, Applied mathematical Science, 8 (141) (2014)