Two new spectral conjugate gradient algorithms based on Hestenes Stiefel

Transcription

1 Research Article Two new spectral conjuate radient alorithms based on Hestenes Stiefel Journal of Alorithms & Computational Technoloy 207, Vol. (4) ! The Author(s) 207 Reprints and permissions: saepub.co.u/journalspermissions.nav DOI: 0.77/ journals.saepub.com/home/act Guofan Wan, Rui Shan, Wei Huan, Wen Liu and Jinyi Zhao Abstract The spectral conjuate radient alorithm, which is a variant of conjuate radient method, is one of the effective methods for solvin unconstrained optimization problems. In this paper, based on Hestenes Stiefel method, two new spectral conjuate radient alorithms (Descend Hestenes-Stiefel (DHS) and Wan-Hestenes-Stiefel (WHS)) are proposed. Under Wolfe line search and mild assumptions on objective function, the two alorithms possess sufficient descent property without any other conditions and are always lobally converent. Numerical results turn out the new alorithms outperform Hestenes Stiefel conjuate radient method. Keywords Spectral conjuate radient alorithm, unconstrained optimization, Wolfe line search, sufficient descent property, lobally converent Date received: 2 May 207; revised: 23 May 207; accepted: 20 June 207 Introduction Consider the unconstrained optimization problem (UP) min f ðxþ, x 2 R n where the function f : R n! R is continuously differentiable. The most commonly used method for solvin this ind of problem is the conjuate radient (CG) method, which is especially suitable for solvin lare dimension or non-linear problems. Its converence rate is between Newton method and steepest descent method, CG method avoids the shortcomins of the Newton method to calculate the Hessen matrix, and also has a secondary termination. Its main iterative format is x þ ¼ x þ d d ¼, ¼ 0; þ d, where is the step factor, which can be determined by some methods (line search, etc.), d is the down search direction, is a scalar. Different CG methods are ðþ ð2þ ð3þ enerated accordin to the different formulae of scalar parameters, and different spectral CG methods are enerated accordin to different search directions d. The expressions used in some well now CG alorithms are list below HS ¼ T ð Þ d T ð Þ alorithm, 2 FR ¼ T T PRP DY ¼ T ð Þ T HS ðhestenes StiefelÞ FR ðfletcher ReevesÞalorithm, 3 PRP (Pola Ribie` re Polya) alorithm, 4,5 ¼ T d T ð Þ DY ðdai YuanÞ alorithm: 6 Collee of Science, Yanshan University, Qinhuandao, China Correspondin author: Rui Shan, Collee of Science, Yanshan University, Qinhuandao , China @qq.com Creative Commons CC BY-NC: This article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 License ( creativecommons.or/licenses/by-nc/4.0/) which permits non-commercial use, reproduction and distribution of the wor without further permission provided the oriinal wor is attributed as specified on the SAGE and Open Access paes (

2 346 Journal of Alorithms & Computational Technoloy (4) Amon these four alorithms, FR and DY alorithm have ood lobal converence, while Hestenes Stiefel (HS) and PRP alorithm have fantastic numerical performance. HS alorithm for strict convex quadratic function has finite step converence under exact line search, but for eneral non strict convex quadratic objective function, even under exact line search can t uarantee converent in finite steps, and lobal converence cannot be uaranteed. 7 Combinin with the advantae of HS and DY alorithm, references 8 proposed a new conjuate method 8 T ð Þ, ð cos Þjj ð ¼ T d Þ 2 d T jj 2 4 T >< ; >: maxf jj jj 2 d T y,0, otherwise, (, ¼ ; d ¼ ð þ T d Þ jj jj 2 þ d, 4 NLS NLS DY alorithm: 8 The motivation of this paper is to combine the advantaes of HS 3 and NLS-DY 8 in order to provide novel alorithms with better converence. The new alorithms Consider the unconstrained optimization problem (), combinin with the literature. 3,8 The formulae of DHS and WHS are constructed as follows WHS T ¼ ð Þ ð T d Þ 2 d T 8 <, ¼ ; d ¼ þ T d : jj jj 2 þ d, 4 Compared with HS alorithm, DHS alorithm s innovation lies in d. In the HS alorithm, iteration format is (2), search direction is (3), and search method is line search. However, in the DHS alorithm, iteration format is (2), search direction is (5), and search method is Wolfe line search. Under the same scalar parameter HS, usin different search direction and search method, the DHS alorithm has better numerical results under the premise of converence. WHS alorithm also uses search direction (5) and Wolfe line search, the scalar parameter NLS in Shi et al. 8 is modified to et the scalar parameter WHS : The parameter in WHS is a constant and 0, different parameters have different iterative effects. ð4þ ð5þ DHS alorithm implementation process:. Given a initial value x 2 R n, " 4 0, d ¼, ¼ : 2. Perform the Wolfe line search f ðx þ d Þfðx Þþ T d ; d T ðx þ d Þd T ð6þ where , and are real numbers. From (6), we et, and accordin to (2), we obtain x þ : Then calculate f þ ¼ f ðx þ Þ, þ ¼ ðx þ Þ: 3. If jj þ jj 5 ", the minimum value is x þ ; but if jj þ jj ", o to the next step. 4. Calculate formula HS and formula d from (5). 5. Put ¼ þ, and turn to 2. WHS alorithm implementation process:. Given a initial value x 2 R n, " 4 0, d ¼, ¼ : 2. Perform the Wolfe line search (6) ,we et, and accordin to (2), we obtain x þ : Then calculate f þ ¼ f ðx þ Þ, þ ¼ ðx þ Þ: 3. If jj þ jj 5 ", the minimum value is x þ ; but if jj þ jj ", o to the next step. 4. Calculate formula (4) and (5). 5. Put ¼ þ, and turn to 2. Global converence Assumptions: 9. The level set ¼fx 2 R n f ðxþ f ðx 0 Þ is bounded, where x 0 is the initial point. 2. The function f is continuously differentiable in a neihborhood of, and the function radient satisfies the Lipschitz continuity condition, that is, there exists a positive constant L such that the followin holds jjðx Þ ðx 2 Þjj 5 Ljjx x 2 jj, 8x, x 2 2 Theorem 2.. If 6¼ 0, the directions enerated by DHS and WHS alorithm are descendant, that is, 8, T d 5 0: Proof. While ¼, d ¼, T d ¼ jj jj 2 ; while 4, d ¼ þ T d jj jj 2 þ d, T d ¼ jj jj 2 þ T d ¼ jj jj 2 þ T d jj jj 2

3 Wan et al. 347 So it is easy to verify HS and WHS descent property. satisfy sufficient Lemma 2.. Suppose that f satisfies the above premises, x from (2), d from (5), satisfies the Wolfe line search (6). 0 Then the Zoutendij holds X ¼ ð T d Þ 2 5 þ 2 ð7þ jjd jj Theorem 2.2. Assumin that assumptions () and (2) is established, consider the CG method with the form of (2) and (5), and ¼ HS, the followin holds lim inf jj jj ¼ 0 ð8þ! Proof. (Reduction to absurdity) first of all, assume that the conclusion is not established, then 8 4 0, 9" 4 0is a real constant, and jj jj 4 " holds. Accordin to (5), T d ¼ jj jj 2 and let l ¼ þ T d, while 4, (5) is squared, taen jj jj 2 norm, and simplified as follow jjd jj 2 ¼ð Þ 2 jjd jj 2 2l d T l 2 jj jj 2, and ¼ HS, then we have jjd jj 2 jjd jj 2 ð T d Þ 2 ¼ðHS Þ2 ð T d Þ 2 2l d T ð T d Þ 2 l2 jj jj 2 ð T d Þ 2 ð HS Þ2 jjd jj 2 jj jj 4 ðl Þ 2 jj jj 2 þ jj jj 2 ¼ ½ T ð ÞŠ 2 ½d T ð ÞŠ 2 jjd jj 2 jj jj 4 ðl Þ 2 jj jj 2 þ jj jj 2 jj jj 4 ð T d Þ 2 jjd jj 2 jj jj 4 þ jj jj 2 ¼ jjd jj 2 ð T d Þ 2 þ jj jj 2 we can delivery as follows jjd jj 2 ð T d Þ 2 X X ¼ ð T d Þ 2 jjd jj 2 i¼ jj i jj 2 X X ¼ i¼ " 2 X ¼ "2 " 2 ¼ " 2, ðt d 2 Þ jjd jj 2 "2, ¼ ¼þ it is contrary to condition (7) of Lemma 2.. ((7) holds), then Theorem 2.2. holds, DHS alorithm has lobal converence. Theorem 2.3. Assumin that assumptions () and (2) is established, consider the CG method with the form of (2) and (5), and ¼ WHS, the followin holds lim! inf jj jj ¼ 0 ð9þ Proof. (Reduction to absurdity) first of all, assume that the conclusion is not established, then 8 4 0, 9" 4 0is a real constant, and jj jj 4 " holds. Accordin to (5), T d ¼ jj jj 2 and let l ¼ þ T d, while 4, (4) is squared, taen jj jj 2 norm, and simplified as follow jjd jj 2 ¼ð Þ 2 jjd jj 2 2l d T l 2 jj jj 2, and ¼ WHS, then we have jjd jj 2 ð T d Þ 2 ¼ðWHS Þ 2 jjd jj 2 ð T d Þ 2 2l d T ð T d Þ 2 l2 jj jj 2 ð T d Þ 2 ð WHS Þ 2 jjd jj 2 jj jj 4 ðl Þ 2 jj jj 2 þ jj jj 2 ¼ ½ T ð ÞŠ 2 jjd jj 2 ½ð T d Þ 2 d T Š 2 jj jj 4 ðl Þ 2 jj jj 2 þ jj jj 2 jj jj 4 ð T d Þ 2 jjd jj 2 jj jj 4 þ jj jj 2 ¼ jjd jj 2 ð T d Þ 2 þ jj jj 2 we can delivery as follows jjd jj 2 ð T d Þ 2 X X ¼ ð T d Þ 2 jjd jj 2 i¼ jj i jj 2 X X ¼ i¼ " 2 X ¼ "2 " 2 ¼ " 2, ðt d 2 Þ jjd jj 2 "2, ¼ ¼þ it is contrary to condition (7) of Lemma 2.. ((7) holds), then Theorem 2.3. holds, WHS alorithm has lobal converence. Numerical experiments In this section, we use some test functions of More et al., under the Wolfe line search, to balance the numerical performance of the two new spectral CG alorithms (DHS and WHS) and traditional HS alorithm. The proram 2 is written in the MATLAB 200b, and run on the computer with Intel(R)

4 348 Journal of Alorithms & Computational Technoloy (4) Table. Iterative comparison of two alorithms DHS and HS. DHS HS Function Dim NI NF NG t f* NI NF NG t f* Brown e e-03 Rosenbroc e e-02 Beale e e-02 Jennrich ** ** ** ** ** Jennrich Jennrich e e-0 Helical e e-02 Box e-06 Powell e e-07 Wood e e-04 Dennis e e-005 Osborne e-006 ** ** ** ** ** Bis e e-03 Osborne ** ** ** ** ** HS: Hestenes Stiefel; NI: number of iterations; NF: number of times that the function is evaluated; NG: number of radient function calculations. Table 2. Iterative comparison of two alorithms WHS and HS. WHS HS Function Dim NI NF NG t f* NI NF NG t f* Brown e e-03 Rosenbroc e e-02 Beal e e-02 Helical e e-02 Jennrich ** ** ** ** ** Jennrich Box e e-06 Box e e-06 Powell e e-07 Wood e e-04 Kowali e e-02 Dennis e e-08 Osbornel e-006 ** ** ** ** ** HS: Hestenes Stiefel; NI: number of iterations; NF: number of times that the function is evaluated; NG: number of radient function calculations. Core(TM) i5-5200u and 4.00 GB SDRAM. Durin the test, the parameters are set as follows ¼ 0:0, ¼ 0:05, " ¼ 0 6,NI 0000 The test results are shown in Tables to 3, where Dim is the dimension of the function, NI is the number of iterations, NF is the number of times that the function is evaluated, NG is the number of radient function calculations, t is the proram run time, and f* is optimal function value. The sin ** means that run stopped because the line search procedure failed to find a step lenth, this means that the alorithm has poor converence. The data in Table show that most of the test functions NI, NF, and NG which are calculated by DHS alorithm are less than HS, so the new iterative method is effective. And t of DHS obviously lower than HS. The reduction of the number of iterations and the

5 Wan et al. 349 Table 3. Iterative comparison of two alorithms DHS and WHS. WHS DHS Function Dim NI NF NG t f* NI NF NG t f* Rosenbroc e e-0 Freudenstein e þ 0 Jennrich e-0 Beal e e-02 Helical e e-07 Box e Powell e e-07 Wood e e-09 Kowali e e-0 Osbornel e e-006 HS: Hestenes Stiefel; NI: number of iterations; NF: number of times that the function is evaluated; NG: number of radient function calculations. runnin time reflects the stron converence of the alorithm, the decrease of the error indicates that the alorithm has better numerical results. DHS is more useful for solvin unconstrained problems. In Table 2, different function select different value of (at present, the choice of is uniform discrete, and 0 ). The data in Table 2 shows that most of the test functions NI, NF and NG which are calculated by WHS alorithm are less than by HS, and t of WHS obviously lower than HS. WHS is better than HS. In Table 3, after selectin the appropriate. Calculatin some of the functions, for example function Rosenbroc, Jennrich3, Helical and Box3, WHS alorithm perform slihtly better than DHS. Looin the others, for example function Freudenstein, Beal, Powell, Wood, Kowali and Osbornel, DHS alorithm perform slihtly better than WHS. DHS and WHS methods are approximately equal. To sum up, DHS and WHS both perform better than HS. The comparison between DHS and WHS needs a concrete analysis, but they are approximately equal. In addition, we also put the performance profiles of WHS with uniform discrete in Table 4 of Appendix. Note: In each function that in Tables 2 and 3, the value of selects the best one of the iterations. Conclusion In this paper, based on the classical HS method, we present two improved CG methods, that is, DHS and WHS methods. In Section 3, we obtain the followin theoretical results: The DHS has sufficient descent property, and is lobally converent if the Wolfe line search (6) is used, and the parameter The WHS has sufficient descent property, and is lobally converent if the Wolfe line search (6) is used, and the parameter On the other hand, numerical results reported in Section 4 show that: The averae performance of the DHS and WHS methods proposed in this paper are enerally better than that of the HS method. The averae performance of the DHS and WHS methods are approximately equal. Acnowledements The authors are very rateful to the anonymous referees for their valuable comments and useful suestions, which improved the quality of this paper. Declaration of conflictin interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Fundin The author(s) disclosed receipt of the followin financial support for the research, authorship, and/or publication of this article: National Nature Science Foundation of China (No , , ) and Basic Research Project of Yanshan University (6LGY02). References. Fu YD, Chen XY and Tan YH. Optimization theory and method (Chinese). Beijin: National Defense Industry Press, Hestenes MR and Stiefel E. Method of conjuate radient for solvin linear equations. J Res Natl Bureau Standards 952; 49: Fletcher R and Reeves C. Function minimization by conjuate radients. Comput J 964; 7:

6 350 Journal of Alorithms & Computational Technoloy (4) 4. Polya E and Ribiere G. Note sur la converence de methode de directions conjuuees. Revue Française Informatique Recherche Ope rationnelle 969; 6: Polya B. The conjuate radient method in extreme problems. USSR Comp Math Math Phys 969; 9: Dai YH and Yuan Y. A nonlinear conjuate radient with a stron lobal converence property. SIAM J Optim 999; 0: Gibert JC and Nocedal J. Global converence properties of conjuate radient methods for optimization. SIMA J Optim 992; 2: Shi ST, Shan R and Liu W. The conjuate radient alorithm with perturbation factor and its converence. J Henan Univ Sci Technol 203; 34: Wan KR and Gao PT. Two mixed conjuate radient methods based on DY. J Shandon Univ (Nat Sci) 206; 5: Zoutendij G. Nonlinear prorammin computational methods. Inteer Nonlin Proram 970; 43: More JJ, Garbow BS and Hilstrome KE. Testin unconstrained optimization software. ACM Trans Math Softw 98; 7: Liu XG and Hu YQ. Application of optimization method and MATLAB implementation (Chinese). Beijin: Science Press, 204. Appendix Table 4. Numerical results of the WHS method with. Function f* NI t NG NF Kowali e e e e e e e e e e e Freudenstein Jennrich (continued)

7 Wan et al. 35 Table 4. Continued Function f* NI t NG NF Beale e e e e e e e e e e e Helical e e e e e e e e e e e Box e e þ e e e e þ e e e þ e e Powell e e e e e e e e e e e (continued)

8 352 Journal of Alorithms & Computational Technoloy (4) Table 4. Continued Function f* NI t NG NF Wood e e e e e e e e e e e Rosenbroc e e e e e e e e e e e NI: number of iterations; NG: number of radient function calculations; NF: number of times that the function is evaluated.