A New Conjugate Gradient Method. with Exact Line Search

Transcription

1 Applie Mathematical Sciences, Vol. 9, 5, no. 9, HIKARI Lt, A New Conjuate Graient Metho with Exact Line Search Syazni Shoi, Moh Rivaie, Mustafa Mamat an Zabiin Salleh 3 Faculty of Informatics an Computin Universiti Sultan Zainal Abiin (Unisza), erenanu, Malaysia Department of Computer Sciences an Mathematics Universiti enoloi MARA (UiM), erenanu, Malaysia 3 School of Informatics an Applie Mathematics Universiti Malaysia erenanu (UM), erenanu, Malaysia Copyriht 5 Syazni Shoi et al. his article is istribute uner the Creative Commons Attribution License, which permits unrestricte use, istribution, an reprouction in any meium, provie the oriinal wor is properly cite. Abstract Conjuate raient (CG) methos have been practically use to solve lare-scale unconstraine optimization problems ue to their simplicity an low memory storae. In this paper, we propose a new type of CG coefficients ( ). he is compute as an averae between two ifferent types of metho which are Pola an Ribiere (PR) an Norrlaili et al. (NRMI). Numerical comparisons are mae with the five others propose by the early researches. A set of eiht unconstraine optimization problems with several ifferent variables are use in this paper. It is shown that, the new propose with an exact line search is possesse lobal converence properties. Numerical results also show that this new outperforms some of these CG methos. Keywors: Conjuate raient metho, conjuate raient coefficient, lobal converence, exact line searches Introuction Conjuate raient (CG) metho is one of methos for solvin lare-scale problems where it oes not require matrix storae an its iteration cost is low

2 8 Syazni Shoi et al. compare with others metho. Consier the optimization problem which is to be minimize as a function of n variables as, min f ( x) subject to x X () where f (x) is a real-value function calle the objective function. he n x R is n enote as ecision variable an X R is a constraint set or feasible set. If n x R, then optimization problem () can be express as an unconstraine optimization problem, min f ( x). xr n n his problem is solve iteratively, which is x R become an initial point. By usin the recurrence formula, the CG metho has the followin form, x x =,,,... () where x is the current iteration point an the is a stepsize which is obtaine by some line search metho. Most line searches use in practice is inexact line searches an also nown as approximate line search. In this proceure, the value of is estimate that will ive sufficient ecrease of the objective function. One of the most common an popular of an inexact line search is Wolfe line search []. his line search introuce two conitions as follow, f ( x ) f ( x ) (3) () where. By replacin () with conition (5), then this is calle as stron Wolfe line search. (5) Another well-nown line searches proceure is an exact line search. Lately, an exact line search has been a sinificant increase use in the number of research ue to new eneration of computer processors. In this proceure, the value of is etermine, such that the objective function with the minimize. he formula is, irection is exactly f ( x ) min f ( x ). () he avantae of usin this line search is, it will minimizes the complexity of the alorithm. An exact line search also ives an accurate escent of the objective

3 A new conjuate raient metho with exact line search 8 function. Furthermore, converence provin for exact line search became much easier as compare to inexact line search []. herefore, in this paper an exact line search is use. he is the search irection efine by if if (7) where f x ), an R is a coefficients which etermines the ifferent ( conjuate raient methos. he followin are the most common the early researches, propose by FR, (8) PR, (9) HS, () RMIL, () ( ) where f x ) an. enotes the Eucliian norm of vectors. he above ( corresponin methos are nown as Fletcher an Reeves (FR) metho [9], Pola an Ribiere (PR) metho [], Hestenes an Stiefel (HS) metho [], an Rivaie et al. (RMIL) metho [5]. Recently, [-8], an [, 3] also ave a new in orer to improve this CG metho. NRMI, () ( ) RAMI ( ), (3)

4 8 Syazni Shoi et al. ( ) NMR, () AMRI. (5) he FR has a stron converence properties but it may has averae practical performance ue to jammin. For the methos of PR, HS an RMIL they may not always be converent but often have better computational performance []. In theoretically, if f (x) is a stronly convex quaratic, all these methos are equivalent with the use of an exact line search. For non quaratic functions, ifferent choice of will leas to ifferent performance [9, ]. he lobal converences of CG methos have been stuie by many researchers such as the first lobal converence result for the FR metho was iven by Zoutenij [] in 97. He prove that, the FR metho poses lobally converence when the line search is exact. But in 977, Powell [7] has proven the poor performance of the FR metho ue to jammin phenomenon. In [7], the lobal converence of the PR metho is establishe when the functions is stronly convex uner the exact line search. However, Powell [] later showe that the PR an HS methos coul cycle infinitely without converin to minimizer when usin an exact line search. he stepsize is efine alon the search irection after the is calculate at each iteration. Proress towar minimum has been mae if f x ) f ( x ) (. he structure of this paper is oranize as follows. In the section a new type of CG coefficient is presente. In section 3, we presente the sufficient escent conition an lobal converence proof of eneral CG methos, while in Section, some numerical results corresponin to the to this new are iven. Lastly, our iscussion an conclusion are presente in Section 5 an Section respectively. New ype CG Coefficient SRMI In this paper, we evelop a new formula name as []. Where, SRMI enotes the researchers name Syazni, Rivaie, Mustafa an Ismail. his new

5 A new conjuate raient metho with exact line search 83 is resulte by averae between Pola an Ribiere (PR) metho in (3) an Norrlaili et al. (NRMI) metho in (). Hence, ( ) ( ) SRMI () PR NRMI. he followin alorithm is the eneral alorithm of CG metho use in this stuy. Alorithm. he basic of Conjuate Graient alorithm Step : Given an initial point x an set Step : Computin conjuate raient coefficient Compute base on (8) until (5) Step 3: Computin search irection if. If, terminate the if execution of the alorithm. Step : Computin step size by exact line search rule Solve min f ( x ), Step 5: Upatin new point Let x x Step : Converent test an stoppin criteria. If f ( x ) f ( x ) an, then terminate. Otherwise o to Step with. 3 Converent Analysis SRMI he converent properties of will also be stuie. We only show the result of converence for the eneral CG metho. o prove the converence, we assume that every search irection shoul satisfy escent conition (7) for all. If there exists a constant C for all then, the search irections satisfy followin sufficiently escent conition. C (8)

6 8 Syazni Shoi et al. heorem SRMI Consier a CG metho with the line search irection (7) an iven as (), then conition () hols for all. Proof: If, then it is clear that C. Hence, conition () hols true. We also nee to show that for, conition (), will also hol true. From (7), multiply by then, ( ) For exact line search, we now that. hus,., which implies that is a sufficient escent irection. Hence, hols true. he proof is complete. C We also nee the followin assumption []. Assumptions f has lower boun, on the level set x f x) f ( x ) (i) (x) ( where x is the startin point. (ii) In a neihbourhoo N of, the function f (x) is continuously ifferentiable, an its raient is Lipschitz continuous; then, there exists a constant L such that ( x) ( y) L x y, for all x, y N. Lemma Assume that conitions (i) an (ii) hol an f ( x ) f ( x) ( x t ) x, x N. hen, t f ( x ) f ( x) ( x) L Let x an, then become x f ( x ) f ( x ) ( x t ) f ( x ) f ( x ) ( x ) L (see [5]). t

7 A new conjuate raient metho with exact line search 85 Note that, the converence properties are the same either by usin the exact or inexact. heorem If conition (3) an Assumption hols, then for any line search rule the followin converence properties hols, Proof: lim. By exact line search, mean value theorem, Cauchy-Schwartz inequality, Assumption (ii), set, then we obtaine L f ( x ) f ( x ) f ( x ) f ( x ) ( x t ) ( ( x t ) ) ( x t ) L L By Assumption (i) an (3), it follows that f ( x ) is a monotone ecreasin number sequence an has boun below, thus f ( x ) has a limit, an therefore converence properties hols [3]. Numerical Results We analyze the efficiency of the newly propose CG coefficient SRMI, as compare to other classical CG methos such as FR, PR, HS an RMIL. he comparisons are base on the number of iterations an Central Processin Unit (CPU) time per secon to reach minimizer. All these methos have been teste usin eiht ifferent stanar functions problems. Besies that, these test problems was teste several times for selecte ranes number of variable n=, an. On the other han, all the comparisons are one with four ifferent initial points, startin from a point that is closer to the solution point, to the point further away from the solution point. We consiere an all these methos terminate when the stoppin criteria is fulfille. All the problems mention below are solve by Maple3 subroutine proram usin the exact line search. We recor the number of t t t

8 8 Syazni Shoi et al. iteration an CPU time in purpose of our comparisons. he results will be shown in able an able respectively. here are two conitions where are the iteration is consiere as faile. he first conition is when the routines is stoppe since it is fail to fin the positive value of stepsize an the secon conition is when iteration is excee. In the able, the wor Fail is represent the first conition while the wor is represent the secon conition. In the able, the symbol which enote not available represente the result for both conition. We further simplifies able an able an shown the percentae performance of SRMI as compare to the other metho in able 3 an able respectively. he followin test functions are base on Anrei [8] an Mola an Smutnici [3]. ABLE : Performance comparison of ifferent CG metho base on number of Iterations No. Function Initial Point SRMI FR PR HS RMIL Rosenbroc (n=) (,) (5,5) (,) (5,5) Rosenbroc (n=) 3 Shalow (n=) Shalow (n=) 5 Cube (n=) Woo (n=) (,...,) (5,...,5) (,...,) (5,...,5) (5,5, 5,5) (,,,) (5,5, 5,5) (8,8, 8,8) (5,...,5) (,...,) (5,...,5) (8,...,8) (7,7, 7,7) (,,,) (,,,) (8,8,8,8) (3,3,3,3) (5,5,5,5) (,,,) (3,3,3,3) Fail Fail Fail Fail

9 A new conjuate raient metho with exact line search 87 ABLE : (Continue): Performance comparison of ifferent CG metho base on number of Iterations 7 Liarwh (n=) 8 hree Hump(n=) 9 Six Hump (n=) White an Holst (n=) (,) (5,5) (5,5) (,) (,-) (8,-8) (,-) (3,-3) (-,-) (-5,-5) (,) (5,5) (,,,) (8,8,8,8) (,,,) (,,,) ABLE : Performance comparison of ifferent CG metho base on CPU time No Function Initial Point (,) Rosenbroc (5,5) (n=) (,) (5,5) Rosenbroc (n=) 3 Shalow (n=) Shalow (n=) 5 Cube (n=) (,...,) (5,...,5) (,...,) (5,...,5) (5,...,5) (,...,) (5,...,5) (8,...,8) (5,...,5) (,...,) (5,...,5) (8,...,8) (7,...,7) (,...,) (,...,) (8,...,8) SRMI FR PR HS RMIL

10 88 Syazni Shoi et al. ABLE : (Continue): Performance comparison of ifferent CG metho base on CPU time Woo (n=) 7 Liarwh (n=) 8 hree Hump(n=) 9 Six Hump (n=) White an Holst (n=) (3,...,3) (5,...,5) (,...,) (3,...,3) (,) (5,5) (5,5) (,) (,-) (8,-8) (,-) (3,-3) (-,-) (-5,-5) (,) (5,5) (,...,) (8,...,8) (,...,) (,...,) Discussion From able, we see that for all iven problems, SRMI an PR successfully reach solution point without exceeinly iteration. Otherwise, in certain problems, FR HS an RMIL is consiere as faile once excee or fail to fin the positive value of stepsize. hus, in able there is no recore of CPU time for faile problems. Besies that, SRMI also outperforme FR, HS an RMIL in almost all the problems. he wors successful in able 3 means that SRMI has achieve the minimizer with less number of iterations compare to FR, PR, HS an RMIL. Besies that, SRMI to other methos et equivalent in number of iteration an the worl unsuccessful, means SRMI et worse result compare to others methos. In able, the SRMI is sai to be as successful when it has achieve the minimizer with the least uration of CPU time compare to others methos. In some problems, SRMI has achieve equivalent to others methos in CPU time to reach minimizer. he SRMI is sai to be as unsuccessful, when it neee loner time to reach minimizer compare to others methos.

11 A new conjuate raient metho with exact line search 89 ABLE 3: Percentae performance of SRMI compare to other CG methos base on number iteration Metho Comparison FR PR HS RMIL Successful 9.% 5.% 57.5% 7.5% SRMI Equivalent.% 5.% 5.%.% Unsuccessful.%.% 7.5% 7.5% From able 3, it is shown that SRMI is superior when compare to FR, HS, an RMIL. he hihest percentae of successful comparison is with FR which is 9.% an followe by RMIL which is 7.5% an HS which is 57.5%. houh the successful rate comparison for PR is the lowest at 5.%, their combine rate of successful rate an equivalent rate are equal to.%. Above all, almost all the comparisons showe that the combine rate of successful an equivalent rate excee 5.%. herefore, we consiere that, SRMI is superior compare to FR, PR, HS, an RMIL in term number of iteration. ABLE : Percentae performance of SRMI compare to other CG methos base on CPU time. Metho Comparison FR PR HS RMIL Successful 87.5%.%.% 75.% SRMI Equivalent.% 7.5%.5%.% Unsuccessful.5% 5.5% 37.5% 5.% From able, it is shown that SRMI is superior whem compare to FR, HS, an RMIL with the least uration of CPU time. he hihest percentae of successful comparison is with FR at 87.5%, followe by RMIL which is 75.% an HS which is.%. However, the successful rate comparison for PR is low at.%. Above all, almost all the comparisons showe that the combine rate of successful an equivalent rate excee 5.% except PR. herefore, we consiere that, SRMI is superior compare to FR, HS, an RMIL but inferior when compare to PR in term of CPU time. Conclusion In this paper, a new name as SRMI has been presente. Base on the result, SRMI shows that it satisfie the sufficiently escent conition. From the above numerical experiments with 8 test problems we have the computational evience that SRMI is the best when compare to others stanar CG methos. houh the successful rate of SRMI is low compare to PR, but it coul be an alternative metho when the other methos fail by usin the exact line search. For

12 8 Syazni Shoi et al. further research, we shoul o more numerical experiment with the other stanar test functions with larer scale or variables. We also hope to establish the lobal converence properties an the linear converence rate theoretically. Acnowlements. We woul lie to than the anonymous referees for valuable suestions an comments which improve our paper reatly. We also rateful to he Ministry of Hiher Eucation of Malaysia for the funin of this research uner the Funamental Research Grant Scheme (Vot 595). References [] E. Pola, an G. Ribiere, Note sur la converence e irections conjuees. Rev.Francaise Informat Recherche Operationelle, 3E Annee, (99), [] G. Zoutenij, Nonlinear Prorammin Computational Methos, in Inteer an Nonlinear Prorammin, J. Abaie (eitor), (97), [3] M. Mola an C. Smutnici, est Functions for optimization nees, (5). [] M. R. Hestenes an E. Steifel, Metho of conjuate raient for solvin linear equations, J,Res.Nat.Bur.Stan., 9 (95), [5] M. Rivaie, M. Mamat, W. J. Leon an M. Ismail, A new class of nonlinear Conjuate Graient Coefficient with lobal converence properties, Applie Mathematics an Computation, 8 (), [] M. J. D. Powell, Nonconvex Minimizations Calculations an the Conjuate Graient Metho, Lecture Notes in Mathematics, Berlin, Spriner, (98), -. [7] M. J. D. Powell, Restart Proceures for the Conjuate Graient Metho, Mathematical Prorammin, (977), [8] N. Anrei, An Unconstraine Optimization est Function Collection, J. Av. Moelin an Optimization, (8), 7-. [9] R. Fletcher an C. Reeves, Function minimization by conjuate raients, Comput.J., 7 (9), [] A. Y. Al-Bayati, an R. Z. Al-Kawaz, A new hybri WC-FR conjuate raient alorithm with moifie secant conition for unconstraine optimization. J. Math. Comp. Sci., (),

13 A new conjuate raient metho with exact line search 8 [] Y. H. Dai, J. Y. Han, G. H. Liu, D. F. Sun, X. Yin, an Y. Yuan, (Converence properties of nonlinear conjuate raient metho. SIAM J. Optim., (999), [] P. Wolfe, Converence conitions for ascent metho. II: some corrections. SIAM Rev., 3 () (97), [3] Z. J. Shi, Converence of line search methos for unconstraine optimization. Applie Mathematics an Computation., 57 (), [] W. Sun, an Y. X. Yuan, Optimization theory an metho (nonlinear prorammin). Spriner Science an Business Meia, LLC (). [5] Z. J. S. Shi, an J. Shen, On step-size estimation of line search methos. Applie Mathematics an Computation., 73 (), [] N. Shapiee, R. M. Mamat, an I. Moh, A new moification of Hestenes- Stiefel metho with escent properties, AIP Conference Proceeins, (), [7] M. Rivaie, A. Abashar, M. Mamat an I. Moh, he converence properties of a new type of conjuate raient methos, Applie Mathematical Sciences, 8 (), [8] N. H. M. Yussoff, M. Mamat, M. Rivaie an I. Moh, A new conjuate raient metho for unconstraine optimization with sufficient escent, AIP Conference Proceeins, (), [9] Y. Dai an Y. Yuan, Nonlinear conjuate raient metho, Shanhai Scientific an echnical Publisher, Beijin (998). [] Y. Yuan an W. Sun, heory an methos of optimization, Science Press of China, Beijin (999). [] A. Abashar, M. Mamat, M. Rivaie an I. Moh, Global converence properties of a new class of conjuate raient metho for unconstraine optimization, Applie Mathematics Sciences, 8 (7) (), [] S. Shoi, M. Rivaie, M. Mamat an I. Moh, Solvin unconstraine optimization with a new type of conjuate raient metho, AIP Conference Proceeins, (),

14 8 Syazni Shoi et al. [3] A. Abashar, M. Mamat, M. Rivaie, I. Moh an O. Omer, he proof of sufficient escent conition for a new type of conjuate raient methos, AIP Conference Proceeins, (), Receive: March, 5; Publishe: July 8, 5