A Global Convergent Spectral Conjugate Gradient Method

Transcription

1 Australian Journal of Basi an Applie Sienes 77: ISSN A Global Converent Spetral Conjuate Graient Metho Abbas Y. Al-Bayati an Hawraz N. Al-Khayat Collee of Basi Euation elafer Mosul University Iraq. Collee of Computer Sienes an Mathematis Mosul University Mosul Iraq. Abstrat: In this paper we are onerne with the Conjuate Graient CG methos for solvin unonstraine optimization problems. It is well-nown that the iretion enerate by a CG-metho may not be a esent iretion of the objetive funtion. In this paper we have one a little moifiation to the Conjuate Desent metho suh that the iretion enerate by the moifie metho provies a esent iretion for the objetive funtion. his property epens neither on the line searh use nor on the onveity of the objetive funtion. Moreover the moifie metho reues to the stanar metho if line searh is eat. Uner mil onitions we prove that the moifie metho with stron Wolfe line searh is lobally onverent even if the objetive funtion is non onve. We also present some numerial results to show the effiieny of the propose metho. Key wors: Spetral Conjuate Graient Global Converene Unonstraine Optimization Desent Diretion Line Searh. INRODUCION Our aim in this paper is to stuy the lobal onverene properties an pratial omputational performane of a new nonlinear spetral CG-metho for unonstraine optimization with Powell restartin riterion an with appropriate onitions. We onsier the followin unonstraine optimization problem: n minf R. where f : R n R is a ontinuously ifferentiable funtion. Nonlinear CG-methos are effiient for solvin.. he nonlinear CG-methos enerate iterates by lettin : with if 0.3 if where is the urrent iteration 0 is the step-lenth whih is etermine by some line searh is the new searh iretion enotes the raient of f at an is a suitable parameter. here are many well-nown formulas for suh as the Flether-Reeves FR Rayan M. 997 Pola- Ribiere PR Anrei N. 008 Hestenes-Stiefel HS Hestenes M.R. an E. Stiefel 95 an Conjuate- Desent Wolfe P he CG-metho is a powerful line searh metho for solvin optimization problems an it remains very popular for enineers an mathematiians who are intereste in solvin laresale problems. his metho an avoi lie Steepest Desent SD metho the omputation an storae of some matries assoiate with the Hessian of objetive funtions. he oriinal -metho propose by Flether Wolfe P. 969 in whih is efine by the followin:.4 where. enotes the Euliean norm of vetors. An important property of the - metho is that the metho will proue a esent iretion uner the stron Wolfe line searh. In the stron Wolfe line searh the step-lenth is require to satisfy the followin: f f.5 Corresponin Author: Abbas Y. Al-Bayati Collee of Basi Euation elafer Mosul University Iraq. profabbasalbayati@yahoo.om 30

2 Aust. J. Basi & Appl. Si. 77: where 0. Another popular metho to solvin problem. is the Spetral Graient SG metho whih was evelope oriinally by Barzilai an Borwein Birin E.G. an J.M. Martinez 00. In 988 Rayan Liu J. an Y. Jian 0 further introue the SG-metho for potentially lare-sale unonstraine optimization problems. he main feature of this metho is that only raient iretions are use at eah line searh whereas a non-monotone stratey uarantees lobal onverene. As well as this metho outperforms sophistiate CGmetho in many problems. Birin an Martinez Al-Baali M. 985 propose three ins of spetral CGmethos. he iretion is iven by the followin way.6 where the parameter is ompute the followin y s y 3.7 s y respetively an is taen to be the spetral raient an ompute by the followin s s.8 s y where y s. he numerial results show that these methos are very effetive. Unfortunately they annot uarantee to enerate esent iretions. More spetral CG-methos have also been reporte in Gilbert J.C. an J. Noeal 99; Liu J. et al. 0; Pola E. an G. Ribiere 969. In the ase where Armijo-type line searh or Wolfe-type line searh is use the esent property of etermine by.3 is in eneral not uarantee. In orer to ensure esent property Dion Gilbert J.C. an J. Noeal 99 an Al-Baali Haer W.W. an H. Zhan 005 sueste to use the SD-iretion instea of etermine by.3 in the ase where is not a esent iretion. By the use of this hybri tehnique Dion Gilbert J.C. an J. Noeal 99 an Al-Baali Haer W.W. an H. Zhan 005 obtaine the lobal onverene of the CG-methos with some ineat line searhes. Quite reently it is notie that there are many moifie CG-methos stuie. Liu et al. 0 tae moifiation to the -metho suh that the iretion enerate is always a esent iretion an is efine by the followin: if.9 if where is speifie by the followin if else an. hey prove that this metho an uarantee to enerate esent iretions an is lobally onverent. he paper has the followin struture; in the net setion a new spetral CG-metho is propose. Setion 3 will be evote to prove the lobal onverene of the new propose metho. In Setion 4 some numerial eperiments will been reporte to test the effiieny espeially in omparison with the eistin other methos. Some onluin remars will be iven in the last setion.. Spetral Conjuate Desent Metho: In this setion we have first to investiate how to etermine a esent iretion of objetive funtion. Let be the urrent iterate. Let be efine by 303

3 Aust. J. Basi & Appl. Si. 77: if if. where is speifie by.4 an let us onsier the followin new parameter: y. he new metho reues to the stanar metho if the line searh is eat. But enerally we refer to use the ineat line searh s.t. Wolfe line searh. We first prove that is a suffiiently esent iretion.. Lemma: Suppose that is iven by. an.. Furthermore assume that satisfies stron Wolfe onition.5 with 5 0. an if Powell restart is use i.e. 0.. hen the followin result.3 hols for any 0. Proof. If 0 then. hen from.4. an. it is follows that: y y From seon Wolfe.5 yiels sine the Powell restartin riterion Liu J. et al. 0 is efine as follows: where

4 Aust. J. Basi & Appl. Si. 77: an we obtain the esire result. From Lemma. it is nown that is a esent iretion of f at Furthermore if the eat line searh is use then: y In this ase the propose spetral -metho reues to the stanar - metho However it is often that the eat line searh is time-onsumin an sometimes is unneessary. In the followin we are oin to evelop a new alorithm where the searh iretion is hosen by.-. an the step-lenth is etermine by stron Wolfe-type ineat line searh.. Alorithm. Step : Initialization: ae n 0 R an the parameter 0. Compute f 0 an 0 f 0 an set 0 0 for 0. Step : Computation of the Line Searh: Compute satisfyin Wolfe onitions s.t: f f where 0 an then evaluate Step 3: est for Converene: If 05 or 0 0 f is satisfie then the iterations are stoppe. Step 4: Restartin Criterion: If Powell restartin riterion s.t. 0. is satisfie then o a restart step by SD iretion; otherwise ontinue. Step 5: Computation of the Salar Parameters: ompute the followin parameter from: y an Step 6: Searh Diretion: Compute the new searh iretion as Step 7: Set =+ an o to Step. It is well nown that if f is boune alon the iretion then there eists a step lenth α satisfyin the Wolfe line searh onitions.5. In our alorithm when the Powell restartin onition.6 is satisfie then we restart the alorithm with the neative raient. More sophistiate reasons for restartin the alorithms have been propose in the literature Pola E. an G. Ribiere 969; Powell M.J.D. 977 but we are intereste in the performane of a CG-Alorithm that uses this restart riterion assoiate to a iretion satisfyin the onjuay onition. Uner reasonable assumptions onitions.5 an.6 are suffiient to prove the lobal onverene of the alorithm. 305

5 Aust. J. Basi & Appl. Si. 77: Converene Analysis: In this setion we are in a position to stuy the lobal onverene of Alorithm..We first state the followin mil assumptions whih will be use in the proof of lobal onverene property. Assumption H: i he level set } : { f f R S n is boune where is the startin point. ii In a neihborhoo Ω of S f is ontinuously ifferentiable an its raient is Lipshitz ontinuously namely there eists a onstant 0 L suh that - L - 3. Obviously from the Assumption H i there eists a positive onstant D suh that: } ma{ S D 3. where D is the iameter of Ω. From Assumption H ii we also now that there eists a onstant 0 suh that: S 3.3 On some stuies of the CG-methos the suffiient esent or esent onition plays an important role. Unfortunately this onition is har to hol. 3. heorem.: Uner Assumptions H i an H ii suppose that is iven by. an. where satisfies stron Wolfe onition.5 with 5 0. then it hols that 0 inf lim 3.4 Proof. Suppose that there eists a positive onstant 0 suh that 3.5 For all. hen from. it follows that 3.6 Diviin the both sies of the above equality by then from an 3.6 we obtain: 4 4

6 Aust. J. Basi & Appl. Si. 77: Sine So that hus Whih is ontrary to proof this theorem. Hene the proof is omplete. In the last year we interest to the eneral nonlinear funtions an the onverene analysis that often eploits insihts evelope by Gilbert an Noeal 99 by Haer an Zhan 005. he lobal onverene proof of the new alorithm is base on the Zoutenij onition 970 ombine with the analysis showin that the suffiient esent onition hols an is boune. 4. Numerial Eperiments: he main wor of this setion is to report the performane of the new methos on a set of test problems. he oes were written in Fortran an in ouble preision arithmeti. All the tests were performe on a PC. Our eperiments were performe on a set of 35-nonlinear unonstraine problems that have seon erivatives available. hese test problems are ontribute in CUE Flether R. 987 an their etails are iven in the Appeni. for eah test funtion we have onsiere 0 numerial eperiments with number of variable n= In orer to assess the reliability of our new propose methos we have teste them aainst stanar & FR lassial CG-methos an MFR usin the same test problems. All these methos terminate when the followin stoppin riterion is met. 05 or 0 0 f 4. we also fore these routines stoppe if the iterations eee 000 or the number of funtion evaluations reah 000 without ahievin the minimum. We use in the Wolfe line searh routine. ables 4. ompares some numerial result for spetral CG-methos aainst & FR & MFR CGmethos respetively this table iniate for n as a imension of the problem;noi number of iterations; NOFG number of funtion an raient evaluation;ime the total time require to omplete the evaluation proess for eah test problem. In able 4. we have ompare the perentae performane of the new spetral CG-metho an & FRCG an MFR methos tain over all the tools as 00%. In orer to summarize our numerial results we have onerne only on the total of ifferent imensions n= for all tools use in these omparisons. able 4.: Comparison between new spetral CG-metho an FRCG an MFR-CG methos for the total of n ifferent imensions n= for eah test problems. Prob. Classial metho Classial FR metho Moifie FR metho spetral metho NOI NOFG CPU NOI NOFG CPU NOI NOFG CPU NOI NOFG CPU

7 Aust. J. Basi & Appl. Si. 77: otal Perentae performane of the new alorithms aainst 00% FR MFR alorithms respetively as follows in ables an 4.4. able 4.: ools Classial Metho Spetral Metho NOI 00% 43.5 % NOFG 00% 3.6 % CPU 00% 4.8 % Clearly from the above table we have foun that the new propose alorithm beats lassial alorithm in about 56.5% NOI; 67.4% NOFG an 57.% ime. able 4.3: ools Classial FR Metho Spetral Metho NOI 00% 60. % NOFG 00% 66 % CPU 00% 56.8 % Clearly from the above table we have foun that the new propose alorithm beats lassial FR alorithm in about 39.8% NOI; 34% NOFG an 43.% ime. able 4.4: ools MFR Metho Spetral Metho NOI 00% 69. % NOFG 00% 7.7 % CPU 00% 66.4 % Clearly from the above table we have foun that the new propose alorithm beats moifie FR alorithm in about 30.8% NOI; 7.3% NOFG an 33.6% ime. Appeni. rionometri Penalty 3Rayan 4Haer 5Generalize ri-iaonal 6Etene hree Ep-erms 7Diaonal4 8Diaonal 9Etene Himmelblau 0Etene PSC Etene BD Etene Quarati Penalty QP 3Etene EP 4Etene riiaonal- 5ARWHEAD CUE 6DIXMAANA CUE 7DIXMAANB CUE 8DIXMAANC CUE 9EDENSCH CUE 0DIAGONAL-6 ENGVAL CUE DENSCHNA CUE 3DENSCHNC CUE 4DENSCHNB CUE 5DENSCHNF CUE 6Etene Blo-Diaonal BD 7Generalize quartigq 8DIAGONAL 7 9DIAGONAL-8 30Full Hessian 3SINCOS 3Generalize quarti GQ 33ARGLINB CUE 34HIMMELBG CUE 35HIMMELBH CUE. REFERENCES 308

8 Aust. J. Basi & Appl. Si. 77: Al-Baali M ''Desent Property an Global Converene of he Flether Reeves Metho with Ineat Line Searh''. IMA Journal Numer. Anal. 5: -4. Anrei N ''40 Conjuate Graient Alorithms for Unonstraine Optimization a Survey on their Definition''. ICI ehnial Report. No. 3/08 Marh 4. Barzilai J. an J.M. Borwein 988. ''wo-point Step Size Graient Methos''. IMA Journal of Numerial Analysis. 8: Birin E.G. an J.M. Martinez 00. ''A Spetral Conjuate Graient Metho for Unonstraine Optimization''. Applie Mathematis an Optimization 43: 7-8. Bonartz K.E. A.R. Conn N.I.M. Goul an P.L. oint 995. ''CUE: Constraine an Unonstraine estin Environments''. ACM rans Math. Software. : Dion L.C.W Nonlinear Optimization: a Survey of the State of the Art. In: Evans D.J. es. Software for Numerial Mathematis. Aaemi Yor pp: Flether R. an C.M. Reeves 964. ''Funtion Minimization by Conjuate Graients'' he Computer Journal. 7: Flether R Pratial Methos of Optimization: Unonstraine Optimization. John Wiley & Sons Yor NY USA. Gilbert J.C. an J. Noeal 99. ''Global Converene Properties of Conjuate Graient Methos for Optimization''. SIAM Journal Optimization : -4. Haer W.W. an H. Zhan 005. ''A Conjuate Graient Metho with Guarantee Desent an an Effiient Line Searh''. SIAM Journal Optimization. 6: Hestenes M.R. an E. Stiefel 95. ''Methos of Conjuate Graients for Solvin Linear Systems''. Journal of Researh of the National Bureau of Stanars. 49: Liu J. an Y. Jian 0. ''Global Converene of a Spetral Conjuate Graient Metho for Unonstraine Optimization''. Hinawi Publishin Corporation Abstrat an Applie Analysis. Artile ID oi:0.55/0/ Liu J. X. Du an K. Wan 0. ''A Mie Spetral -DY Conjuate Graient Metho''. Hinawi Publishin Corporation Journal of Applie Mathematis. Artile ID oi:0.55/0/ Pola E. an G. Ribiere 969. ''Note Sur la Converene e Methos e Diretions Conjuates'' 36: Powell M.J.D ''Restart Proeures for the Conjuate Graient Metho''. Mathematial Proram : Rayan M ''he Barzilai an Borwein Graient Metho for the Lare Sale Unonstraine Minimization Problem''. SIAM Journal on Optimization. 7: Wolfe P ''Converene Conitions for Asent Methos''. SIAM Rev. : Zoutenij G ''Nonlinear Prorammin Computational Methos in Inteer an Nonlinear Prorammin''. North-Hollan Amsteram. pp: