The Algorithms of Broyden-CG for. Unconstrained Optimization Problems

Size: px

Start display at page:

Download "The Algorithms of Broyden-CG for. Unconstrained Optimization Problems"

Branden Armstrong
6 years ago
Views:

1 Internatonal Journal of Mathematcal Analyss Vol. 8, 014, no. 5, HIKARI Ltd, he Algorthms of Broyden-CG for Unconstraned Optmzaton Problems Mohd Asrul Hery Ibrahm Faculty of Entrepreneurshp and Busness Unverst Malaysa Kelantan (UMK) Kampus Kota, Malaysa Mustafa Mamat Faculty of Informatcs and Computng Unverst Sultan Zanal Abdn (UNISZA) Kampus embla, Malaysa Leong Wah June Department of Mathematcs Faculty of Scence, Unversty Putra Malaysa (UPM) Malaysa Azf Zad Sof Faculty of Scence and Informaton echnology, Kolej Unverst Islam Antarabangsa Selangor (KUIS) Malaysa Copyrght 014 Mohd Asrul Hery Ibrahm et al. hs s an open access artcle dstrbuted under the Creatve Commons Attrbuton Lcense, whch permts unrestrcted use, dstrbuton, and reproducton n any medum, provded the orgnal work s properly cted. Abstract he conjugate gradent method plays an mportant role n solvng large-scaled problems and the quas-newton method s known as the most effcent method n solvng unconstraned optmzaton problems. herefore, n ths paper, the new hybrd

2 59 Mohd Asrul Hery Ibrahm et al. method between the conjugate gradent method and the quas-newton method for solvng optmzaton problem s suggested. he Broyden famly formula s used as an approxmaton of Hessan n the hybrd method and the quas-newton method. Our numercal analyss provdes strong evdence that our Broyden-CG method s more effcent than the ordnary Broyden method. Furthermore, we also prove that new algorthm s globally convergent and gratfy the suffcent descent condton. Keywords: Broyden method, conjugate gradent method, search drecton, global convergent 1 Introducton Consder the unconstraned optmzaton problems: mnf xr n x n and let f : R R be contnuously dfferentable. he Broyden s famly method s an teratve method. On the th teraton, an approxmaton pont x and the ( 1) th teraton of x s gven by (1) where the search drecton, x 1 x d, () d s calculated by whch d B g( x ) (3) 1 g s a gradent of f. he search drecton must satsfy the relaton gd 0, whch guarantee that d s a descent drecton of f( x ) at x [1, ]. hen the step sze, n () was obtaned usng the Armjo lne search as suggested by [3] such as: such that s 0, (0,1), (0,1) max s, s, s, (4) and f( x) f( x d) g d, 0,1,,. hen, the sequence of x s converged to the optmal pont, * x, 0 whch mnmzes [4]. he updated Hessan approxmaton formula n (3), requre B postve defnte and satsfyng the quas-newton equaton where B 1 s y, (5)

3 he algorthms of Broyden-CG 593 s d y g g (6) 1. he Broyden s algorthm for unconstraned optmzaton problem uses the matrces B whch s updated by the formula B s s B y y B B s B s v v where s a scalar and 1, s B y s y v y B s s y s B s. (7) hs algorthm satsfy the quas-newton equaton (7). he choce of the parameter s mportant, snce t can greatly affect the perfomance of the method [5]. When 1 n equaton (7), we obtan the DFP algorthm and 0, we get the BFGS algorthm. But, [, 6] extended hs result to (0,1]. Based on [7], the Broyden s algorthm s one of the most effcent algorthm for solvng the unconstraned osptmzaton problem. hs paper s organsed as follows. In Secton, we elaborate the new algorthm and the convergence analyss. An explanaton about the numercal results s also gven n Secton 3 usng the performance profle s fgure. he paper ends wth a short concluson n secton 4. he Broyden-CG Algorthm he modfcaton of the quas-newton method based on a hybrd method has already been presented by prevous researchers. One of the studes s a hybrdzaton of the quas-newton and Gauss-Sedel methods, desgned n solvng the system of lnear equatons n [8]. Luo et al. [9] suggest the new hybrd method, whch can solve the system of nonlnear equatons by combnng the quas-newton method wth chaos optmzaton. Han and Newman [4] combne the quas-newton methods and Cauchy descent method to solve unconstraned optmzaton problems, whch s known as the quas-newton-sd method. Hence, the modfcaton of the quas-newton method by prevous researchers spawned the new dea of hybrdzng the classcal method to yeld the new hybrd method such as n [10-13]. Hence, ths study proposes a new hybrd search drecton that combnes the concept of search drecton of the quas-newton and CG methods. It yelds a new search drecton of the hybrd method whch s known as the Broyden-CG method. he search drecton for the Broyden-CG method s

4 594 Mohd Asrul Hery Ibrahm et al. d 1 B g 0 1 B g ( g d 1) 1 (8) where 0 and ( g g 1/ g d 1). Hence, the complete algorthms for the Broyden method and the Broyden-CG method wll be organzed n Algorthms.1 and., respectvely. Algorthm.1: Broyden Method Step0. Gven a startng pont x 0 and B0 I n. Choose values for s, and and set 1. 6 Step1. ermnate f g( x 1) 10 or Step. Calculate the search drecton by (3) Step3. Calculate the step sze by (4) Step4. Compute the dfference between s x x 1 and y g g 1. Step5. Update B 1 by (7) to obtan B. Step6. Set 1 and go to Step 1. Algorthm.: Broyden-CG Method Step0. Gven a startng pont x 0 and B0 I n. Choose values for s, and and set 1. 6 Step1. ermnate f g( x 1) 10 or Step. Calculate the search drecton by (8) Step3. Calculate the step sze by (4) Step4. Compute the dfference between s x x 1 and y g g 1. Step5. Update B 1 by (7) to obtan B. Step6. Set 1 and go to Step 1. Based on Algorthms.1 and., we assume that every search drecton satsfed the descent condton gd 0, for all 0. If there exsts a constant c1 0 such that 1 g d c g (9) for all 0, then the search drectons satsfy the suffcent descent condton d

5 he algorthms of Broyden-CG 595 whch can be proved n heorem.. Hence, we need to make a few assumptons based on the objectve functon. Assumpton.1 H1: he objectve functon f s twce contnuously dfferentable. H: he level set L s convex. Moreover, postve constants c 1 and c exst, satsfyng for all z n R and x L, 1 c z z F( x) z c z, where Fx ( ) s the Hessan matrx for f. H3: he Hessan matrx s Lpschtz contnuous at the pont exsts the postve constant c 3 satsfyng n a neghbourhood of x *. * * 3 x *, that s; there g( x) g( x ) c x x for all x heorem.1 (see [, 6]) Let B be generated by the Broyden s famly formula (7), where B 1 s symmetrc and postve defnte, and where ys 0 for all. Furthermore, assume that s and y are such that * y G s s (10) * for some symmetrc and postve defnte matrx Gx, and for some sequence wth the property and the sequence,. hen 1 B B 1 B G* d lm 0, (11) d are bound. heorem. Suppose that Assumpton.1 and heorem.1 hold. hen condton (9) holds for all 0. Proof: From (9), we see that

6 596 Mohd Asrul Hery Ibrahm et al. g d g B g g g g/ g g d d g B g g g g g/ g d g d g B g g g g g. 1 1 Based on Powell [14], g g 1 g wth (0,1], then where c g d g d g B g g g 1, g ( ) g c g whch s bound away from zero. Hence, c g holds. he proof s completed.. 3 Numercal Analyss In ths secton, we use the test problem consdered n Andre [15], Mchalewcz [16] and More et al. [17] n able 1 to analyse the mprovement of the Broyden-CG method compared wth the Broyden method. Each of the test problems are tested wth dmensons vared from to 1,000 varables. hs represents a total of 180 test problems. As suggested by [17], for each of the test problems, the ntal pont, x 0 wll further subtract from the mnmum pont. In dong so, ths leads us to test the global convergence propertes and the robustness of our method. For the Armjo lne search, we use s 1, =0.5 and 0.1. he stoppng crtera we use are g 6 10 and the number of teratons exceeds ts lmt, whch s set to be 10,000. In our mplementaton, the numercal tests were performed on an Acer Aspre wth a Wndows 7 operatng system and usng Matlab 01 to run the programmng for both methods.

7 he algorthms of Broyden-CG 597 ABLE (1): A lst of problem functons. ES PROBLEM N-DIMENSIONAL SOURCES Powell Badly Scaled More et al. [17] Beale More et al. [17] Bggs Exp6 6 More et al. [17] Chebyquad 4,6 More et al. [17] Colvlle Polynomal 4 Mchalewcz [16] Varably Dmensoned 4,8 More et al. [17] Freudensten And Roth More et al. [17] Goldsten Prce polynomal Mchalewcz [16] Hmmelblau Andre[15] Penalty 1,4 More et al. [17] Extended Powell Sngular 4,8 More et al. [17] Extended Rosenbrock,10,100,00,500, Andre [15] 1000 rgonometrc 6 Andre [15] Watson 4,8 More et al. [17] Sx-Hump Camel Back Mchalewcz [16] Polynomal Extended Shallow,4,10,100,00,500, Andre [15] 1000 Extended Strat,4,10,100,00,500, Andre [15] 1000 Scale Mchalewcz [16] Raydan 1,4 Andre [15] Raydan,4 Andre [15] Dagonal 3 Andre [15] Cube,10,100,00 More et al. [17] FIGURE 1: Broyden method versus Broyden-CG method n term of number of teratons

8 598 Mohd Asrul Hery Ibrahm et al. FIGURE : Broyden method versus Broyden-CG method n term of CPU-tme. he performance results wll be shown n Fgures 1 and, respectvely, usng the performance profle ntroduced by Dolan and More [18]. he performance profle seeks to fnd how well the solvers perform relatve to the other solvers on a set of problems. In general, P() s the fracton of problems wth performance rato, thus, a solver wth hgh values of P() or one that s located at the top rght of the fgure s preferable. Fgures 1 and show that the Broyden-CG method has the best performance snce t can solve 84% of the test problems compared wth the Broyden method (83%). Moreover, we can also say that the Broyden-CG s the fastest solver on approxmately 64% of the test problems for teraton and 69% of CPU-tme. 4 Concluson We have presented a new search drecton for Broyden method for solvng unconstraned optmzaton problems. he performance profle for a broad class of test problems show that the Broyden-CG method s effcent and robust n solvng unconstraned optmzaton problem. We also note that as the sze and complexty of the problem ncrease, greater mprovements could be realsed by our Broyden-CG method. Our future research wll be to try the Broyden-CG wth another formula for k. References [1] Nocedal, J. and S. J. Wrght, Numercal Optmzaton. Sprnger Seres n Operatons Research and Fnancal Engneerng. 006: Sprnger. [] Byrd, R. H. and F. Nocedal, A ool for the Analyss of Quas-Newton Methods wth Applcaton to Unconstraned Mnmzaton. SIAM Journal on Numercal Analyss, (3): p

9 he algorthms of Broyden-CG 599 [3] Armjo, L., Mnmzaton of functons havng Lpschtz contnuous partal dervatves. Pacfc Journal of Mathematcs, (1): p [4] Han, L. and M. Neumann, Combnng quas-newton and Cauchy drectons. Internatonal Journal of Appled Mathematcs, (): p [5] Xu, D.-c., Global Convergence of the Broyden's Class of Quas-Newton Methods wth Nonmonotone Lnesearch. Acta Mathematcae Applcatae Snca, (1): p [6] Byrd, R. H., J. Nocedal, and Y.-X. Yuan, Global Convergence of a Class of Quas-Newton Methods on Convex Problems. SIAM Journal on Numercal Analyss, (5): p [7] Chong, E. K. P. and S. H. Zak, An Introducton to Optmzaton. Second ed. 001: A Wley-Interscence Publcaton. [8] Ludwg, A., he Gauss Sedel quas-newton method: A hybrd algorthm for solvng dynamc economc models. Journal of Economc Dynamcs and Control, (5): p [9] Luo, Y.-Z., G.-J. ang, and L.-N. Zhou, Hybrd approach for solvng systems of nonlnear equatons usng chaos optmzaton and quas-newton method. Appled Soft Computng, (): p [10] Ibrahm, M. A. H., M. Mamat, and W. J. Leong, he Hybrd BFGS-CG Method n Solvng Unconstraned Optmzaton Problems. Abstract and Appled Analyss, : p [11] Ibrahm, M. A. H., et al., Alternatve Algorthms Of Broyden Famlyam: For Unconstraned Optmzaton. AIP Conference Proceedngs, (1): p [1] Ibrahm, M. A. H. B., et al., he CG-BFGS method for unconstraned optmzaton problems. AIP Conference Proceedngs, : p [13] Mamat, M., et al., Hybrd Broyden Method for Unconstraned Optmzaton. Internatonal Journal of Numercal Methods and Applcatons, (): p

10 600 Mohd Asrul Hery Ibrahm et al. [14] Powell, M. J. D., Restart procedures for the conjugate gradent method. Mathematcal Programmng, (1): p [15] Andre, N., An Unconstraned Optmzaton est Functons Collecton. Advanced Modellng and Optmzaton, (1): p [16] Zbgnew Mchalewcz, Genetc Algorthms + Data Structures = Evoluton Programs. 1996: Sprnger Verlag. [17] More, J. J., B. S. Garbow, and K. E. Hllstrom, estng Unconstraned Optmzaton Software. ACM rans. Math. Softw., (1): p [18] Dolan, E. D. and J. J. Moré, Benchmarkng optmzaton software wth performance profles. Mathematcal Programmng, (): p Receved: September 9, 014; Publshed: November 0, 014

The Order Relation and Trace Inequalities for. Hermitian Operators

The Order Relation and Trace Inequalities for. Hermitian Operators Internatonal Mathematcal Forum, Vol 3, 08, no, 507-57 HIKARI Ltd, wwwm-hkarcom https://doorg/0988/mf088055 The Order Relaton and Trace Inequaltes for Hermtan Operators Y Huang School of Informaton Scence