An Improved BFGS Search Direction Using. Exact Line Search for Solving. Unconstrained Optimization Problems

Size: px

Start display at page:

Download "An Improved BFGS Search Direction Using. Exact Line Search for Solving. Unconstrained Optimization Problems"

Tyrone Gilmore
6 years ago
Views:

1 Applied Mathematical Scieces, Vol. 7, 2013, o. 2, A Improved BGS Search Directio Usi Exact Lie Search for Solvi Ucostraied Optimizatio Problems A. Z. M. Sofi 1, M. Mamat 2 ad I. Mohd 3 1 aculty of Sciece ad Iformatio echoloy, Kolej Uiversiti Islam Atarabasa Selaor (KUIS) Badar Seri Putra, Kaja Selaor Darul Ehsa, Malaysia 2,3 Departmet of Mathematics, aculty of Sciece ad echoloy Uiversiti Malaysia ereau (UM) Kuala ereau, ereau, Malaysia 1 azfizaidi@mail.com Abstract BGS is oe of the Hessia update formula i the well ow Quasi-Newto method. I this paper, we itroduced a parametric hybrid search directio for BGS alorithm usi the cojuate radiet miimizatio techique. Uder suitable coditios/ suitable parameter, we proved that the proposed parametric hybrid search directio is lobally covere by usi the exact lie search. At the ed of this paper, we showed the umerical results of proposed BGS alorithm based o umber of iteratio, umber of fuctio evaluatio ad CPU time computi of the several of ucostraied optimizatio problems tested. Mathematics Subject Classificatio: Keywords: Quasi-Newto, search directio, lobal coverece, superliear coverece, exact lie search

2 74 A. Z. M. Sofi, M. Mamat ad I. Mohd I ucostraied optimizatio, we eed to mi f x ( x) (1.1) R where f is twice cotiuously differetiable fuctio from R ito R ad there several of methods that ca be used to solve (1.1) ad oe of pioeer method is the Newto method. However, i Newto method, the cost to calculate the exact Hessia is quite hih [17] ad hihly time computi ad it happe worse i the case of larer scale ad hiher dimesio of the objective fuctios. herefore, the Quasi-Newto method developed for solvi (1.1). By ot cosideri the exact Hessia, Quasi-Newto method was developed usi the Hessia approximatio formula at every of iteratio ad we call it as update Hessia approximatio formula at ( + 1) th iteratio. here are several of update Hessia formulas i Quasi-Newto method such as BGS, DP, PSB, SR1, Broyde amily update ad other updates by [2], [26], [8], [10], [11] ad self scali techique for the update such as [21], [6] ad may others. However, the most popular ad most effective update is BGS update Hessia formula fouded i 1970 ad it is supported by [1, 2], [16],[ 23-25] ad prove by [5]. he BGS method is a iterative method, whereby at the ( + 1) th iteratio, x + 1 is ive by x = 1 x + + αd (1.2) where d deotes the search directio ad α is its step leth. he search directio, d is calculated usi 1 d = B. (1.3) I (1.3), the quatity = f ( x ) deotes the radiet of f at x while B is the 2 Hessia approximatio f ( x ) that fulfills the Quasi-Newto equatio B s = y (1.4) BGS update Hessia is come from Broyde family update Hessia which is derived by B B B s s B y y s B s v v whe the scalar φ = 0 ad + 1 = ( ), + + φ s B y s y (1.5) y B s v =. s s B s his update satisfied the quasi-newto equatio (1.4). φ is the parameter i φ. Wheφ = 0, BGS update Hessia formula will be derived iterval where [ 0,1] ad DP update Hessia ca be derived wheφ = 1.

3 Improved BGS search directio 75 α i (1.2) is the step leth obtaied by lie search procedure ad there are various of procedures ca be used. or all the lie search procedures, it was divided ito two types that are usi the exact lie search ad aother oe is without exact lie search as proposed by [4], [19], [20] ad others. However, this paper oly cosidered BGS alorithm usi the exact lie search as suested by [7], [22], [12] ad [9] that is f ( x + α d ). (1.6) mi α 0 he exact lie search i (1.6) must satisfy both Wolfe s coditios ad ( + α ) ( ) + β1α, (1.8) f x d f x d ( + α ) β2, (1.9) x d d d 1 with 0 < β 1 < ad β1 < β2 < Parametric Hybrid Search Directio for BGS Alorithm Hybrid search directio i Quasi-Newto for BGS update Hessia is the ewbor i the optimizatio area ad it was proposed by [13-15] usi the Quasi- Newto search directio ad stadard steepest descet radiet the cotiued by [22] usi the same techique with exact lie search procedures ad they were prove to be super liear ad lobally covere with Wolfe coditios (1.8) ad (1.9) is satisfied. he, the research usi hybrid search directio o Quasi- Newto for solvi ucostraied optimizatio problems was cotiued by [9] usi the iexact lie search i Broyde family update Hessia ad they were prove to covere efficietly for the ucostraied optimizatio problems tested. However, all the hybrid search directio i [13-15], [22] ad [9] do ot have the parameter boudary ad this stuatio will lead to the possibility of failure to the method for solvi some of the ucostraied optimizatio. Hece, proposed i this paper is the ew hybrid search directio usi the cojuate radiet idea this method ca be cosider oe of cotiuum of the method that writte i [18]. Defiitio 3.1 Suppose that x R if x R ad 0,1,... =.he sequece of { x } is said to be covere to lim( x x ) = 0. he Defiitio 3.1 is the coverece defiitio for the iteratio type method. Assumptio 3.1 he level set ξ is bouded

4 76 A. Z. M. Sofi, M. Mamat ad I. Mohd if f : R R ad f C. { x : f ( x) f ( x0 )} ξ = Assumptio 3.2 he fuctio f is Lipschitz cotiuous ad differetiable i the eihborhood Ω of ξ, there exist a positive costat, c > 0 for all x, x Ω ad c R. H ( x) H ( x ) c x x Lemma 3.1 If Assumptio 3.1 ad Assumptio 3.2 hold ad suppose that{ x } for all x, x Ω. Lemma 3.2 Suppose that Lemma 3.1 holds ad f ( x ) < f ( x) lim( x x ) = 0, the, there exist a search directio d = H G where = 1 that satisfy G satisfyi x, the f ( x ) f ( x 1 α 1d 1) f ( x α d ) f ( x0 ) * with α obtai the exact lie search (1.6) ad satisfy the Wolfe coditios (1.7) ad (1.8). heorem 3.1 Suppose that Lemma 3.1 holds toether Lemma 3.2, the the search directio d = H G (3.1) will esure{ x } x. Proof: By tai the search directio at ( + 1) iteratio, the search directio d = H he, let it be multiplied at the both sided with, d = H ad by ad assumi + 1 0, we will obtai

5 Improved BGS search directio 77 with δ 0 adη = δ + 1 η (3.2) d H he by Quasi-Newto equatio B s = y ad Wolfe coditios, we obtai + 1 y + 1 = ( ( x α + 1( H )) + 1 ( 1). β + 1 herefore by Lipschitz coditio, we obtai ( ( x α + 1( H )) + 1 α + 1L d. + 1 Straihtforward from both iequalities above, we obtai ad that implies α β β =. 2 2 L d L + 1 d d 1 + α d β 0 L > 2 = = + ad it is obvious here that whe, the rom (3.2), d = δ H η = d = δ H η with δ 0 ad η 0 ad by assumi H + 1 is positive defiite, thus lim = 0. Hece, heorem 3.1 toether with Lemma 3.1 ad Lemma 3.2 will esure the search directio proposed will esure { x } x ad covere lobally heorem 3.2 Whe theorem 3.1 holds, the exists η (0,1) where d = H ηg for f : R R ad η R that will lead lim = 0. Proof: By cotradictio, cosider the riht side value of the iterval, [1, ), we obtai

6 78 A. Z. M. Sofi, M. Mamat ad I. Mohd 1 <... < η 2 < η 1 <... < η (3.3) forη R. By choosi ay η R i (3.3), it will lead to η G =, for all > 0. he, cosider the left side value of the iterval, (,0], we obtai η <... < η 3 < η 2 <... < 0 (3.4) forη R. By choosi ay η R i (3.4), it will lead to η G =, for all > 0. At the same time, clearly we ca see by choosi ay η R i (3.4), it does ot fit (3.1) i the heorem 3.1. By usi η R i (3.3) ad (3.4), it will lead to lim d =, ad + lim d =. his is certaily aaist the lobal coverece theorem lim = 0. Here, clearly it ca be prove that the selectio of η R outside the boudaries of iterval η (0,1) will lead the search directio to ad. Prove here that the sufficietly small positive η R i the iterval η (0,1) will esure { x } x Corollary 3.1 or f : R R, the search directio d = H ηg (3.5) will esure to lim = 0 for ay Hessia update formula H + 1. Proof: rom the Quasi-Newto equatio B s = y ad y = + 1, 1 1 B = α B ηg (3.6) ( ) ( ) with s = α d. (Noted here that B + 1 = H + 1 ) By usi the exact lie search α that satisfy (1.6), ad from (3.2), the equatio (3.6) will be 1 1 G α B + 1 ( + 1 ) = B + 1 ( δ ) α ( η ) 1. B + 1 B are bouded ad (3.6) will implies Suppose that { } H ad { } ( η ) ( δ ) α G α lim = lim = lim = 0. 1 ( 1 ) + B + 1

7 Improved BGS search directio 79 It ca be see i [22] that sequeces { H } ad { } B are prove bouded. Clearly we ca see here for ay Hessia update formula H + 1, the search directio (3.5) will esure lim = 0 2. Numerical Results We start this sectio with the proposed BGS alorithm the followed by the umerical results. Alorithm 3.1: HBGS 2 Step 1 : Iitializatio of data f C, x D, η ( 0,1). Step 2 : Set = 0 ad B : 0 = I. Step 3 : or the search directio 1 d = B, for = 0, d = H ηg, for > 0, 2 2 with =. G 1 * Step 4 : Calculate exact lie search α usi (1.6) ad set * x = 1 x + + αd. * Exact lie search α forced to satisfy (1.8) ad (1.9). 8 Step 5 : Set the stoppi criteria ε = 10. If + 1 ε, the stop. 1 Step 6 : Calculate B * + 1 i (1.5) with φ = 0, s = αd ad y = + 1. Set = + 1. No. ested Problems Dim 1 Strait 2 2 Strait 4 3 Strait 6 4 Strait 10 5 Quartic 2 6 Rosebroc 2 7 Rosebroc 4 8 Rosebroc 6 9 Rosebroc Himmelblau I 2 11 Himmelblau II 2 12 Cube 2 13 Cube 4

8 80 A. Z. M. Sofi, M. Mamat ad I. Mohd 14 Cube 6 15 Cube hree Hump Camel 2 17 Shalow 2 18 Shalow 4 19 Shalow 6 20 Shalow Gold & Price 2 22 NONSCOMP 2 (CUE) 23 NONSCOMP 3 (CUE) 24 NONSCOMP 4 (CUE) 25 Beale 2 26 Beale 4 able 3.1 : Ucostraied tested fuctios able 3.1 shows several of ucostraied optimizatio tested problems. Others of ucostraied optimizatio tested fuctios ca be foud i [3]. or each of the problem tested, 4 radom poits were selected to start the alorithm. No Iitial Poit, x 0 1 (50,50) (30,60) (100,100) (-50,250) 2 (50,50,50,50) (30,60,30,60) (100,100,100,100) (-50,250,-50,250) 3 (50,50,,50,50) (30,60,,30,60) (100,100,,100,100) (-50,250,,-50,250) 4 (50,50,,50,50) (30,60,,30,60) (100,100,,100,100) (-50,250,,-50,250) 5 (3,-3) (-15,50) (40,60) (100,250) BGS i/f/time 8/ 24/ / 24/ / 27/ / 30/ / 24/ / 24/ / 27/ / 30/ / 24/ / 24/ / 27/ / 30/ / 24/ / 24/ / 27/ / 30/ / 30/ / 33/ / 27/ / 30/ HBGS i/f/time 7/ 21/ / 21/ / 21/ / 27/ / 21/ / 21/ / 21/ / 27/ / 21/ / 21/ / 21/ / 27/ / 21/ / 21/ / 21/ / 27/ / 27/ / 27/ / 21/ / 27/ 0.344

9 Improved BGS search directio 81 6 (4,-4) (8,-8) (10,-5) (20,20) 7 (0,0,0,0) (1.5,2,1.5,2) (13,13,13,13) (10,20,10,20) 8 (0,0,,0,0) (1.5,2,,1.5,2) (13,13,,13,13) (10,20,,10,20) 9 (0,0,,0,0) (1.5,2,,1.5,2) (13,13,,13,13) (10,20,,10,20) 10 (-100,100) (-1000,1000) (-10000,10000) ( ,100000) 11 (20,20) (200,200) (2000,2000) (20000,20000) 12 (2,-2) (5,-10) (-1.2,1.6) (-2,4) 13 (-1.2,1.6,-1.2,1.6) (2,-2,2,-2) (-6,8,-6,8) (-2,4,-2,4) 14 (-1.2,1.6,,-1.2,1.6) (2,-2,,2,-2) (-6,8,,-6,8) (-2,4,,-2,4) 15 (-1.2,1.6,,-1.2,1.6) (2,-2,,2,-2) (-6,8,,-6,8) (-2,4,,-2,4) 16/ 48/ / 66/ / 117/ / 45/ / 45/ / 84/ / 99/ / /42/ / 42/ / 84/ / 99/ / 42/ / 84/ / 99/ / 3/ / 3/ / 3/ / 3/ / 18/ / 18/ / 18/ / 18/ / 155/ / 245/ / 135/ / 180/ / 135/ / 155/ / 175/ / 135/ / 45/ / 48/ / 45/ / 90/ / 42/ / 42/ / 69/ / 90/ / 39/ / 36/ / 69/ / 90/ / 42/ / 36/ / 69/ / 90/ / 3/ / 3/ / 3/ / 3/ / 18/ / 18/ / 18/ / 18/ / 150/ / 215/ / 40/ / 120/ / 40/ / 155/ / 160/ / 120/ / 40/ / 150/ / 155/ / 120/ / 40/ / 155/ / 160/ / 120/ 2.000

10 82 A. Z. M. Sofi, M. Mamat ad I. Mohd 16 (2,-2) (5,-10) (50,-100) (-200,200) 17 (8,15) (-30,60) (100,250) (30,50) 18 (8,15,8,15) (-30,60,-30,60) (30,50,30,50) (100,250,100,250) 19 (8,15,,8,15) (-30,60,,-30,60) (30,50,,30,50) (100,250,,100,250) 20 (8,15,,8,15) (-30,60,,-30,60) (30,50,,30,50) (100,250,,100,250) 21 (1,0) (0.5,-0.5) (1,-1) (2,-3) 22 (3,-3) (-5,5) (15,25) (10,10) 23 (0.5,-1.5,0.5) (-1,3,-1) (3,-3,3) (-5,5,-5) 24 (3,-3,3,-3) (-5,5,-5,5) (15,25,15,25) (10,10,10,10) 25 (-1,1) (2,5) (-3,-3) (10,12) 26 (-1,1,-1,1) (2,5,2,5) (-3,-3,-3,-3) (10,12,10,12) 4/ 20/ / 30/ / 30/ / 40/ / 33/ / 57/ / 63/ / 48/ / 33/ / 57/ / 48/ / 63/ / 33/ / 57/ / 48/ / 63/ / 33/ / 57/ / 48/ / 63/ / 33/ / 45/ / 57/ / 42/ / 54/ / 57/ / 66/ / 66/ / 75/ / 111/ / 171/ / 147/ / 84/ / 154/ / 84/ / 91/ / 112/ / 112/ / 20/ / 25/ / 25/ / 35/ / 27/ / 48/ / 57/ / 39/ / 27/ / 48/ / 39/ / 57/ / 27/ / 48/ / 39/ / 57/ / 27/ / 48/ / 39/ / 57/ / 42/ / 56/ / 56/ / 56/ / 27/ / 33/ / 39/ / 27/ / 39/ / 45/ / 57/ / 63/ / 72/ / 99/ / 138/ / 144/ / 84/ / 91/ / 105/ / 27/ / 84/ / 91/ / 112/ / 63/ able 3.2 : Numerical results of BGS ad HBGS

11 Improved BGS search directio 83 able 3.2 show the umerical result for eeral BGS deoted by BGS ad proposed BGS deoted as HBGS. i = umber of iteratio f = umber of fuctio evaluatio time = time of CPU used to ru the proram he Maple prorammi lauae was used to ru the both alorithms ad was ru o AMD Athlo 7750 Dual Core Processor 2.71 Ghz CPU ad 1 Gb of RAM. 3. Discussio I able 3.2, there are 16 of s for eeral BGS alorithm which bris % of failure for the problems tested usi the exact lie search ad 0 of i HBGS. Pheomea that cause s i BGS alorithm are because of: 1.A usual stop occur before the alorithm reach the stoppi criteria. 2.he alorithm still caot fid the solutio althouh it reach umber of iteratio 10, he solutio poit obtaied is ot the real solutio. (it meas the solutio determied by the alorithm does ot satisfied the miimizatio properties ( ) ( α ) ( ) ( ) f x + 1 = f x + d f x f x0. Based o able 3.2, HBGS outperformed BGS with 87.5% with 91 of the total outcomes ad had aother 12.5% of equivalet results. We ca see i able 3.2 that HBGS was outperformed BGS based o umber of iteratios, umber of fuctio evaluatio ad CPU time computi to solve the problems. 4. Coclusio We have proposed a alorithm for BGS method ad we prove that uder suitable coditios, our proposed alorithm is lobally covere ad outperformed eeral BGS alorithm. However, we lie to study the proposed alorithm further without the exact lie search. Refereces [1] Al-Baali, M. (1998). Numerical experiece with a class of self-scali quasi-ewto alorithms, Joural of Optimizatio heory ad Applicatios, Vol. 96, No. 3:

12 84 A. Z. M. Sofi, M. Mamat ad I. Mohd [2] Al-Baali, M. (2000). Extra updates for the BGS method, Optimizatio Meth. ad Soft. 13: [3] Adrei, N. (2008). A ucostraied optimizatio test fuctios collectio, Advace Modelli ad Optimizatio, 10(1): [4] Armijo, L. (1966). Miimizatio of fuctios havi Lipschitz cotiuous first partial derivatives, Pacific J. Math. 16: 1 3. [5] Byrd, H.B., Nocedal, J. & Yua, Y. (1987). Coverece of A Class of Quasi-Newto Method o Covex Problems, SIAM Joural o Numerical Aalysis, Vol. 24, No.5, [6] Che, W.Y. & Li, D.H. (2010). Spectral scali BGS method, Joural of Optimizatio heory Applicatios (146): [7] Curry H. B., (1944). he method of steepest descet for oliear miimizatio problems, Quart. Appl. Math., 2, [8] ord, J.A. & harmliit, S. (2003). New implicit updates i multi-steps quasi-newto methods for ucostraied optimizatio, Joural of Computatioal ad Applied Mathematics, Vol. 152, No. 1: [9] Ibrahim, M.A.H., Mamat, M., Sofi, A.Z.M., Mohd,. I., Ahmad, W.M.A.W. (2010). Alterative Alorithms of Broyde AMILY AMI for Ucostraied Optimizatio: ICMS Iteratioal Coferece O Mathematical Sciece, AIP Coferece Proceedis, Vol : [10] Li, D. & uushima, M. (2001). A modified BGS method ad its lobal coverece i ocovex miimizatio, J. Comput. Appl. Math. (129), [11] Li, D., Qi, L. & Roshchia, V. (2008). A ew class of quasi-newto updati formulas, Optimizatio Methods ad Softwares (23): [12] Lo, J., Hu, X. & Zha, L. (2008). Improved Newto s method with exact lie searches to solve quadratic matrix equatio, Joural of Computatioal ad Applied Mathematics (222): [13] Mamat, M., Dasril, Y., & Mohd, I. (2004). Kaedah quasi-newto utu peoptimuma ta bereaa, Prosidi Simposium Kebasaa Sais Matemati e 12, UIA, Gomba Selaor: [14] Mamat, M., Dasril, Y. & Mohd, I. (2004). Aloritma arah caria aedah BGS-SD dalam peoptimuma, Prosidi Semiar Kebasaa Sais Pemutusa, UUM di Holiday I Resort P.Pia: [15] Mamat, M., Dasril, Y. & Mohd, I. (2006). Aalisis awal Keputusa Beraa Kaedah BGS-SD bai masalah peoptimuma ta bereaa, Prosidi Simposium Kebasaa Sais Matemati e 14, UM di PNB Darby Par: [16] Mascarehas, W.. (2004). he BGS method with exact lie searches fails for o-covex objective fuctios, Math. Pro. Ser. A 99: [17] Nash, S.G. & Sofer, A. (1996). Liear ad oliar prorammi. Geore Maso Uiversity, he McGraw-Hill Compaies, Ic.

13 Improved BGS search directio 85 [18] Nazareth, L. (1979). A relatioship betwee the BGS ad cojuate radiet alorithms ad its implicatios for ew alorithms, Joural of Numerical Aalysis, Vol. 16, No. 5: [19] Pu, D. (1992). A Class of DP Alorithm Without Exact Lie Search, Asia Pacific Joural of Operatio Research 9, [20] Shi, Z.J. (2006). Coverece of quasi-newto method with ew iexact lie search, J. Math. Aal. Appl. (315): [21] Sieel, D. (1993). Modifyi the BGS update by a ew colum scali techique, Joural of Mathematical Prorammi (66): [22] Sofi, A.Z.M., Mamat, M., Mohd,. I. & Dasril, Y. (2008). A alterative hybrid search directio for ucostraied optimizatio, Joural of Iterdiscipliary Mathematics, Vol. 11, No. 5: [23] Wei, Z., Yu, G., Yua, G. & Lia, Z. (2004). he superliear coverece of a modified BGS-type method for ucostraied optimizatio, Computatioal Optimizatio ad Applicatios 29: [24] Wei, Z., Li, G. & Qi, L. (2006). New quasi-newto method for ucostraied optimizatio problems, Joural of Applied Mathematics ad Computatio 175: [25] Wu,. & Su, L. (2006). A quasi-newto based patter search alorithm for ucostraied optimizatio, Applied Mathematics ad Computatio 183: [26] Xiao, Y., Wei, Z. & Wa, Z. (2008). A limited memory BGS-type method for lare-scale ucostraied optimizatio, Joural of Computers ad Mathematics with Applicatios, Vol. 56, No. 4: Received: September, 2012

A PENALTY-CONJUGATE GRADIENT ALGORITHM FOR SOLVING CONSTRAINED OPTIMIZATION PROBLEMS

A PENALTY-CONJUGATE GRADIENT ALGORITHM FOR SOLVING CONSTRAINED OPTIMIZATION PROBLEMS IJRRAS 35 () April 8 wwwarpapresscom/volumes/vol35issue/ijrras_35 pdf A PENALY-CONJUGAE GRADIEN ALGORIHM FOR SOLVING CONSRAINED OPIMIZAION PROBLEMS Hoia Zha Zhuohua Lia & Weli Zhao School of Mathematics