A NON-MONOTONE DIMENSION-REDUCING CONIC METHOD FOR UNCONSTRAINED OPTIMIZATION. G. E. Manoussakis, D. G. Sotiropoulos, T. N. Grapsa, and C. A.

Transcription

1 st Iteratoal Coferece From Scetfc Comutg to Comutatoal Egeerg st IC-SCCE Athes, 8- Setember, 4 IC-SCCE A NON-MONOONE DIMENSION-REDUCING CONIC MEHOD FOR UNCONSRAINED OPIMIZAION G E Maoussaks, D G Sotrooulos, N Grasa, ad C A Botsars Deartmet of Mathematcs, Uversty of Patras, GR-6 Patras, Greece E-mal: {gem, dgs, grasa}@mathuatrasgr, botsars@oteetgr Keywords: Ucostraed Otmzato, Coc model, Dmeso-Reducg Method, Barzla-Borwe ste legth, No-mootoe Abstract I a recet artcle, we troduced a method based o a coc model for ucostraed otmzato he accelerato of the covergece of ths method was obtaed by choosg more arorate ots order to aly the coc model I artcular, we aled the gradet of the objectve fucto a dmeso-reducg method for the umercal soluto of a system of algebrac equatos I ths work, we cororate the revous method the o-mootoe Armjo le search, troduced by Gro, Lamarello ad Lucd, combed wth the Barzla ad Borwe stelegth, order to further accelerate the covergece he ew method does ot guaratee descet the objectve fucto value at each terato Nevertheless, the use of ths o-mootoe le search allows the objectve fucto to crease at some terato wthout affectg the global covergece roertes he ew method has bee mlemeted ad tested well kow test fuctos It coverges + teratos o coc fuctos ad, as umercal results dcate, radly mmzes geeral fuctos INRODUCION We deal wth the geeral ucostraed mmzato roblem: m f ( x ), x where f : s a twce-cotuously dfferetable fucto varables x = ( x, x, K x ) Most methods for ucostraed otmzato are based o a quadratc model Varous authors have troduced o-quadratc algorthms [, 4, 3] Davdo [4] troduced a coc model for ucostraed otmzato he coc model fucto has the form: xqx bx c( x) = + a x + x +, () + where Q s a symmetrc matrx ad s the vector defg the horzo of the coc fucto, e the hyer lae where c( x ) takes a fte value, whch s defed by the equato+ x = Botsars ad Bacooulos [] reseted a coc model of the form + x c( x) = c ( x)( x β ) + c( β ), + β where β s the locato of the mmum As ca be easly show ths form of the coc fucto s equvalet to () he coc model ths form does ot volve ad, therefore, t does ot requre, a estmate of the cojugacy matrx Q or the Hessa matrx of the objectve fucto he coc fucto c( x ) has a uque mmum wheever the symmetrc matrx Q s ostve defte he locato of the mmzer s determed by solvg the equato: β Q β + Q β = β =, + β + Q β rovded that such a soluto exsts, e + Q β I [3] a gradet method has bee roosed, where the search drecto s always the egatve gradet drecto, but the choce of the ste legth s ot the classcal choce of the steeest descet method he motvato for ths choce s that t rovdes two-ot aroxmato to the secat equato uderlyg quas-newto methods [3] hs yelds the terato:

2 GEMaoussaks, DGSotrooulos, N Grasa ad CA Botsars x+ = x tg x, =,,, K where g = g( x) s the gradet of the objectve fucto at x ad the ste t s gve by: ( x+ x) ( x+ x) t+ = ( x+ x) ( g+ g) he choce of the ste legth s related to the egevalues of the Hessa at the mmzer ad ot to the fucto value I [] a o-mootoe le search for Newto-tye methods has bee roosed ad [], [4] some comutatoal advatages of ths techque have bee oted out he method moses that the objectve fucto value f at each terato must satsfy the Armjo s codto wth resect to the maxmum objectve fucto value of a refxed umber M of revous teratos Formally: f x t f x max f x σ t f x ( ) ( ) { } k k k k j k k j M where M s a oegatve teger, ad < σ < he above codto allows a crease the fucto values wthout affectg the global covergece roertes [, 6] hs method has low storage requremets ad exesve comutatos I [7] we mroved the covergece of the coc method reseted [], substtutg the classcal Armjo s le search ad ste wth the above o-mootoe le search ad the Barzla ad Borwe ste resectvely I [] Grasa ad Vrahats roosed a dmeso-reducg method for ucostraed otmzato (the DROP method) hs method cororates the advatages of Newto ad SOR algorthms I artcular, although t uses reducto to smler oe-dmesoal olear equatos, t geerates a sequece whch coverges quadratcally to the comoets of the otmum whle the remag comoet s evaluated searately usg the fal aroxmatos of the others For ths comoet a tal guess s ot ecessary ad t s at the user s dsosal to choose whch wll be the remag comoet, accordg to the roblem Also the DROP method does ot drectly eed ay gradet evaluato I [8] we have used the DR-method to obta more arorate ots to aly the coc model I ths way we have accelerated more the covergece of the coc method I ths aer we use the DR-Method to obta more arorate ots to aly the coc model combato wth the Barzla ad Borwe ste We also cororate the o-mootoe hlosohy, alyg the accetace crtero for each terato wth resect of the maxmum objectve fucto value of M revous teratos HE NEW CONIC MEHOD If we assume that the fucto f to be mmzed s coc, the the followg equato s satsfed: f ( x) ( x) g ( x) + + β ( + β) f ( β) = ( + x) g ( x) x f ( x) () If the horzo s kow, the by calculatg () at + dstct ots x, x, K, x + we have a ( ) ( ) where + + lear system: Aα = s φ, f f = + M f + A G β α = ω ( x ) ( x ) + g + g G = M M ( + x+ ) g+ ()

3 GEMaoussaks, DGSotrooulos, N Grasa ad CA Botsars ( + x ) ( + x ) g x f gx f s = φ = ω = M ( + M β) f ( β) ( + x+ ) g+ x f + + he + dstct ots x, x, K, x + eeded for the formulato of the system (), are foud wth the hel of the DROP-method hs way we obta a better set of ots, so the covergece of the coc algorthm s accelerated k Accordg to the DROP method, to obta a sequece { x }, k =,, K of ots local otmum ot x ( x, x, K, x ) comoet of g (,, K, ) whch coverges to a = of the fucto f, we cosder the sets B, =,, K, to be the coected cotag x o whch g, for =,, K, resectvely, where g x = g x g x g x = f the gradet of the objectve fucto f Alyg the Imlct Fucto heorem [5, 7, ], for each oe of the comoets g, =,, K, we ca fd oe eghborhoods A ad A,, =,, K of the ots w = ( x, x, K, x ) ad x resectvely, such that for ay w= ( x, x, K, x ) A there exst uque mags φ defed ad cotuous A such that: ad x ( w) = ϕ A, =, K,, ( ) g w, ϕ w =, =, K, ϕ Moreover, the artal dervatves ϕ =, j =, K, exst cotuous j x j A for each A ad they are gve by: jg( w; ϕ( w) ) jϕ( w) =, g w; ϕ w =, K,, j =, K, ( ) Next, we utlze aylor's formula to exad the ( w) ϕ, =, K,, about we ca obta the followg teratve scheme for the comutato of the comoets of + w = w + A V, =,, K where: w = x, =, K, wth, x ϕ( w ) A g w ; x g w ; x,, j j =,, g w ; x g w ; x,, [ ] ϕ, =, K,, they are w By straghtforward calculatos, x :,, j =, K, (3) V = v = x x, =, K, (4) = After a desred umber of teratos, say = m, the th comoet of x ca be aroxmated by meas of the followg relato: m m,, jg( w ; x m+ m m m ) + x = x ( xj xj ) m m, (5) j= g( w ; x )

4 GEMaoussaks, DGSotrooulos, N Grasa ad CA Botsars Remark Relatve rocedures for obtag x, K, x, for examle t x ca be costructed by relacg x wth ay oe of the comoets x, ad takg w ( x x x x ) =, K,,, K, t t +, Remark he above descrbed method does ot requre the exressos ϕ but oly the values x whch are g x,, x, = w = x, K, x gve by the soluto of the oe-dmesoal equatos ( K ) So, by holdg fxed, we ca solve the equatos g ( w ; r ) =, =, K, for r the terval ( ab, ) wth accuracy δ Of course we ca use ay other oe-dmesoal method to solve the above equatos Here we emloy a modfed bsecto method [, 8, 3] he oly comutable formato requred by ths bsecto method s the algebrac sgs of the fucto g Moreover, t s the oly method that ca be aled to roblems wth mrecse fucto values After the system () s formulated, usg the DR ots, we roceed wth the coc method So, let + x+ f + k ρ = = + x g x where f = f+ f, x = x+ x, ad f g+ xg x rovded that the quatty uder the square root s o-egatve If ths quatty s egatve the the coc method caot roceed I ths case, usg DR Method, we get a ew ot x, evaluate a ew equato () ad we restart the coc rocedure to solve the modfed system () he gauge vector ca be determed by solvg the lear system: Z= r (6) + where z = x+ ρx, Z z z, η = ρ, M z = r η η M η = From (), (6), we have that the locato of the mmum β ca be determed through the equato: = Z r, α = A ( s φ ) We carry out the ecessary versos recursvely as ew ots are costructed by the algorthm Usg Householder's formula for matrx verso t ca be verfed that: Z el( z Z e l ) Z = Z (7) zz el ad Z el( η z ) = + (8) zz el rovded that zz el s bouded away from zero We ote that e l s a vector wth zero elemets excet the osto l =, where t has uty he soluto of the lear system s roved to be: + qu α = u v, (9) + qv where q =, u = G s ad v= G φ Let us further defe:

5 GEMaoussaks, DGSotrooulos, N Grasa ad CA Botsars L L + x + x + Λ =, λ M M O M = ad y + = g+ () + x λ L λ he G + ca be comuted accordg to the recursve formula: G ej( y G e Λ j ) + G+ = G () λ y+ G e j rovded that y+ G ec s bouded away from zero he recursve equatos for the vectors u ad v, requred to comute α + from (9), are foud to be: G ej( θ+ y u) + Λ G ej( ξ ) + y v + u+ =Λ u ad v + = v () y+ G e j λ y ι + G e j where + x+ θ+ = g+ x+ s ad ξ + = f + (3) λ he roosed method s llustrated the followg algorthms seudo-code where x dcates the startg ot, a = ( a, a, K, a ), b = ( b, b, K, b ) dcate the edots each coordate drecto whch are used for the above metoed oe-dmesoal bsecto method, wth redetermed accuracy δ, MI the maxmum umber of teratos requred ad ε, ε, ε3, ε 4 the redetermed desred accuraces Procedure DROP (Dmeso-Reducg Otmzato) [] Ste Iut x ; a ; b ; δ ; MI; ε ; ε Ste Set = Ste 3 If < MI relace by + ad go to ext ste; otherwse, go to Ste 4 g x < ε go to Ste 4 Ste 4 If Ste 5 Fd a coordate t such that the followg relato holds: sg g( x, K, xt, at, xt+, K, x ) sg g( x, K, xt, bt, xt+, K, x ) =, for all =,, K, If ths s mossble, aly Armjo's method ad go to Ste 4 Ste 6 Comute the aroxmate solutos r, =,, K, of the equato sg g x,, x, r, x,, x a, b wth ( t t ) K K by alyg the modfed bsecto + =, xt = r,, t, t+,, accuracy δ Set Ste 7 Set y ( x x x x ) = K K Ste 8 Set the elemets of the matrx A of Relato (3) usg x t stead of x Ste 9 Set the elemets of the vector V Relato (4) usg x t stead of x Ste Solve the ( ) ( ) lear system As = V for s Ste Set y + = y + s Ste Comute x t by vrtue of Relato (5) ad set x = ( y ; xt ) Ste 3 If s ε go to Ste 4; otherwse retur to Ste 3 Ste 4 Outut { x } t t

6 GEMaoussaks, DGSotrooulos, N Grasa ad CA Botsars he crtero Ste 5 esures the exstece of the soluto r whch wll be comuted at Ste 6 If ths crtero s ot satsfed we aly Armjo's method [,, 8] for a few stes ad the try aga wth DR method Our exerece s that may examles studed varous dmesos as well as for all the roblems studed ths aer (see below Numercal Alcatos) the alcato of such a subrocedure s ot ecessary We have merged t our algorthm for comleteess Ma Algorthm: No Mootoe Dmeso Reducg Coc Method Ste Assume x s gve Set = ; alha = ; k = ; j = ; W ( j) = f Ste Set d = g Ste 3 Use DR-Method to get a ot x Ste 4 Set α = x = u, G = Z = I, v = =, λ = jc= lc= Ste 5 If g ε( + f ) the sto; else go to Ste 6 Ste 6 Use () to calculate L +, y + Ste 7 If y+ G ec < ε3, the set x = x + ad go to Ste 3; else go to Ste 8 Ste 8 Use (9), (), (), (3) to calculate a + Ste 9 If ( x β) g < ε 4 the set x = x + ad go to Ste 3; else go to Ste Ste If f ( β + ) fmax, the store the fucto value the record, set x+ = β+ ad go to ste ; else go to ste 3 Ste Set = + ; If jc= + the reset jc =, else jc= jc+ Ste Set d = µ ( x β) where µ = sg{ g ( x β) } Ste 3 If d + γ M, the determe the Barzla stet, set x+ = x + td ad go to ste 4; else set x = x ad go to Ste 3 δ f+ f g+ x+ x g x+ x = <, the usg DR-Method fd a x +, ad reeat ths rocedure utl the ew x + so obtaed satsfesδ > ; go to Ste 5 Ste 4 If Ste 5 If zz el < ε3, the set x = x + ad go to Ste 3; else go to Ste 6 Ste 6 Use (7), (8) to calculate Z +, + Ste 7 If lc =, the reset lc = ; else set lc = lc + Ste 8 Go to Ste 5 3 NUMERICAL APPLICAIONS he ew method has bee mlemeted usg a ew FORRAN9 rogram amed SCONIC SCONIC has bee tested o a Petum IV PC comatble wth radom roblems of varous dmesos Our exerece s that the algorthm behaves redctably ad relably ad the results have bee qute satsfactory Next we comare the umercal results obtaed, for varous startg ots, by alyg algorthms (Armjo s quadratc method [], Fletcher-Reeves [6], Polak-Rbere [3]), cludg the classc coc method, wth the corresodg results of our method For the followg roblems, the reorted arameters are: - x = ( x, x, K, x ) : the startg ot, - x = ( x, x, K, x ) : aroxmate local mmum, - I: the total umber of teratos requred obtag x, - FE: the total umber of fucto ad gradet evaluatos, - AS: the total umber of algebrac sgs eeded by DROP

7 GEMaoussaks, DGSotrooulos, N Grasa ad CA Botsars he dex α dcates the classcal startg ot ad D dcates dvergece or o covergece after teratos he aroxmate local otmum x, as log as all the fucto values were comuted wth a accuracy 5 of ε = We set the sze of the le search record to be M = Examle : Exteded Rosebrock Fucto [9] = ( ) + ( ) f x x x x f x = at x = (,) wth Armjo FR PR CONIC SCONIC x I FE I FE I FE I FE I FE AS (-, ) (-3, 6) (-, ) D 5 6 (-3, 3) (, ) D D 6 (, ) D D 5 6 (, ) D D D (-, ) D D 5 6 (-, -) (, -) (3, 3) D 37 8 (, -) Examle : Freudeste ad Roth Fucto [9] x = ( 5, 4) ad f ( x ) = at x = ( 4 K, 8968K ) f x = + x + x x x + + x + x + x x f x = at wth Armjo FR PR CONIC SCONIC x I FE I FE I FE I FE I FE AS (5, -) (5, ) D D (-, -) (-, ) (45, 45) (, ) (, ) (4, -) D D Examle 3: Brow badly scaled Fucto [9] = ( ) + ( ) + ( ) wth f ( x ) = at x = (, ) f x x x x x Armjo FR PR CONIC SCONIC x I FE I FE I FE I FE I FE AS (, ) D D D (-, ) D D D (, ) D D D (, ) D D D (-, ) D D D (, ) D D D

8 GEMaoussaks, DGSotrooulos, N Grasa ad CA Botsars REFERENCES [] Armjo, L (966), Mmzato of fuctos havg Lschtz cotuous frst artal dervatves, Pacfc J Math, 6, 3 [] Bacooulos A ad Botsars, CA (99), A ew coc method for ucostraed mmzato, J Math Aal Al, 67, 3 [3] Barzla, J ad Borwe, JM (988), wo-ot ste sze gradet methods, IMA J Num Aal, 8, 4 48 [4] Davdo, WC (959), Varable metrc methods for mmzato, A E C, Research ad Develomet Reort, No ANL 599, Argoe Nat l Lab, Argoe, Illos [5] J Deudoe (969), Foudatos of moder aalyss, Academc Press, New York [6] Fletcher, R ad Reeves, C (964), Fucto mmzato by cojugate gradets, Comut J, 7, [7] N Grasa ad MN Vrahats (989), he mlct fucto theorem for solvg systems of olear equatos R, Iter J Comuter Math [8] N Grasa ad MN Vrahats (99), A dmeso reducg method for solvg systems of olear equatos R, Iter J Comuter Math [9] N Grasa, MN Vrahats ad C Bouts (99), Solvg systems of olear equatos R usg a rotatg hyerlae R +, Iter J Comuter Math [] N Grasa ad MN Vrahats (995), A ew dmeso reducg method for solvg systems of olear equatos, Iter J Comuter Math [] N Grasa ad MN Vrahats (996), A dmeso reducg method for ucostraed otmzato, Joural ofcomutatoal ad Aled Mathematcs [] Gro, L, Lamarello, F ad Lycd, S (986), A omootoe le search techque for Newto s method, SIAM J Numer Aal, 3, [3] Jacobso, DH ad Oksma, W,(97), A algorthm that mmzes homogeeous fuctos of varables + teratos ad radly mmzes geeral fuctos, J Math Aal Al 38, [4] B Kearfott (979), A effcet degree comutato method for a geeralzed method of bsecto, Numer Math 3, 9 7 [5] R B Kearfott (987), Some tests of geeralzed bsecto, ACM ras Math Software 3, 97 [6] M Kuferschmd ad JG Ecker (987), A ote o soluto of olear rogrammg roblems wth mrecse fucto ad gradet values, Math Program Study [7] Maoussaks, GE, Sotrooulos, DG, Botsars, CA, ad Grasa, N (), A No Mootoe Coc Method for Ucostraed Otmzato, I: Proceedgs of 4th GRACM, Cogress o Comutatoal Mechacs, 7-9 Jue, Uversty of Patras, Greece [8] Maoussaks, G E, Grasa, N, Bosars, C A (3), A Dmeso - Reducg Coc Method for Ucostraed Otmzato, I Proceedgs of HERCMA 3, Setember 3, Athes [9] More, BJ, Garbow, BS ad Hllstrom, KE (98), estg ucostraed otmzato, ACM ras Math Software, 7, 7 4 [] JM Ortega ad WC Rhebolt (97), Iteratve Soluto of Nolear Equatos Several Varables, Academc Press, New York [] A Ostrowsk (973), Soluto of equatos Eucldea ad Baach saces, hrd Edto, Academc Press, Lodo [] Plagaakos, VP, Sotrooulos, DG ad Vrahats, MN (998), A Nomootoe Backroagato rag Method for Neural Networks, Det of Mathematcs, Uv of Patras, echcal Reort No98-4 [3] Polak, E (97), Comutatoal Methods Otmzato: A Ufed Aroach, Academc Press, New York [4] Rayda, M (997), he Barzla ad Borwe gradet method for the large scale ucostraed mmzato roblem, SIAM J Otm, 7, 6 33 [5] W C Rheboldt (974), Methods for solvg systems of equatos, SIAM, Phladelha [6] J F raub (964), Iteratve methods for the soluto of equatos, Pretce Hall, Ic, Eglewood Clffs, NJ [7] M N Vrahats (988), CHABIS: A mathematcal software ackage for locatg ad evaluatg roots of systems of olear equatos, ACM ras Math Software 4, [8] MN Vrahats, GS Adroulaks ad GE Maoussaks (996), A ew ucostraed otmzato method for mrecse fucto ad gradet values, J Math Aal Al, 97, [9] M N Vrahats ad K I Iordads (986), A rad geeralzed method of bsecto for solvg systems of o lear equatos, Numer Math 49, 3 38