A Derivative-Free Algorithm for Bound Constrained Optimization

Computatona Optmzaton and Appcatons, 21, 119 142, 2002 c 2002 Kuwer Academc Pubshers. Manufactured n The Netherands. A Dervatve-Free Agorthm for Bound Constraned Optmzaton STEFANO LUCIDI ucd@ds.unroma.t Dpartmento d Informatca e Sstemstca, Unverstà d Roma La Sapenza, Roma, Itay MARCO SCIANDRONE Isttuto d Anas de Sstem ed Informatca, CNR, Roma, Itay scandro@as.rm.cnr.t Revsed 15 January, 2001 Abstract. In ths wor, we propose a new gobay convergent dervatve-free agorthm for the mnmzaton of a contnuousy dfferentabe functon n the case that some of (or a) the varabes are bounded. Ths agorthm nvestgates the oca behavour of the objectve functon on the feasbe set by sampng t aong the coordnate drectons. Whenever a sutabe descent feasbe coordnate drecton s detected a new pont s produced by performng a nesearch aong ths drecton. The nformaton progressvey obtaned durng the terates of the agorthm can be used to bud an approxmaton mode of the objectve functon. The mnmum of such a mode s accepted f t produces an mprovement of the objectve functon vaue. We aso derve a bound for the mt accuracy of the agorthm n the mnmzaton of nosy functons. Fnay, we report the resuts of a premnary numerca experence. Keywords: dervatve-free agorthm, bound constrants, nesearch technque 1. Introducton Practca appcatons very often ead to the mnmzaton of a smooth functon whose varabes are subject to bound constrants. In many of these cases the objectve functon vaue s obtaned by drect measurements or t s the resut of a compex system of cacuatons, such as a smuaton. Therefore, even f t s nown that the objectve functon s smooth, ts anaytca expresson s not avaabe and the computaton of ts vaues may be expensve and/or affected by the presence of nose. Hence, the frst order dervatves cannot be expcty cacuated or approxmated. Ths motvates the ncreasng nterest n studyng new dervatve-free methods for bound constraned optmzaton. Such methods shoud present strong goba convergence propertes, shoud be abe to mae sgnfcant progresses wth few functon evauatons, and shoud be suffcenty robust n the nosy case. Wth ths n mnd, n ths paper we consder the probem mnmze f (x) (1) subject to x u, where x,, u R n, wth < u, and f : R n R s a contnuousy dfferentabe functon, but ts frst order dervatves cannot be expcty cacuated or approxmated. We aow the possbty that some of the varabes are unbounded by permttng both = and

120 LUCIDI AND SCIANDRONE u = for some {1,...,n}. We denote the feasbe set by F ={x R n : x u}. We defne a statonary pont of probem (1) a feasbe pont x that satsfes the foowng frst-order necessary optmaty condton: f ( x) T (y x) 0 for a y F. (2) We reca that at any statonary pont x of f (x) ( x F), we have that red f ( x) = 0 where the reduced gradent red f (x) s defned as foows max( f (x), 0) f x = u red f (x) = mn( f (x), 0) f x = f (x) otherwse (3) In order to overcome the ac of gradent nformaton, many gobay convergent dervatve-free agorthms proposed n terature are based on the dea of performng fner and fner sampngs of the objectve functon aong sutabe sets of search drectons (see, for nstance, [2, 11, 14 16] and the references quoted there). In [8] t has been performed a genera anayss of the requrements on the search drectons and the sampng technques, whch ensure the goba convergence of a dervatve-free agorthm for unconstraned mnmzaton probems. Roughy speang, at every non-statonary pont, the set of search drectons must contan a descent drecton, and the sampng technque must produce a sutabe pont aong such a drecton. As ceary descrbed n [7], the presence of bound constrants mposes stronger restrctons on the choce of search drectons. In partcuar, at every non-statonary pont, the set of search drectons must contan a descent drecton whch s aso feasbe, n the sense that (suffcenty) sma stepszes aong such drecton must produce feasbe ponts where the objectve functon s reduced. In [7] t s shown that the set of coordnate drectons satsfes ths property. Gobay convergent agorthms usng these drectons have been proposed n [7] and [3]. More n partcuar, the agorthm proposed n [7] foows a pattern search strategy by evauatng the objectve functon on specfed geometrc patterns. Whe the method ntroduced n [3] s based on the dea of usng approxmatng modes of the objectve functon whch are but by usng sutabe grd ponts. In ths paper, we propose a new agorthm mode for sovng probem (1). In order to try to guarantee both goba convergence propertes and a good effcency of the proposed agorthm, we have drawn our nspraton from the strategy underyng the gradent based methods. The goba convergence and the good computatona behavour of these methods foow from the fact that they are abe () to fnd a good feasbe descent drecton (namey a feasbe drecton aong whch the objectve functon suffcenty decreases); () to perform a suffcenty arge stepength aong ths drecton;

BOUND CONSTRAINED OPTIMIZATION 121 () to expot the nformaton on the objectve functon obtaned durng the terates of the agorthm. Here, startng from the approaches proposed n [3, 7] and [8], we have defned an agorthm whch tres to foow the ponts () () wthout usng any nformaton on the frst order dervatves and tang account the partcuar structure of the feasbe set. The man features of the agorthm are the foowng: a good feasbe descent drecton s determned by nvestgatng the oca behavour of the objectve functon on the feasbe set aong the coordnate drectons; whenever a sutabe descent feasbe coordnate drecton s detected, a new pont s produced by performng a dervatve free nesearch aong ths drecton; the nformaton progressvey obtaned durng the terates of the agorthm can be used to bud an approxmaton mode of the objectve functon n order to mprove the oca behavour of the agorthm. As regards the theoretca propertes of the proposed agorthm, we prove that every mt pont of the sequence produced s a statonary pont for probem (1). Smary to [1, 3, 6, 17], we consder aso the mnmzaton of nosy functons whch are perturbatons of smooth functons. For ths case, by requrng the standard assumpton that the gradent of the objectve functon s Lpschtz contnuous (whch s not requred n the noseess case), we derve a bound for the mt accuracy of the agorthm. The paper s organzed as foows. In Secton 2 we descrbe the proposed agorthm mode. In Secton 3 we prove the goba convergence of the agorthm. In Secton 4, we characterze the behavour of the agorthm n the mnmzaton of nosy functons. Fnay, n Secton 5 we report the resuts of a premnary numerca experence performed on both standard test probems and a rea appcaton. Notaton. The j-th component of a vector v R n s ndcated by v j. We denote the Eucdean norm (on the approprate space) by. A subsequence of {x } correspondng to an nfnte subset K w be denoted by {x } K. We ndcate by e, wth = 1,...,n, the orthonorma set of the coordnate drectons and by F the boundary of the feasbe set F. Fnay, gven a rea number a, a represents the argest nteger that s not greater than a. 2. A new agorthm mode In ths secton we propose a new cass of dervatve-free agorthms for the mnmzaton of a contnuousy dfferentabe functon n the case that some of (or a) the varabes are bounded. As sad before, the approach s based on the dea of performng sutabe sampngs of the objectve functon aong the coordnate drectons. As ponted out n [7], the coordnate drectons aow us to cope wth the presence of box constrants. Ths can be easy derved from the optmaty condtons (2). In fact, f a feasbe pont x s not a statonary pont of f, then there must exst a feasbe pont y and an nteger h {1,...,n} such that h f ( x) T (y x) h < 0. If ᾱ = (y x) h > 0, then, tang nto account that F s defned by box constrants, we have ᾱ f ( x) T e h < 0, x ᾱe h F

122 LUCIDI AND SCIANDRONE The contnuty of f and the convexty of the feasbe set F mpy that there exsts a postve vaue such that: f ( x αe h )< f ( x), x αe h F, for a α (0, ). The case ᾱ = (y x) h < 0 eads to the same concusons wth e h repaced by e h. Hence, n correspondence to any feasbe pont x whch s not a statonary pont, there s a coordnate drecton aong whch (or aong ts opposte) there must exst feasbe ponts where the functon s strcty decreased (ths property s not necessary true for dfferent sets of n neary ndependent drectons). Therefore, performng fnder and fner sampngs of the objectve functon aong the coordnate drectons and ther opposte, t s possbe ether to understand that a pont s a good approxmaton of a statonary pont of f,orto determne a specfc drecton aong whch the objectve functon decreases. On ths bass, we propose an agorthm mode whch sampes the objectve functon aong the coordnate drectons, wth the am of detectng a feasbe drecton where the objectve functon s suffcenty decreased. Once such a drecton has been ndvduated, a dervatvefree nesearch technque s adopted for performng a suffcenty arge step aong t, so as to expot the descent property of the search drecton as much as possbe. Perodcay, t s admtted the possbty of generatng a pont by a movement of arbtrary ength aong an arbtrary drecton. The use of the coordnate drectons as search drectons and the partcuar sampng technque adopted aow us to overcome the ac of gradent nformaton and to ensure that every mt pont of the sequence produced s a statonary pont for probem (1). Formay, the agorthm mode s descrbed as foows. Agorthm Mode Data. x 0 F, (0, 1), γ > 0, 0 < 0 <, d = e for = 1,...,n. Step 0. Set = 0, = 1, h = 1. Step 1. Compute α max s.t. x α max d F and set α = mn{,α max}. If α>0and f (x αd ) f (x ) γ(α) 2 go to Step 3. Step 2. Compute α max s.t. x α max d F and set α = mn{,α max}. If α>0and f (x αd ) f (x ) γ(α) 2 then set d = d and go to Step 3. ese set α = 0, 1 = α, and go to Step 4. Step 3. Compute α by the Expanson Step(d,α,α max,γ)and set 1 = α. Step 4. Set x 1 = x α d, j 1 = j, for j {1,...,n} and j. Step 5. If h n then fnd x 1 such that f (x 1 ) f ( x 1 ) and x 1 F, (4) ese set x 1 = x 1. If x 1 x 1 then set h 1 = 1 ese set h 1 = h 1. Set = mod(, n) 1, = 1 and go to Step 1.

BOUND CONSTRAINED OPTIMIZATION 123 Expanson Step (d,α,α max,γ). Data. δ (0, 1). Step 1. Let = mn{α max, α δ }. If α = α max or f (x d )> f (x ) γ 2 set α = α and stop. Step 2. Set α = and go to Step 1. More n partcuar, the steps of the agorthm can be summarzed as foows. At Step 1 the drecton d s examned wth the am of determnng (f possbe) a feasbe pont where the objectve functon s suffcenty decreased. Frst, t s computed the maxmum feasbe stepength α max whch can be performed aong the drecton d startng from the pont x. Then, the tra stepsze α s determned by choosng the mnmum between α max and. The scaar has been computed on the bass of the behavour of the objectve functon aong the same drecton showed at the prevous teratons. Therefore, the scaar shoud tae nto account the senstvty of the objectve functon wth respect to the -th varabe, and hence t shoud provde a promsng nta stepsze for the drecton d. Fnay, t s verfed f the movng of ength α aong d produces a feasbe pont where the functon s suffcenty reduced. If such a pont s produced then a nesearch technque s performed aong d to provde a sutabe stepsze α (Step 3). Otherwse, the drecton d s consdered (Step 2). Step 2 s smar to Step 1, wth d repaced by d. In ths case, f the tra pont x αd does not produce a suffcent decrease of f then the stepsze α s set equa to zero and the scaar s reduced. In ths way, when the drectons d and d w be consdered agan by the agorthm, the nta stepsze w be chosen n an nterva contanng smaer vaues. At Step 3 a sutabe arge stepsze α s computed by a dervatve-free nesearch technque. Such technque derves from the ones proposed n [4]. It computes a suffcenty good estmate of the mnmum of f aong d wthout requrng any nformaton on the sope of the objectve functon. The am of ths step s to expot the good descent drecton d dentfed at Step 1 or Step 2. Then, the scaar 1 s set equa to α. The motvaton of ths choce derves from the fact that the stepsze α produced by a nesearch technque shoud dentfy promsng vaues for the nta stepsze when the drecton d (or d ) w be nvestgated. At Step 4 the canddate pont x 1 s generated. At Step 5 the new pont x 1 s produced and, for the next teraton, a coordnate drecton s seected by foowng the cycc order. At each teraton x 1 can be aways set equa to the canddate pont x 1 produced at Step 5. The ndex h counts the number of successve teratons characterzed by the fact that x 1 = x 1. The condton h n ndcates that the ast n ponts generated have been produced by nvestgatng the behavour of f aong a the coordnate drectons and that, roughy speang, the agorthm has obtaned enough nformaton about the oca behavour of the objectve functon. In ths case, the agorthm admts the possbty of acceptng as x 1 any feasbe pont whch produces a reducton of the objectve functon. Therefore, n these teratons the pont x 1 can be generated, for nstance, by mnmzng any approxmaton mode of the objectve functon but by usng the nformaton progressvey obtaned by the agorthm. Ths

124 LUCIDI AND SCIANDRONE possbty does not affect the convergence propertes of the agorthm, but t can ncrease ts effcency. 3. Convergence anayss In ths secton we study the theoretca propertes of Agorthm Mode. In partcuar, we show that any accumuaton pont of the sequence generated by the proposed agorthm s a statonary pont of probem (1). Frst, we state the foowng proposton. Proposton 1. Suppose that f s bounded beow on the feasbe set F and et {x } be the sequence produced by Agorthm Mode. Then: () Agorthm Mode s we defned; () every mt pont of {x } beongs to F; () we have m = 0 (5) m = 0 for = 1,...,n. (6) Proof: In order to prove that Agorthm Mode s we defned, we have to ensure that the Expanson Step, when performed aong a drecton d, wth {1,...,n}, termnates n a fnte number j of steps. At ths am, by contradcton we assume that for a gven d and x δ j αd F for a j, f (x δ j αd )< f (x ) γ(δ j α) 2 d 2 for a j, whch voates the assumpton that f s bounded beow on F. As regards asserton (), we have that the nstructons of Agorthm Mode mpy that x F for a. Snce F s a cosed set, the asserton s proved. In order to prove (5), we spt the teraton sequence {} nto two parts, K and K.We dentfy wth K those teratons where α = 0 (7) and wth K those teratons where α 0 s produced by Expanson Step. Then, Steps 3 and 5 mpy f (x 1 ) f (x α d ) f (x ) γ(α ) 2 d 2. (8) Tang nto account the boundedness assumpton on f, t foows from (8) that { f (x )} tends to a mt f.ifk s an nfnte subset, recang that d =1 we obtan m, K α = 0. (9) Therefore, (7) and (9) mpy (5).

BOUND CONSTRAINED OPTIMIZATION 125 In order to prove (6), for each {1,...,n} we spt the teraton sequence {} nto three parts, K 1, K 2 and K 3. We dentfy wth K 1 those teratons where Expanson Step has been performed usng the drecton d, namey f (x α d ) f (x ) γ(α ) 2 d 2 (10) 1 = α. (11) We denote by K 2 those teratons where we have faed n decreasng the objectve functon aong the drectons d and d. By the nstructons of the agorthm t foows that for a K 2 1, (12) where (0, 1). Fnay, K 3 denotes the teratons where the drectons d and d are not used as search drectons. Then, for K 3 we have 1 =. (13) If K 1 s an nfnte subset, from (11) and (5) we get that m 1 = 0. (14), K 1 Now, et us assume that K 2 s an nfnte subset. For each K 2, et m (we omt the dependence from ) be the bggest ndex such that m < and m K 1. Then we have: 1 (1 m ) n m m (15) (we can assume m = 0 f the ndex m does not exst, that s, K 1 s empty). As and K 2, ether K 1 s an nfnte subset mpyng m,ork 1 s fnte mpyng ( 1 m ). Hence, f K 2 s an nfnte subset, (15) together wth (14), or the fact that (0, 1), yeds m 1 = 0. (16), K 2 Fnay, et us consder the nfnte subset K 3 (note that the nstructons of the agorthm mpy that K 3 s aways an nfnte set). The nstructons of the agorthm mpy that, for a K 3 and suffcenty arge, there exsts a nonnegatve ndex ν n such that ν K 1 K 2 1 = ν 1. Therefore, from (14) and (16), we get that m 1 = 0, (17), K 3 so that (6) s proved, and ths concudes the proof.

126 LUCIDI AND SCIANDRONE Now we are ready to prove the man convergence resut. Proposton 2. Suppose that f s bounded beow on the feasbe set F and et {x } be the sequence produced by Agorthm Mode. Then every mt pont of {x } s a statonary pont for probem (1). Proof: Let x be any mt pont of {x }, that s m x = x, (18), K where K {0, 1,...}. By () of Proposton 1 we have that x F. In order to prove the thess, et us suppose by contradcton that x s not a statonary pont. Therefore, there exsts a pont ȳ F such that f ( x) T (ȳ x) <0. (19) Ths mpes that h f ( x)(ȳ x) h < 0, (20) for some h {1,...,n}. Let us defne the sequence of scaars β = (ȳ x ) h. Now, by (20) we have where m, K β f (x ) T e h = β f ( x) T e h < 0, (21) β = (ȳ x) h > 0. (22) For each K, et us consder the smaest ndex m and the bggest ndex m such that n the teratons m and m the drecton e h or/and the drecton e h are nvestgated. The nstructons mpy m n m n. (23) Moreover we have that at east one of the foowng occurrence must happen () a the ponts x j, wth 0 j (m ), are produced by Step 4, that s x j1 = x j α j e ( j) (or x j1 = x j α j e ( j) ); () a the ponts x j, wth 0 j m, are produced by Step 4, that s x j1 = x j α j e ( j) (or x j1 = x j α j e ( j) ),

BOUND CONSTRAINED OPTIMIZATION 127 where the ndex ( j) {1,...,n} dentfes the search drecton nvestgated. Now et K K and K K be the subsets such that condton () s verfed for a K and condton () s verfed for a K. Suppose frst that K s nfnte. For sempcty we rename K as K. Hence, () of Proposton 1 mpes that, for 1 j m, t foows m, K x j x j 1 =0, (24) from whch, recang (23), we get m, K x m x = x. (25) By (21), (22) and (25) we obtan m, K β m = β. (26) Condtons (21), (25) and (26) mpy that for K and suffcenty arge the drecton sgn(β m )e h s an ascent drecton n x m because t maes an acute ange wth the gradent, and moreover, recang (6) we can prove that for K and suffcenty arge f ( x m sgn ( β m ) h m 1 In fact, by appyng the Mean Vaue Theorem we have f ( x m sgn ( β m = sgn ( β m ) h m 1 ) e h > f ( ) x m. (27) ) e h f ( ) x m ) h m 1 [ f ( x) T e h ( f (ξ ) T e h f ( x) T e h )], ) h m where ξ = x m λ sgn(β m 1 e h, wth λ (0, 1). Then, snce ξ x m 0, recang (21), (26) and the contnuty of f, t foows that (27) hods. Hence, for suffcenty arge, at the m -th teraton, the nstructons of the agorthm (n partcuar, Step 2) mpy that the drecton sgn(β m )e h s nvestgated. Snce x F and h ȳ h u h t foows x m tsgn ( ) β m eh = x m t β m (ȳ xm ) heh F t [ 0, β m ]. (28) Now, by () of Proposton 1 we have that α 0 and h m for K and suffcenty arge, (22), (26) and (28) yed 0 for, and hence, x m sgn ( β m ) h m 1 e h F (29)

128 LUCIDI AND SCIANDRONE and x m sgn ( )α h m β m δ e h F (30) Therefore we have that ether or α m = 0,(29) hods and ( f x m sgn ( β m ) h m 1 ) e h > f ( ( ) h ) 2 m x m γ. (31) α m 0,(30) hods and ( f x m sgn ( ) )α m β m δ e h > f ( x m By appyng the Mean Vaue Theorem n (31) and (32) we can wrte ether ( ) ) αm 2 γ. (32) δ or sgn ( β m ) ( ) T h m f um eh 1 where sgn ( β m u m v m = x m = x m ) ( ) T f vm eh > α m δ sgn ( β m sgn ( β m ) h λ 1 m 1 e h, ) α λ 2 m δ e h, wth λ 1,λ2 (0, 1). Therefore, tang the mts for and K, by usng (5), (6) and the contnuty assumpton on f, t foows sgn( β) f ( x) T e h 0, whch contradcts (21). Now, f K s an nfnte set then we can repeat the same reasonngs by mnor modfcatons and obtan a contradcton wth (21). Remar 1. If we repace at Step 5 of Agorthm Mode condton (4) wth the foowng stronger condton f (x 1 ) f ( x 1 ) and x 1 F (33) x 1 x max{ f,ρα } (34)

BOUND CONSTRAINED OPTIMIZATION 129 where f = f (x 1 ) f (x ) and ρ (0, 1), we obtan that the resuts of Proposton 1 and Proposton 2 st hod wth the addtona property of the sequence {x }: m x 1 x =0. (35) In fact, snce f s bounded beow on F and the sequence { f (x )} s decreasng, we have that f 0. On the other hand, by (5) of Proposton 1 we have that α 0. Therefore, (35) foows mmedatey. 4. Convergence anayss n presence of nose In ths secton, we consder the case where the vaues of the objectve functon of probem (1) are corrupted by the presence of nose. In other words, we can ony observe the perturbaton f (x) gven by f (x) = f (x) (x), where (x) represents the amount of nose. Furthermore, we assume that there exsts a constant >0 such that (x) for every x F. (36) In Agorthm Mode the vaue f (x) has to be repaced by f (x). Frst of a, we remar that t s possbe to show that Proposton 1 st hod aso n presence of nose satsfyng (36). Further resuts about the propertes of the proposed agorthm can be stated under the assumpton that the gradent of the objectve functon s Lpschtz contnuous. In partcuar, t s possbe to derve a bound on the norm of the reduced gradent. At ths am, for each teraton we consder a ba B of radus r = 2 mn{,δ} n (37) =1 about x (where and δ are user chosen parameters of the agorthm). For each teraton such that h n we have that every pont where the functon has been evauated n one of the precedng n teratons s contaned n the ba B about x. In fact, as regards the generated ponts we have x x n n α 1 =1 n. =1 Then, n correspondence to each teraton j, the agorthm can sampe the objectve functon aong a gven coordnate drecton e h n the nterva [ ] x j h j1 mn{,δ} e h, x j h j1 mn{,δ} e h. Then, we have the foowng proposton.

130 LUCIDI AND SCIANDRONE Proposton 3. Suppose that f s bounded beow on the feasbe set F and that ts gradent s Lpschtz contnuous (wth constant L) on F. Let {x } be the sequence produced by Agorthm Mode and et = sup x B (x). Then, there exst two constants c 1, c 2 > 0 such that, for each teraton for whch h n, we have red f (x ) c 1 max =1,...,n { } c2 mn =1,...,n { }. Proof: Let us consder the -th component of red f (x ). Now et () be the bggest ndex such that at the () teraton the drecton e (and/or e ) has been nvestgated. Frst of a, we note that by the nstructons of the agorthm, and recang the assumpton h n, t foows that () n, and by the defntons of B and we have that the objectve functon vaues computed n the teratons (), () 1,..., are corrupted by an amount of nose bounded by. Now we dstngush the foowng cases: () x = u () x = () < x < u Case (). In ths case we have ether (a) x () = u or (b) x () < u. In case (a), as x = x (), from the nstructons of the agorthm we have α () = 0, ()1 =, and ( ) f x () e > f ( ( ) ) 2 x () γ 2. By appyng the Mean Vaue Theorem we obtan f ( ( ) T ) 2 u () e > γ 2, where u () = x () λ () e wth λ () (0, 1). Then, we can wrte [ ( ) f u() f (x ) f (x ) ] T e <γ 2,

BOUND CONSTRAINED OPTIMIZATION 131 from whch, tang nto account the Lpschtz assumpton on f, t foows f (x ) T e <γ 2 L x u () γ 2 L x x () L. (38) We have () 1 x = x () α () j d () j, j=0 where d () j {e 1, e 1,...,e n, e n }. For each j such that α () j 0, recang the nstructons of the agorthm, we have that there exsts an ndex {1,...,n} such that () j1 = α () j, and = () j1. Therefore, t foows { } α () j max, =1,...,n and we can wrte { } x x () n max. =1,...,n From (38) we get red (γ L(n 1)) { } f (x )< max 2 { }. (39) =1,...,n mn =1,...,n In case (b), we have α () 0, ()1 = α () =, and f ( x () e ) f ( x() ) γ ( ) 2 2 f ( x () ) γ ( ) 2 2. Then, by appyng the Mean Vaue Theorem, we obtan f ( ) T v () e <γ 2. where v () = x () λ () e wth λ () (0, 1). Then, by repeatng the precedng reasonngs we obtan red { } f (x )<(γ L(n 1)) max 2 { } (40) =1,...,n mn =1,...,n Case (). It s anaogous to Case (), so that condtons (39) and (40) hod.

132 LUCIDI AND SCIANDRONE Case (). In ths case we have ether (a) x () = x or (b) x () x. In case (a), from the nstructons of the agorthm, recang that ()1 = = α, where α s the nta stepsze, we have ( ) f x () e f ( ( ) ) 2 x () γ 2 (41) ( ) f x () e f ( ( ) ) 2 x () γ 2. (42) By appyng the Mean Vaue Theorem we obtan f ( ( ) T ) 2 u () e γ 2 (43) f ( ( ) T ) 2 v () e γ 2, (44) where u () = x () λ() 1 e,v () = x () λ() 2 e, wth λ() 1,λ2 () (0, 1). From (43), tang nto account the Lpschtz assumpton on f we get f (x ) T e γ 2 L x u () γ 2 L x x () L γ 2 { } nl max L =1,...,n = (γ L) 2 { } nl max. =1,...,n Hence, t foows red (γ L(n 1)) { } f (x ) max 2 { }. (45) =1,...,n mn =1,...,n From (44), by repeatng the same reasonngs, we obtan red (γ L(n 1)) { } f (x ) max 2 { }. (46) =1,...,n mn =1,...,n

BOUND CONSTRAINED OPTIMIZATION 133 From (45) and (46) t foows red f (x ) (γ L(n 1)) { } max 2 { }. (47) =1,...,n mn =1,...,n Let us consder the case (b). Wthout oss of generaty, we can assume that n ths case we have α () 0, ()1 = = α (), and wth f ( x () e ) f ( x() ) γ ( ) 2 2 f ( x () ᾱ e ) > f ( x() ) γ (ᾱ ) 2 2, ᾱ = δ () where δ () = δ f x () δ e F, and δ () (δ, 1) otherwse (n ths case δ () s such that x () δ () e F ). By appyng the Mean Vaue Theorem, we can wrte and f ( u () ) T e γ 2 γ 2 α, f ( ) T v () e > γ 2 δ (), δ () where u () = x () λ() 1 α ()e,v () = x () λ() 2 δ () e wth λ() 1,λ2 () (0, 1). Tang nto account the Lpschtz assumpton on f we can wrte f (x ) T e γ 2 γ 2 L x u () L x x () L ()1 (γ L) 2 { } nl max, =1,...,n f (x ) T e > γ 2 δ () δ () L x v () γ δ 2 L x x () L (γ L) δ 2 { } nl max. =1,...,n

134 LUCIDI AND SCIANDRONE Then we have red f (x ) (γ L(n 1)) { } max 2 δ { } (48) =1,...,n mn =1,...,n Fnay, from (39), (40) and (48) we obtan ( ) red f (x ) n 1/2 (γ L(n 1)) { } max 2 max{,δ} { }, =1,...,n mn =1,...,n and ths concudes the proof. Remar 2. We note that f at each teraton we set x 1 = x 1 (namey x 1 = x α d, where d {e 1,...,e n, e 1,..., e n } and where α s equa to zero or s produced by the Expanson Step) then the asserton of Proposton 3 hods for a. In fact, n ths case we have h n for a > n. Fnay, by requrng a stronger assumpton on the nose, we have the foowng convergence resut. Proposton 4. Suppose that f s bounded beow on the feasbe set F and that ts gradent s Lpschtz contnuous (wth constant L) on F. Let {x } be the sequence produced by Agorthm Mode. Then f m mn =1,...,n { } = 0, (49) then every mt pont of {x } s a statonary pont for probem (1). Proof: Let x be any mt pont of {x }, that s m, K x = x, where K {0, 1,...}. In order to prove the thess, by contradcton, et us suppose that x s not a statonary pont for probem (1), that s red f ( x) ɛ, (50) for some ɛ>0. From the nstructons of the agorthm we have that, for each K, there exsts an ndex ν, wth 0 ν < n such that h ν n and x j = x j 1 α j 1 e ( j) for j = 1,...,ν. (51)

BOUND CONSTRAINED OPTIMIZATION 135 By Proposton 3 we have red f (x ν ) c 1 max =1,...,n { ν } c2 ν mn =1,...,n { ν }, hence, recang (49) and that pont () of Proposton 1 st hods aso n presence of bounded nose, t foows m, K red f (x ν ) =0. (52) Then, by usng the Lpschtz assumpton on f, we can wrte ν red f (x ) red f (x ν ) L x x ν L α j 1, from whch, recang agan () of Proposton 1, the contnuty assumpton of f and (52) we obtan red f ( x) =0, j=1 whch contradcts (50). 5. Premnary computatona resuts In order to evauate a possbe practca nterest of the proposed agorthm mode, we have used 41 standard test probems of dmenson n ranng form 2 to 10. These probems are seected from two test sets: the frst one s made of 23 probems defned n [3], whch are obtaned from the set of functons suggested n [10]. The second set conssts of the a box constraned probems of the Hoc-Schttows coectons [5, 12]. Furthermore, we have consdered a rea test probem whch derves from an appcaton [13] regardng the desgn of nstruments for magnetc resonance. On the bass of the proposed agorthm mode, we have mpemented n Fortran code two dfferent agorthms. Agorthm 1. The pont x 1 produced at Step 5 s set equa to x 1, therefore, ether x 1 = x or x 1 = x α e (x α e ), where {1,...,n}. Agorthm 2. It tres to produce the pont x 1 at Step 5 by usng an approxmaton scheme. Now we descrbe more n deta the choces made n these mpementatons. 5.1. Choce of parameters The parameters whch appear n the agorthm mode have been set as foows. γ = 10 6 δ = 0.25 = 0.5 0 = 0.5 = 1,...,n.

136 LUCIDI AND SCIANDRONE We note that we have not performed an extensve emprca tunng of the parameters of the agorthm. We have adapted choces usuay adopted n nesearch technques of gradentbased agorthms. 5.2. Approxmaton scheme used n Agorthm 2 In Agorthm 2 we have made an nta attempt to expot the nformaton on the objectve functon obtaned at the prevous teratons n order to produce a better pont x 1 wth respect to x 1. In partcuar, every n teratons, we construct (when the agorthm has produced a suffcent number of ponts cose to x ) a smpe quadratc mode q(x) = 1 2 x T Qx c T x b of the objectve functon. The N = n(n 1)/2 n 1 free parameters of the mode are determned by mnmzng the error M (q(x j ) f (x j )) 2 (53) j=1 beng M > 0 and x 1,...,x M the ast M ponts where f has been evauated and such that x j ũ, wth = max {, x 100 } ũ = max { u, x 100 } = 1,...,n = 1,...,n. Then, we appy a mnmzaton method for computng a statonary pont of the defned quadratc probem mn q(x) (54) x ũ and the obtaned pont s accepted f t produces a reducton of f. We note that the box [, ũ] has the roe of seectng ponts whch are not too far from the current pont x. In our tests we have chosen M = N 5 and we have used the routnes F04JAF and E04NAF of NAG brary to sove, respectvey, the near east squares probem (53) and the box constraned quadratc probem (54). 5.3. Stoppng crteron Snce the defnton of an effcent stoppng crteron s out of the scope of the wor, we have adapted the same approach proposed n [3]. Let f 0 be the vaue of f at the startng

BOUND CONSTRAINED OPTIMIZATION 137 pont x 0 and f be the best nown functon vaue. Then we ntroduce the quotent q = f (x ) f f 0 f, whch can be consdered a measure of the speed of convergence and we have stopped an agorthm whenever q ɛ, (55) where ɛ>0saprefxed vaue. By usng dfferent vaues of ɛ we can have an dea on the effcency of an agorthm. However, n some test probems the goba mnmum s not the unque statonary pont. Therefore, an agorthm coud generate a sequence convergng towards a statonary pont x, wth f ( x) > f, and coud never satsfy the crteron (55). To tace ths possbe occurrence, we have ntroduced aso the foowng stoppng crteron max =1,...,n { } 10 5. (56) Fnay, we say that an agorthm has faed when t has performed a number N max = 1000 of functon evauatons wthout satsfyng any of the two the stoppng crtera. 5.4. Numerca experence We have tested Agorthms 1 and 2 on the set of standard probems wth three dfferent vaues of ɛ n crteron (55), namey ɛ = 10 1, ɛ = 10 3, ɛ = 10 6. The compete resuts are reported n [9]. Here we descrbe some summares of these resuts. In the foowng tabe we report the tota number of functon evauatons needed to sove the probems where both the agorthms have been abe to satsfy crteron (55), the tota number of faures and the tota number of stops due to crteron (56). From the resuts of Tabe 1 we can note that for ɛ = 10 1 both the agorthms have been abe to sove a the test probems. For ɛ = 10 3 most of the test probems are st soved by the two agorthms. Whereas the case ɛ = 10 6 ponts out the utty of approxmaton technques n a dervatve-free agorthm. In fact, when the degree of requred precson s hgh, t seems to be necessary to expot as much as possbe the nformaton on the Tabe 1. Cumuatve resuts. Tota n f No. of faures No. of stops ɛ = 10 1 Ag. 1 2045 0 1 Ag. 2 1099 0 1 ɛ = 10 3 Ag. 1 3556 5 1 Ag. 2 1974 3 2 ɛ = 10 6 Ag. 1 3375 10 5 Ag. 2 1590 6 5

138 LUCIDI AND SCIANDRONE Tabe 2. Number of WINS. No. of wns n terms of n f No. of baances ɛ = 10 1 Ag. 1 0 34 Ag. 2 6 34 ɛ = 10 3 Ag. 1 3 22 Ag. 2 10 22 ɛ = 10 6 Ag. 1 4 14 Ag. 2 7 14 objectve functon obtaned durng the teratons of the agorthm. From Tabe 1 we can aso note that the smpe approxmaton scheme used n Agorthm 2 has aowed us a sgnfcant computatona savng n terms of number of functon evauatons. Then we have anaysed more n deta the behavour of Agorthms 1 and 2. In partcuar, we say that an agorthm wns f the number of functon evauatons requred to sove a test probem s smaer or equa to the 95% of the number requred by the other agorthm. In Tabe 2 we report the tota number of wns. Tabe 2 shows that n many soved probems the performances of the two agorthms are comparabe and hence, that Agorthm 2 outperforms Agorthm 1 ony n few probems. However, tang nto account aso the resuts of Tabe 1, we can concude that n such probems the mprovement due to the use of an approxmaton s consderabe. In order to have a frst dea on the practca nterest of the proposed approach, we have compared Agorthm 2 wth a method usng fnte-dfferences gradents. As t s we nown, methods of ths cass are very effcent n absence of nose. Therefore, the comparson s an hard test for Agorthm 2 whch s a drect search method, namey a method whch does not try to approxmate expcty the frst order dervatves. In partcuar, we have used E04UCF routne of NAG brary, whch s a sequenta quadratc programmng method. Smary to Agorthm 2 we have stopped the NAG routne whenever (55) has been satsfed (aso n ths case we have used three dfferent vaues of ɛ:10 1, 10 3, 10 6 ). In some runs E04UCF routne was termnated by ts stoppng crteron and the vector returned x was a statonary pont such that f ( x) > f. In two test probems, E04UCF was unabe to produce a pont dfferent from the startng pont. We have consdered these cases as faures of E04UCF. The compete resuts regardng E04UCF are reported n [9]. Here, for brevty s sae, we report a summary of the comparsons between Agorthm 2 and E04UCF (n Tabes 3 and 4). In partcuar, n Tabe 3 we report the tota number of functon evauatons needed to sove the probems where both the Agorthm 2 and the consdered NAG routne have been abe to satsfy crteron (55), the tota number of faures and the tota number of stops due to crteron (56) for Agorthm 2, and to the defaut stoppng crteron for the NAG routne. In Tabe 4 we show the tota number of wns n terms of number of functon evauatons. From the resuts of Tabe 3, we can note that for ɛ = 10 1 Agorthm 2 outperforms E04UCF. For ɛ = 10 3 t s st compettve n terms of number of functon evauatons wth E04UCF and the two agorthms are comparabe n terms of faures. When ɛ = 10 6, the

BOUND CONSTRAINED OPTIMIZATION 139 Tabe 3. Cumuatve resuts. Tota n f No. of faures No. of stops ɛ = 10 1 Ag. 2 1069 0 1 E04UCF 1601 2 1 ɛ = 10 3 Ag. 2 1989 3 2 E04UCF 2052 2 4 ɛ = 10 6 Ag. 2 2381 6 5 E04UCF 2195 2 4 Tabe 4. Number of WINS. No. of wns n terms of n f No. of baances ɛ = 10 1 Ag. 2 29 1 E04UCF 8 1 ɛ = 10 3 Ag. 2 21 1 E04UCF 10 1 ɛ = 10 6 Ag. 2 15 0 E04UCF 11 0 behavour of Agorthm 2 s comparabe wth the one of E04UCF n terms of number of functon evauatons. However, E04UCF s abe to sove a (sghty) arger number of probems. The precedng resuts, athough far from beng exhaustve, show a satsfactory behavour of Agorthm 2 compared wth a method usng fnte-dfferences dervatves. We reca that methods of ths nd are effcent n the case that the objectve functon vaues are not affected by any nose, but they are not consdered reabe for sovng nosy probems. In fact, the behavour of methods usng fnte-dfferences dervatves deterorates aso wth a sma nose. In partcuar, we have repeated the tests by consderng functon vaues gven by f (x) = f (x)(1 η), η N(0,σ 2 ), where N(0,σ 2 ) denotes a Gaussan dstrbuted random number wth zero mean and varance σ 2 = 10 9. We note that the convergence anayss deveoped n Secton 4 hods for bounded nose, whe the one consdered s unbounded. However, due to the sma vaue of the varance, the nose can be consdered bounded n practce. We have obtaned the foowng resuts. Agorthm 2: for the three vaues of ɛ, the number of faures s not changed wth respect to the noseess case. E04UCF routne: for ɛ = 10 1 the number of faures changes from 2 to 27; for ɛ = 10 3 the number of faures changes from 2 to 32; for ɛ = 10 6 the number of faures changes from 2 to 32.

140 LUCIDI AND SCIANDRONE 5.5. A rea desgn probem We consder a rea probem arsng from an appcaton [13] whch regards the optma desgn of apparat for magnetc resonance. The appcaton deas wth the constructon of apparat wth reduced szes to be used for cnca anayss of perphera regons of the body. These apparat are based on resstve magnets that can be manufactured at very ow cost. However, they requre a good magnetc fed unformty n a arge part of ther voume, and the reduced szes mae t dffcut to obtan ths fed unformty. In partcuar, the magnetc fed s generated by four currents: I 1, I 2, I 3, I 4. The requred fed s B 0 and t must be as unform as possbe n a spherca regon at the center of the magnet. The vector of currents s denoted by I = (I 1, I 2, I 3, I 4 ) T and the z component of the magnetc fed generated n a pont r j by B z (I ; r j ). More formay, the probem s mn f (I ) = N p [B z (I ; r j ) B 0 ] 2 (57) s.t. 0 I U j=1 The ower bounds (0 I ) are mposed because the currents vaues must have the same sgn for reasons reated to the constructon of the magnet. The upper bounds (I U) depend on the aowed power dsspaton. We note that, for each r j, wth j = 1,...,N p, the functon B z (.; r j ) : R 4 R s not nown anaytcay, but for each I R 4 the vaue B z (I ; r J ) can be drecty measured by a Gaussmeter. However, the evauaton of the objectve functon s not expensve. In [13], neura modes have been defned for approxmatng the functons B z (.; r j ), wth j = 1,...,N p, by usng a massve data set generated offne. Then, a gradent-based method has been apped to probem (57), where the terms B z (I ; r j ) have been repaced by the anaytca neura mode determned. In practce some parameters characterzng the magnet w change over the tme, and hence, the fed unformty obtaned by the procedure proposed n [13] can become unacceptabe. In ths case, an onne procedure s necessary for recomputng qucy the vector of currents startng from the currents prevousy determned offne. Ths procedure shoud be abe to sove probem (57) by usng drect measurements for computng the vaues B z (I ; r J ). Snce the objectve functon vaues w be affected by an amount of nose (due to the drect measurements), the use of a method based on fnte-dfferences gradents w be mpractca, whe the adopton of a dervatve-free method appears sutabe. In order to evauate the potentates of ths approach, we have apped Agorthm 2 to probem (57), where for each feasbe vector I, the vaues B z (I ; r j ) are determned by a smuaton program. In partcuar, the stoppng crteron (55) s we suted for ths cass of appcatons. In fact, the vaue f can correspond to an acceptabe vaue of the fed unformty. Startng from a vector of nta currents such that f 0 = 11.986, assumng f = 0.049, we have used three dfferent vaues of ɛ n the stoppng crteron, and we have obtaned the foowng resuts (where and n f are, respectvey, the number of teratons and the number of functon evauatons requred to satsfy the stoppng crteron (55), whe f s the objectve functon attaned).

BOUND CONSTRAINED OPTIMIZATION 141 ɛ = 10 1 = 32 n f = 102 f = 0.067 ɛ = 10 3 = 37 n f = 120 f = 0.061 ɛ = 10 6 = 377 n f = 1060 f = 0.050 To examne more n deta the behavour of Agorthm 2, we aso report some ntermedate resuts. The resuts show the practcabty of usng Agorthm 2 to update the current vaues. 0 5 10 20 50 100 200 377 n f 1 31 45 73 156 293 573 1060 f 11.9869 0.4751 0.2203 0.1237 0.05430 0.0515 0.0507 0.0500 In fact Agorthm 2 has been abe to reobtan a good unform fed by performng a mted number of functon evauatons. References 1. E.J. Anderson and M.C. Ferrs, A drect search agorthm for optmzaton wth nosy functons evauatons, Mathematca Programmng Technca Report 96-11, Computer Scences Department, Unversty of Wsconsn, Madson, Wsconsn, 1996. 2. A.R. Conn, K. Schenberg, and Ph.L. Tont, Recent progress n unconstraned nonnear optmzaton wthout dervatves, Mathematca Programmng, vo. 79, pp. 397 415, 1997. 3. C. Ester and A. Neumaer, A grd agorthm for bound constraned optmzaton of nosy functons, IMA Journa of Numerca Anayss, vo. 15, pp. 585 608, 1995. 4. L. Grppo, F. Lampareo, and S. Lucd, Goba convergence and stabzaton of unconstraned mnmzaton methods wthout dervatves, Journa of Optmzaton Theory and Appcatons, vo. 56, no. 3, pp. 385 406, 1988. 5. W. Hoc and K. Schttows, Test Exampes for Nonnear Programmng Codes, Sprnger-Verag: Bern, 1981. Lectures Notes n Economcs and Mathematca Systems, vo. 187. 6. C.T. Keey, Iteratve Methods for Optmzaton. Fronters n Apped Mathematcs, Socety for Industra and Apped Mathematcs: Phadepha, 1999. 7. R.M. Lews and V. Torczon, Pattern search methods for bound constraned mnmzaton, SIAM Journa on Optmzaton, vo. 9, pp. 1082 1099, 1999. 8. S. Lucd and M. Scandrone, On the goba convergence of dervatve free methods for unconstraned optmzaton, Technca Report 18-96, DIS, Unverstà d Roma La Sapenza. 9. S. Lucd and M. Scandrone, A dervatve-free agorthm for bound constraned optmzaton, R. 498 1999, IASI, Consgo Nazonae dee Rcerche. 10. J.J. Moré, B.S. Garbow, and K.E. Hstrom, Testng unconstraned optmzaton software; ACM Trans. on Math. Software, vo. 7, pp. 17 41, 1981. 11. M.J.D. Powe, Drect search agorthms for optmzaton cacuatons, Acta Numerca, vo. 7, pp. 287 336, 1998. 12. K. Schttows, More Test Exampes for Nonnear Programmng Codes. Sprnger-Verag: Bern, 1987. Lectures Notes n Economcs and Mathematca Systems, vo. 282. 13. M. Scandrone, G. Pacd, L. Testa, and A. Sotgu, Compact ow fed magnetc resonance magng magnet: Desgn and optmzaton, Revew of Scentfc Instruments, vo. 71, no. 3, pp. 1534 1538, 2000. 14. V. Torczon, On the convergence of pattern search methods, SIAM Journa on Optmzaton, vo. 7, pp. 1 25, 1997.

142 LUCIDI AND SCIANDRONE 15. P. Tseng, Fortfed-descent smpca search method: A genera approach, SIAM Journa on Optmzaton, vo. 10, pp. 269 288, 2000. 16. M.H. Wrght, Drect search methods: Once scorned, now respectabe, Numerca Anayss 1995, Proceedngs of the 1995 Dundee Benna Conference n Numerca Anayss, D.F. Grffths and G.A. Watson (Eds.) Addson-Wesey Longman: Harow, UK, 1996, pp. 191 208. 17. S.K. Zavrev, On the goba optmzaton propertes of fnte-dfferences oca descent agorthms, Journa of Goba Optmzaton, vo. 3, pp. 67 78, 1993.