7Applying Nelder Mead s Optiization Algorith APPLYING NELDER MEAD S OPTIMIZATION ALGORITHM FOR MULTIPLE GLOBAL MINIMA Abstract Ştefan ŞTEFĂNESCU * The iterative deterinistic optiization ethod could not ore find ultiple global inia of a given objective function ( [6] ). Generally, the probabilistic optiization algoriths have not this restrictive behaviour, to deterine only a single global iniu point. In this context we ll prove experientally that Nelder-Mead s heuristic procedure can detect successfully ultiple global extreal points. Key words : global optiization, Nelder-Mead algorith, ultiple inia JEL Classification code : C61, C02. 1. Introduction For an arbitrary function h : D R with D R we intend to find those points * * * * x* D, x * = ( x1, x2, x3,..., x ) such that x* = arg in h( w) (1) Therefore w D h( x*) = in h( x) (2) x D where x = ( x1, x2, x3,..., x ). In fact h (x*) is the iniu global value for the function h (x), x D. In the literature ( [6] ) are very present the classical derivative optiization ethods, based on the gradient direction for finding the iniu global value h (x*). * Associate Professor, PhD., Faculty of Matheatics and Inforatics, University of Bucharest, e-ail : stefanst@fi.unibuc.ro Roanian Journal of Econoic Forecasting 4/2007 97
Institute of Econoic Forecasting But always in practice the exact expression of the gradient function could be extreely hard for coputing. For this reason the gradient expression is often approxiated by finite differences. The non-derivative ethods use directly only soe selected values h (x) ( [1]-[5], [7]- [10] ). In this context we reark the odel-based variant and the geoetry-based ethod too. More precisely, the odel-based procedures work with an interpolation or also with a least-squares approxiation of the objective function h (x) to copute the next iteration in searching process of x *. Contrary, the geoetry-based algoriths do not necessary involve an explicit auxiliary for of the function h (x) and essentially produce saples fro x D which have iposed properties. The Nelder-Mead ( NM ) ethod is oriented for solving a continuous unconstrained optiization proble of type (2). A NP type algorith is clearly an authentic geoetry-based procedure whose flexibility is given by its four paraeters α, β, γ, δ which adjust the search process for the iniu function values. In general, the geoetry-based procedures and particularly the NP algorith are easily to be prograed. Their ajor advantage is iposed by a relative non frequently evaluation of the function h (x). Usually, in practice, the coputation of a coplex objective function h (x) is very tie-consuing. Often the evaluation of h (x) deands before an auxiliary data collected activity. 2. An ipleentation of the NM algorith The iterative optiization procedures generally use only a starting point x 1 D, chosen by specific rules. Contrary, the NP algorith consider a nondegenerate siplex inside the doain D R as starting figure. At every iteration step the NP algorith odifies a single vertex of the current siplex by applying a λ-transfor. In this way it results another nondegenerate siplex. More precisely, for any two points y R and z R we can produce a new point w R by using a λ-rule, that is w = z + λ ( y z), λ R (3) So, if y = ( y1, y 2, y 3,..., y ), z = ( z1, z2, z3,..., z ), w = ( w1, w 2, w 3,..., w ) we get w = z + λ y z ), 1 j (4) j j ( j j Depending on the value of the coefficient λ, λ { α, β, γ, δ }, and also on the individual significance of the points y and z, we can siulate ore geoetric type 98 Roanian Journal of Econoic Forecasting 4/2007
Applying Nelder Mead s Optiization Algorith operations as a α-reflection, a β-expansion, a γ-contraction or a δ-shrinkage ( [1}, [2], [5], [9] ). The classical Nelder-Mead algorith [5] has a lot of little odified fors ( copare, for exaple, the NP procedures presented in [1]-[3], [7], [9] ). For the present study it was ipleented in MatLab the variant given in [3]. This variant operates with the following λ-paraeters : α = 1 β = 2 γ = 0. 5 δ = 0. 5 (5) 3. Multiple global inia In the subsequent we intend to test the NM algorith when the function h (x) has ultiple inia. We are interested to see if the NM procedure could find all the global extreal values x *. The following exaple will give us the right answer. Exaple 1. For = 2 we will consider the function h1 : D R with D = [ 0, 6] x [ 3, 12] 2 D R (6) h w) = h( w1, w 2 )) = 4+ ( w1 1)( w1 5) + w 2 ( w1 2)( w1 3) Obviously h s) = h ( t) = inf h ( w) 4 1 ( 1 1 = w D where s = (1, 2) t = (5, 6) (8) and ore s D, t D. Fro a straightforward reasoning we deduce that the function h w ) has, on the doain D, only two global extreal points. These special points are just the vectors s and t defined by the forulas (8). (7) Graphic 1 gives us an iagine about how the function h w) fluctuates. We intend to verify if the NP procedure could find both iniizer points s and t. The Graphic 1 does not suggest us clearly the exact places where we have the two global extreal points s, t. For this reason we can study the variability of the function h 2( w), h2 ( w) = h2 (( w1, w 2 )) = h( w1, w 2 )) (9) The iniu values of the function h w ) becae the axiu values for the application h 2( w ). The Graphic 2 suggests at least two global axiization points for the function h ( ). So, h ( ) has ultiple global iniizer points. 2 w 1 w Roanian Journal of Econoic Forecasting 4/2007 99
Institute of Econoic Forecasting 100 Roanian Journal of Econoic Forecasting 4/2007
Applying Nelder Mead s Optiization Algorith But the correct answer regarding the nuber of the global extreal points of h w ) is obtain after an interpretation of the contour lines structure. So, we conclude that the function h w ) has only two iniizer points ( see Graphic 3 ). Running 100 ties the NM algorith we get always only the iniizer vectors s or t but after a different nuber n of iterations. More, the variants s and t appeared randoly and around the sae proportion ( see Table 1 ). Table 1 The iniization value x * obtained after n iterations ( NM algorith, x* { s, t}, function h ( ) ). 1 w x* n x* n x* n x* n x* n t 63 s 58 S 60 t 56 s 56 t 61 s 56 S 56 t 59 t 63 s 66 t 61 S 59 s 60 t 55 s 52 s 59 S 54 t 60 t 72 s 59 s 58 T 61 t 66 s 56 s 56 t 67 T 54 s 56 s 53 s 53 s 48 S 54 t 59 t 62 s 56 s 59 T 55 t 56 s 55 Roanian Journal of Econoic Forecasting 4/2007 101
Institute of Econoic Forecasting s 57 t 54 T 66 s 55 s 55 t 61 t 91 S 55 s 62 t 58 t 70 t 81 S 62 s 55 t 68 s 62 s 65 T 69 t 60 s 57 s 57 t 66 S 55 t 59 s 60 s 123 t 54 T 54 t 58 s 56 t 65 t 59 T 62 s 55 t 52 t 53 s 59 S 68 t 57 t 57 s 58 s 73 S 56 t 56 t 116 s 55 s 66 S 61 s 57 t 62 s 55 t 57 S 72 s 63 t 77 t 61 t 53 T 62 t 60 t 60 4. Concluding rearks It is very known fro the literature that the iterative deterinistic optiization ethods could not usualy find ore ultiple inia of a given objective function h (w) ( details in [6] ). But this behavioural restriction isn t generally true for the probabilistic optiization algoriths. In the present paper we proved experientaly that the Nelder-Mead heuristic procedure can detect successfully ultiple extreal global points. More, in exaple 1, the NP procedure identified approxiately in the sae proportion the both global iniizer points ( see Table 1 ). References R, Barton, J. S. Ivey, Nelder Mead siplex odifications for siulation optiization, Manageent Science 42, 7(1996), 954 973. L. Han, M. Newann, Effect of diensionality on the Nelder-Mead siplex ethod, Optiization Methods and Software, 21, 2006), 1-16. J.C. Lagarias, J.A. Reeds, M.H. Wright, P.E. Wright, Convergence properties of the Nelder-Mead siplex ethod in low diensions, SIAM Journal on Optiization, 9, 1998), 112-147. K.I.M. McKinnon, Convergence of the Nelder-Mead siplex ethod to a nonstationary point, SIAM Journal on Optiization, 9, (1998), 148-158. J.A. Nelder, R. Mead, A siplex ethod for function iniization, Coputer Journal 7, (1965), 308-313. W.H. Press, B.P. Flannery, S.A. Teukolsky, W.T. Vettering, Nuerical Recipes in C. Cabridge University Press, Cabridge, UK, 1988. 102 Roanian Journal of Econoic Forecasting 4/2007
Applying Nelder Mead s Optiization Algorith C.J. Price, I.D. Coope, D. Byatt, A convergent variant of the Nelder-Mead algorith, Journal of Optiization Theory and Applications, 113, (2002), 5 19. A.S. Rykov, Siplex algoriths for unconstrained optiization, Probles of Control and Inforation Theory, 12, (1983), 195-208. P. Tseng, Fortified-descent siplicial search ethod: a general approach SIAM Journal on Optiization 10, (2000), 269 288. W.C. Yu, The convergence property of the siplex evolutionary techniques, Scientia Sinica,Special issue of Matheatics 1, (1979), 68 77. Roanian Journal of Econoic Forecasting 4/2007 103