Author manuscript, published in "GECCO workshop on Black-Box Optimization Benchmarking (BBOB') () 9-" DOI :./8.8 Mirrored Variants of the (,)-CMA-ES Compared on the Noiseless BBOB- Testbed [Black-Box Optimization Benchmarking Workshop] Anne Auger, Dimo Brockhoff, and Nikolaus Hansen Projet TAO, INRIA Saclay Ile-de-France LRI, Bât 9, Univ. Paris-Sud 9 Orsay Cedex, France firstname.lastname@inria.fr inria-, version - Jul ABSTRACT Derandomization by means of mirrored samples has been recently introduced to enhance the performances of (, λ)- Evolution-Strategies (ESs) with the aim of designing fast and robust stochastic local search algorithms. This paper compares on the BBOB- noiseless benchmark testbed two variants of the (,)-CMA-ES where the mirrored samples are used. Independent restarts are conducted up to a total budget of D function evaluations, where D is the dimension of the search space. The results show that the improved variants are significantly faster than the baseline (,)-CMA-ES on functions in D (respectively when using sequential selection in addition) by a factor of up to (on the attractive sector function). In no case, the (,)-CMA-ES is significantly faster on any tested target function value in D and D. Moreover, the algorithm employing both mirroring and sequential selection is significantly better than the algorithm without sequentialism on five functions in D with expected running times that are about % smaller. Categories and Subject Descriptors G.. [Numerical Analysis]: Optimization global optimization, unconstrained optimization; F.. [Analysis of Algorithms and Problem Complexity]: Numerical Algorithms and Problems General Terms Algorithms. INTRODUCTION Evolution Strategies (ESs) are robust stochastic search algorithms for numerical optimization where the function to be minimized, f, maps the continuous search space R D c ACM,. This is the authors version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published at GECCO, July,, Portland, OR, USA. http://doi.acm.org/./8.8 into R where D is the dimension of the search space. Recently, a new derandomization technique replacing the independent sampling of new solutions by mirrored sampling has been introduced to enhance the performances of ESs []. In this paper, we assess quantitatively the improvements that can be brought when using mirrored samples instead of independent ones. To do so, we compare on the BBOB- noiseless testbed the (,)-Covariance-Matrix- Adaptation Evolution-Strategy (CMA-ES) with two variants: first the (, m)-cma-es where mirrored samples are used, and second the (, s m)-cma-es that in addition to the mirrored samples uses the concept of sequential selection []. Both variants are described in Sec... THE ALGORITHMS TESTED The three algorithms tested are variants of the well-known CMA-ES [, 9, 8] where at each iteration n, λ new solutions are generated by sampling independently λ random vectors (N i(, C n)) i λ following a multivariate normal distribution with mean vector and covariance matrix C n. The vectors are added to the current solution X n to create the λ new solutions or offspring Xn i = X n + σ nn i(, C n) where σ n is a strictly positive parameter called step-size. In the simple (,)-CMA-ES, the number of offspring λ equals and X n+ is set to the best solution among Xn,..., Xn, i.e., X n+ = argmin{f(xn),..., f(xn)}. In the mirrored variant, denoted (, m)-cma-es, the second and fourth offspring are replaced by the offspring symmetric to the first and third with respect to X n, namely Xn = X n σ nn (, C n) and Xn = X n σ nn (, C n), where σ nn (, C n) and σ nn (, C n) are the random vectors added to X n to get Xn and Xn. The first/second and third/fourth offspring are, thus, negatively correlated. The update of X n+ is then identical to the (,)-CMA-ES, namely X n+ = argmin{f(xn),..., f(xn)}. In the (, s m)-cma-es, sequential selection is implemented. The four offspring solutions are generated with mirrored sampling. Evaluations are carried out in a sequential manner: after evaluating solution Xn, i it is compared to X n and if f(xn) i f(x n), the sequence of evaluations is stopped and X n+ = Xn. i In case all four offspring are worse than X n, X n+ = argmin{f(xn),..., f(xn)} according to the comma selection. In sequential selection, the number of offspring evaluated is a random variable by itself ranging from to λ = allowing to reduce the number of offspring adaptively as long as improvements are easy to achieve [].
inria-, version - Jul Covariance matrix and step-size are updated using the selected steps [9, ]. Independent Restarts. Similar to [], we independently restarted all algorithms as long as function evaluations were left, where maximally D function evaluations have been used. Parameter setting. We used the default parameter and termination settings (cf. [,, 8]) found in the source code on the WWW with two exceptions. We rectified the learning rate of the rankone update of the covariance matrix for small values of λ, setting c = min(, λ/)/((d +.) + µ eff ). The original value was not designed to work for λ <. We modified the damping parameter for the step-size to d σ =.+µ eff /λ+c σ. The setting was found by performing experiments on the sphere function, f : d σ was set as large as possible while still showing close to optimal performance, but, at least as large such that decreasing it by a factor of two did not lead to inacceptable performance. For µ eff /λ =. and µ eff D + the former setting of d σ is recovered. For a smaller ratio of µ eff /λ or for µ eff > D +, the new setting allows larger (i.e. faster) changes of σ. Here, µ eff =. For λ, the new setting might be harmful in a noisy or too rugged landscape. Finally, the step-size multiplier was clamped from above at exp(), while we do not believe this had any effect in the presented experiments. Each initial solution X was uniformly sampled in [, ] D and the step-size σ was initialized to. The source code used for the experiments is available at. As the same parameter setting has been used for all test functions, the crafting effort CrE of all algorithms is.. CPU TIMING EXPERIMENTS For the timing experiment, all three algorithms were run on f 8 with a maximum of D function evaluations and restarted until at least seconds have passed (according to Figure in []). The experiments have been conducted with an 8 core Intel Xeon E machine with. GHz under Ubuntu 9. linux and Matlab R8a. The time per function evaluation was.;.;.;.;.;. times seconds for (,)-CMA-ES,.;.;.;.;.;. times seconds for (, m)-cma-es, and.;.;.; 8.;.; 8. times seconds for (, s m)-cma-es in dimensions ; ; ; ; ; respectively. Note that MATLAB distributes the computations over all 8 cores only for D and D.. RESULTS In this section, experiments according to [] on the benchmark functions given in [, ] are presented. The expected running time (ERT), used in the figures and table, depends on a given target function value, f t = f opt + f, and is computed over all relevant trials as the number of function evaluations executed during each trial while the best function value did not reach f t, summed over all trials and divided by the number of trials that actually reached f t [, ]. Statistical significance is tested with the rank-sum test for a given target f t using, for each trial, either the number of needed function evaluations to reach f t (inverted cmaes.m, version..beta, from http://www.lri.fr/ ~hansen/cmaes_inmatlab.html http://coco.gforge.inria.fr/doku.php?id= bbob--results and multiplied by ), or, if the target was not reached, the best f-value achieved, measured only up to the smallest number of overall function evaluations for any unsuccessful trial under consideration.. Comparing (, m )- With (,)-CMA-ES The (, m)-cma-es is compared with the (,)-CMA-ES in Figures and and in Table. Mirroring within the (, m)-cma-es seems to have an important positive impact compared to the baseline (,)-CMA-ES. In D, we observe a slight worsening only on the Gallagher functions f and f which are not statistically significant. On the other hand, the (, m)-cma-es outperforms the (,)-CMA-ES on functions of which show statistically significant differences (in D and for a target of ): on the sphere function (f ), the expected running time is % lower, on the separable ellipsoid (f ) it is 8% lower, on the non-separable ellipsoid (f ) 9% lower and on the attractive sector function (f ) the expected running times even differ by a factor of about. These differences are less significant in D (Table ), but Fig. shows similar improvements also for D and smaller dimensions.. Comparing (, s m)- With (, m )-CMA-ES Results comparing the (, m)-cma-es and the (, s m)- CMA-ES are presented in Figures and and in Table. The results show that the (, s m)-cma-es is statistically significantly better than the (, m)-cma-es on five functions in D with expected running times that are about % smaller on f and f, % smaller on f, % smaller on f, and about % smaller on f than the ones for the (, m)-cma-es. Contrary, there are only two functions where the (, s m)-cma-es is worse than the (, m)-cma-es (% on f and about % on f ) but these differences are not statistically significant. To conclude, the (, s m)-cma- ES should be clearly preferred over the (, m)-cma-es on the BBOB- testbed and, according to Sec.., also over the (,)-CMA-ES. Worth to mention is also the fact, that the (, s m)-cma-es shows better performance than the best algorithm from the BBOB-9 benchmarking on the attractive sector function (f ) in D by about % and a slightly better performance on the ellipsoid (f ) in D, see [].. Comparing (, s m)- With (,)-CMA-ES Regarding this comparison, we refrain from showing the plots and tables due to space limitations. Not surprisingly when considering Sec.., the results show a statistically significant improvement for the (, s m)-cma-es over the (,)- CMA-ES on functions in D: f (% faster), f (9% faster), f (faster by a factor of about ), f (about times faster), f (9% faster), f (% faster), and f (about % faster). Only on the Gallagher function with peaks (f ), an increase of the expected running time by a factor of about can be observed which is, however, not statistically significant due to the low number of successful runs.. CONCLUSIONS The idea behind derandomization by means of mirroring introduced in [] is to use only one random sample from a multivariate normal distribution to create two (negatively correlated or mirrored) offspring. Thereby, one offspring is generated by adding a random sample to the parent solution
Sphere Ellipsoid separable Rastrigin separable Skew Rastrigin-Bueche Linear slope Attractive sector Step-ellipsoid 8 Rosenbrock original inria-, version - Jul Schaffer F, cond. Sharp ridge 9 Rosenbrock rotated 8 Schaffer F, cond. Sum of diff. powers Ellipsoid Discus Rastrigin 9 Griewank-Rosenbrock Schwefel x*sin(x) Weierstrass Bent cigar Gallagher peaks Gallagher peaks Katsuuras Lunacek bi-rastrigin Figure : Expected running time (ERT in log of number of function evaluations) of (, m)-cma-es versus (,)-CMA-ES for target values f [ 8, ] in each dimension for functions f f. Markers on the upper or right edge indicate that the target value was never reached by (, m)-cma-es or (,)-CMA-ES respectively. Markers represent dimension: :+, :, :, :, :.
-D -D all functions of trials.. + - - f- +: / -: /9 -: / : / of trials.. + - - f- +: / -: / -: / : / inria-, version - Jul separable fcts moderate fcts ill-conditioned fcts multi-modal fcts weak structure fcts of trials of trials of trials of trials of trials. log of FEvals / DIM. + f- - -.. log of FEvals / DIM. + f-9 - -.. log of FEvals / DIM. + f- - -.. log of FEvals / DIM. + f-9 - -.. log of FEvals / DIM. + f- - -.. log of FEvals / DIM f- - - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f-9 - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f-9 - - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - - - log of FEvals(A)/FEvals(A) of trials of trials of trials of trials of trials. log of FEvals / DIM. + f- - -.. log of FEvals / DIM. + f-9 - -.. log of FEvals / DIM. + f- - -.. log of FEvals / DIM. + f-9 - -.. log of FEvals / DIM. + f- - -.. log of FEvals / DIM f- - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f-9 - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f-9 - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - - log of FEvals(A)/FEvals(A) Figure : Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in -D (left) and - D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to reach a target value f opt + f with f = k, where k {,,, 8} is given by the first value in the legend, for (, m)-cma-es (solid) and (,)-CMA-ES (dashed). Light beige lines show the ECDF of FEvals for target value f = 8 of algorithms benchmarked during BBOB-9. Right sub-columns: ECDF of FEval ratios of (, m)-cma-es divided by (,)-CMA-ES, all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > or <. The legends indicate the number of functions that were solved in at least one trial ((, m)-cma-es first).
Sphere Ellipsoid separable Rastrigin separable Skew Rastrigin-Bueche Linear slope Attractive sector Step-ellipsoid 8 Rosenbrock original inria-, version - Jul Schaffer F, cond. Sharp ridge 9 Rosenbrock rotated 8 Schaffer F, cond. Sum of diff. powers Ellipsoid Discus Rastrigin 9 Griewank-Rosenbrock Schwefel x*sin(x) Weierstrass Bent cigar Gallagher peaks Gallagher peaks Katsuuras Lunacek bi-rastrigin Figure : Expected running time (ERT in log of number of function evaluations) of (, s m)-cma-es versus (, m)-cma-es for target values f [ 8, ] in each dimension for functions f f. Markers on the upper or right edge indicate that the target value was never reached by (, s m)-cma-es or (, m)-cma-es respectively. Markers represent dimension: :+, :, :, :, :.
inria-, version - Jul f e+ e+ e- e- e- e- #succ f / (,)-CMA-ES..9 / (,m)-cma-es.. 9 / f 8 8 88 9 9 9 / (,)-CMA-ES / (,m)-cma-es 9 / f / (,)-CMA-ES..e / (,m)-cma-es. 9 / f 8 8 9 9 / (,)-CMA-ES.e / (,m)-cma-es..e / f / (,)-CMA-ES...... / (,m)-cma-es...8.8.8.8 / f 8 8 / (,)-CMA-ES..8.8... / (,m)-cma-es.8....9 / f / (,)-CMA-ES...8 9 / (,m)-cma-es... 8 / f 8 9 / (,)-CMA-ES..9..9.. / (,m)-cma-es.9...8. / f 9 / (,)-CMA-ES 9 9. 9 8. / (,m)-cma-es. 9. 8... / f 8 88 / (,)-CMA-ES..9.9. / (,m)-cma-es...... / f / (,)-CMA-ES 9.9.9.. / (,m)-cma-es 9.8.8.. / f / (,)-CMA-ES 9 8.9 8... / (,m)-cma-es.. / f 9 8 / (,)-CMA-ES 9... / (,m)-cma-es. 9.8.8..9 / f 9.8 8 8 / (,)-CMA-ES..9...8. / (,m)-cma-es...9.9. / f 9.9e.e.e.e / (,)-CMA-ES 8. 8.e / (,m)-cma-es. 8.e / f.e.e.e / (,)-CMA-ES..e / (,m)-cma-es.9.8.e / f. 9 9 / (,)-CMA-ES 8.e / (,m)-cma-es.. 9.e / f 8 8 9.e.e / (,)-CMA-ES.e / (,m)-cma-es.9.9 9..e / f 9.e.e.e / (,)-CMA-ES.e.e / (,m)-cma-es 9.e.e / f 8.8e.e.e.e / (,)-CMA-ES. 9.... / (,m)-cma-es.8.9.e / f 8 / (,)-CMA-ES..9.9.8.8. / (,m)-cma-es.... / f 9 9 / (,)-CMA-ES.9 / (,m)-cma-es / f.e.e.e.e / (,)-CMA-ES. 8 / (,m)-cma-es.8..e / f.e.e9.e.e.e / (,)-CMA-ES..e / (,m)-cma-es..e / -D -D f e+ e+ e- e- e- e- #succ f / (,)-CMA-ES. 8 8 9 / (,m)-cma-es. / f 8 9 9 9 9 9 / (,)-CMA-ES 8 9 / (,m)-cma-es 8 / f / (,)-CMA-ES.e / (,m)-cma-es.e / f 8.e 9/ (,)-CMA-ES.e / (,m)-cma-es.e / f / (,)-CMA-ES...... / (,m)-cma-es...... / f 8 / (,)-CMA-ES..9... / (,m)-cma-es....8. / f 9.e.e.e / (,)-CMA-ES.e / (,m)-cma-es.e / f 8 9 / (,)-CMA-ES.9. 8 8. 8 8 / (,m)-cma-es..8... / f 9 / (,)-CMA-ES..... / (,m)-cma-es...... / f 8.e.e.e.e / (,)-CMA-ES.8.8..8.. / (,m)-cma-es...9... / f 98.e.e / (,)-CMA-ES 8...9. / (,m)-cma-es.8.9..8. / f 9.e.e / (,)-CMA-ES 8.8 8.8 8...9.9 / (,m)-cma-es...... / f 8.9e.e.e / (,)-CMA-ES 8..e / (,m)-cma-es. 9.8 8.e / f 9.e / (,)-CMA-ES....9.. / (,m)-cma-es.8.... / f.e.e.e.e.e.e / (,)-CMA-ES.e / (,m)-cma-es.e / f.e.e.9e.e.e / (,)-CMA-ES.e / (,m)-cma-es.e / f.e.e 8.e / (,)-CMA-ES.8e.e / (,m)-cma-es..e / f 8.e.8e.e.e / (,)-CMA-ES.e / (,m)-cma-es.e / f 9.e.e.e.e / (,)-CMA-ES.e / (,m)-cma-es.e.e / f 8.e.e.e.e.e / (,)-CMA-ES..e / (,m)-cma-es..e / f.e.e.e.8e / (,)-CMA-ES......8 / (,m)-cma-es.9..8...9 / f.e.e.e.e / (,)-CMA-ES 8. 8/ (,m)-cma-es...9 / f..e.9e 8.e 8.e / (,)-CMA-ES.e / (,m)-cma-es 9.e / f.e.e.e.e.e.e / (,)-CMA-ES.e / (,m)-cma-es.e / Table : Expected running time (ERT in number of function evaluations) divided by the best ERT measured during BBOB-9 (given in the respective first row) for the algorithms (,)-CMA-ES and is (, m)-cma- ES for different f values for functions f f. The median number of conducted function evaluations is additionally given in italics, if ERT( ) =. #succ is the number of trials that reached the final target f opt + 8. Bold entries are statistically significantly better compared to the other algorithm, with p =. or p = k where k > is the number following the symbol, with Bonferroni correction of 8. and a second offspring then equals the solution which is symmetric to the first one with respect to the parent (by adding the negative sample to the parent). Here, this concept of mirroring has been integrated within two variants of a simple (,)-CMA-ES (of which the (, s m)-cma-es uses sequential selection [] in addition and the (, m)-cma-es does not) and compared on the noiseless BBOB- testbed. The results show that using mirroring gives an improvement of at least % on functions (stat. significant in D and as target). Combining mirroring with sequential selection is even better showing further improvements over the (, m)-cma-es by at least % on functions. Acknowledgments This work was supported by the French national research agency (ANR) in the projects SYSCOMM ANR-SYSC- and COSINUS ANR-COSI--.. REFERENCES [] A. Auger, D. Brockhoff, and N. Hansen. Mirrored sampling and sequential selection for evolution strategies. Rapport de Recherche RR-9, INRIA Saclay Île-de-France, April.
-D -D all functions of trials.. + - - f- +: / -: / -: / : / of trials.. + - - f- +: / -: / -: / : / separable fcts of trials. log of FEvals / DIM. + f- - -. f- - - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / of trials. log of FEvals / DIM. + f- - -. f- - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / inria-, version - Jul moderate fcts ill-conditioned fcts multi-modal fcts weak structure fcts of trials of trials of trials of trials. log of FEvals / DIM. + f-9 - -.. log of FEvals / DIM. + f- - -.. log of FEvals / DIM. + f-9 - -.. log of FEvals / DIM. + f- - -.. log of FEvals / DIM f- - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f-9 - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f-9 - - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - - log of FEvals(A)/FEvals(A) of trials of trials of trials of trials. log of FEvals / DIM. + f-9 - -.. log of FEvals / DIM. + f- - -.. log of FEvals / DIM. + f-9 - -.. log of FEvals / DIM. + f- - -.. log of FEvals / DIM f- - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f-9 - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f-9 - - - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - - log of FEvals(A)/FEvals(A) Figure : Empirical cumulative distributions of run lengths of (, s m)-cma-es (solid) and (, m)-cma-es (dashed) and speed-up ratios of the former divided by the latter in -D (left) and -D (right) as in Fig.. [] A. Auger, S. Finck, N. Hansen, and R. Ros. BBOB 9: Comparison tables of all algorithms on all noiseless functions. Technical Report RT, INRIA, April. [] A. Auger and N. Hansen. Performance evaluation of an advanced local search evolutionary algorithm. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC ), pages 8,. [] S. Finck, N. Hansen, R. Ros, and A. Auger. Real-parameter
inria-, version - Jul f e+ e+ e- e- e- e- #succ f / (,m)-cma-es.. 9 / (, s m )-CMA-ES 8. / f 8 8 88 9 9 9 / (,m)-cma-es 9 / (, s m )-CMA-ES 8 8 / f / (,m)-cma-es. 9 / (, s m )-CMA-ES.8 98.e / f 8 8 9 9 / (,m)-cma-es..e / (, s m )-CMA-ES.9.e / f / (,m)-cma-es...8.8.8.8 / (, s m )-CMA-ES...... / f 8 8 / (,m)-cma-es.8....9 / (, s m )-CMA-ES....9..9 / f / (,m)-cma-es... 8 / (, s m )-CMA-ES..9.9 / f 8 9 / (,m)-cma-es.9...8. / (, s m )-CMA-ES..9...8 / f 9 / (,m)-cma-es. 9. 8... / (, s m )-CMA-ES....8 / f 8 88 / (,m)-cma-es...... / (, s m )-CMA-ES.... / f / (,m)-cma-es 9.8.8.. / (, s m )-CMA-ES 9.8.9.... / f / (,m)-cma-es.. / (, s m )-CMA-ES 9. 8.9 8... / f 9 8 / (,m)-cma-es. 9.8.8..9 / (, s m )-CMA-ES.9.9.... / f 9.8 8 8 / (,m)-cma-es...9.9. / (, s m )-CMA-ES..8..9. / f 9.9e.e.e.e / (,m)-cma-es. 8.e / (, s m )-CMA-ES.8 8 / f.e.e.e / (,m)-cma-es.9.8.e / (, s m )-CMA-ES. 9.e / f. 9 9 / (,m)-cma-es.. 9.e / (, s m )-CMA-ES....e / f 8 8 9.e.e / (,m)-cma-es.9.9 9..e / (, s m )-CMA-ES. 8..e / f 9.e.e.e / (,m)-cma-es 9.e.e / (, s m )-CMA-ES.e 8. / f 8.8e.e.e.e / (,m)-cma-es.8.9.e / (, s m )-CMA-ES...e / f 8 / (,m)-cma-es.... / (, s m )-CMA-ES...... / f 9 9 / (,m)-cma-es / (, s m )-CMA-ES / f.e.e.e.e / (,m)-cma-es.8..e / (, s m )-CMA-ES. 8.9.e / f.e.e 9.e.e.e / (,m)-cma-es..e / (, s m )-CMA-ES..e / -D -D f e+ e+ e- e- e- e- #succ f / (,m)-cma-es. / (, s m )-CMA-ES. 8. 8 / f 8 9 9 9 9 9 / (,m)-cma-es 8 / (, s m )-CMA-ES 9 / f / (,m)-cma-es.e / (, s m )-CMA-ES.e / f 8.e 9/ (,m)-cma-es.e / (, s m )-CMA-ES.e / f / (,m)-cma-es...... / (, s m )-CMA-ES...... / f 8 / (,m)-cma-es....8. / (, s m )-CMA-ES..... / f 9.e.e.e / (,m)-cma-es.e / (, s m )-CMA-ES.e / f 8 9 / (,m)-cma-es..8... / (, s m )-CMA-ES..8.9.9.8.8 / f 9 / (,m)-cma-es...... / (, s m )-CMA-ES...... / f 8.e.e.e.e / (,m)-cma-es...9... / (, s m )-CMA-ES.8....98.98 / f 98.e.e / (,m)-cma-es.8.9..8. / (, s m )-CMA-ES..... / f 9.e.e / (,m)-cma-es...... / (, s m )-CMA-ES.8...8.8 / f 8.9e.e.e / (,m)-cma-es. 9.8 8.e / (, s m )-CMA-ES...e / f 9.e / (,m)-cma-es.8.... / (, s m )-CMA-ES.8.... / f.e.e.e.e.e.e / (,m)-cma-es.e / (, s m )-CMA-ES.e / f.e.e.9e.e.e / (,m)-cma-es.e / (, s m )-CMA-ES 9.e / f.e.e 8.e / (,m)-cma-es..e / (, s m )-CMA-ES..e.e / f 8.e.8e.e.e / (,m)-cma-es.e / (, s m )-CMA-ES.e / f 9.e.e.e.e / (,m)-cma-es.e.e / (, s m )-CMA-ES.8e.e / f 8.e.e.e.e.e / (,m)-cma-es..e / (, s m )-CMA-ES..e / f.e.e.e.8e / (,m)-cma-es.9..8...9 / (, s m )-CMA-ES...8... / f.e.e.e.e / (,m)-cma-es...9 / (, s m )-CMA-ES 9 8.9 / f..e.9e 8.e 8.e / (,m)-cma-es 9.e / (, s m )-CMA-ES.e / f.e.e.e.e.e.e / (,m)-cma-es.e / (, s m )-CMA-ES.e / Table : Expected running time (ERT in number of function evaluations) divided by the best ERT measured during BBOB-9 as in Table but now comparing (, m)-cma-es and (, s m)-cma-es. black-box optimization benchmarking 9: Presentation of the noiseless functions. Technical Report 9/, Research Center PPE, 9. Updated February. [] N. Hansen. Benchmarking a BI-population CMA-ES on the BBOB-9 function testbed. In F. Rothlauf, editor, GECCO (Companion), pages 89 9. ACM, 9. [] N. Hansen, A. Auger, S. Finck, and R. Ros. Real-parameter black-box optimization benchmarking : Experimental setup. Technical Report RR-, INRIA,. [] N. Hansen, S. Finck, R. Ros, and A. Auger. Real-parameter black-box optimization benchmarking 9: Noiseless functions definitions. Technical Report RR9, INRIA, 9. Updated February. [8] N. Hansen and S. Kern. Evaluating the CMA evolution strategy on multimodal test functions. In X. Yao et al., editors, Parallel Problem Solving from Nature PPSN VIII, volume of LNCS, pages 8 9. Springer,. [9] N. Hansen, S. D. Müller, and P. Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation. Evolutionary Computation, (): 8,. [] N. Hansen and A. Ostermeier. Completely derandomized self-adaptation in evolution strategies. Evolutionary Computation, 9():9 9,. [] K. Price. Differential evolution vs. the functions of the second ICEO. In Proceedings of the IEEE International Congress on Evolutionary Computation, pages, 99.