Benchmarking a Weighted Negative Covariance Matrix Update on the BBOB-2010 Noiseless Testbed

Transcription

1 Benchmarking a Weighted Negative Covariance Matrix Update on the BBOB- Noiseless Testbed Nikolaus Hansen, Raymond Ros To cite this version: Nikolaus Hansen, Raymond Ros. Benchmarking a Weighted Negative Covariance Matrix Update on the BBOB- Noiseless Testbed. Genetic And Evolutionary Computation Conference, Jul, Portland, United States. pp.,, <./.>. <halv> HAL Id: hal Submitted on Dec (v), last revised Dec (v) HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

2 Benchmarking a Weighted Negative Covariance Matrix Update on the BBOB- Noiseless Testbed ABSTRACT Nikolaus Hansen INRIA Saclay, TAO team LRI, Bat 9 Univ. Paris-Sud 9 Orsay Cedex, France nikolaus.hansen@inria.fr We implement a weighted negative update of the covariance matrix in the CMA-ES weighted active CMA-ES or, in short, acma-es. We benchmark the IPOP-aCMA-ES and compare the performance with the IPOP-CMA-ES on the BBOB- noiseless testbed in dimensions between and. On nine out of essentially unimodal functions, the acma is faster than CMA, in particular in larger dimension. On at least three functions it also leads to a (slightly) better scaling with the dimension. In none of the benchmark functions acma appears to be significantly worse in any dimension. On two and five functions, IPOP-CMA- ES and IPOP-aCMA-ES respectively exceed the record observed during BBOB-9. Categories and Subject Descriptors G.. [Numerical Analysis]: Optimization global optimization, unconstrained optimization; F.. [Analysis of Algorithms and Problem Complexity]: Numerical Algorithms and Problems General Terms Algorithms, performance, comparison Keywords CMA-ES, IPOP-CMA-ES, active CMA-ES, Benchmarking, Black-box optimization. INTRODUCTION The Covariance Matrix Adaptation Evolution Strategy (CMA-ES) [,, ] is a stochastic search procedure that samples new candidate solutions from a multivariate normal distribution thereof mean and covariance matrix are adapted after each iteration. The (µ/µ w, λ)-cma-es samples λ new candidate solutions and selects the µ best among them. They contribute in a weighted manner to the update of the distribution parameters. The algorithm is non-elitist Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. GECCO, July,, Portland, Oregon, USA. Copyright ACM 9---//...$.. Raymond Ros TAO Team-Project INRIA Saclay LRI, Bat 9, Univ. Paris-Sud F-9 Orsay Cedex, France raymond.ros@inria.fr by nature, but a practical implementation will preserve the best-ever evaluated solution. Elitist variants of the CMA-ES [] are often slightly faster but more susceptible of getting stuck in a suboptimal local optimum. The IPOP-CMA-ES [] implements a restart procedure. Before each restart, λ is doubled. In most cases, doubling λ increases the length of single runs, but it often improves the quality of the best found solution. The BIPOP-CMA-ES [], proposed recently, maintains two budgets. Under the first budget, an IPOP-CMA-ES is executed. Under the second budget, a multi-start (µ/µ w, λ)-cma-es with various small population sizes is entertained. Previous Benchmarking of CMA-ES. The (µ/µ w, λ)- CMA-ES has been quite successful in several elaborate benchmarking exercises. Notably, the IPOP-CMA-ES [] in the special session on real parameter optimization CEC- and the BIPOP-CMA-ES [] in the black-box optimization benchmarking BBOB-9 have shown excellent performance [, ] on respectively and uni- and multi-modal benchmark functions in search space dimension up to. A comparison of two Estimation of Distribution Algorithms and three Evolution Strategies is presented in [] on functions. The CMA-ES performed best on functions with a median speed-up by a factor of at least compared to any other algorithm. On the remaining functions the maximum loss factor was.. A comparison on a small collection of multi-modal function with three other algorithms is presented in []. The CMA-ES with optimal population size performs clearly best on the non-separable functions and is clearly outperformed on additively decomposable functions by Differential Evolution (DE). A comparison of derivative-free optimization methods and BFGS on smooth unimodal functions is presented in []. On these functions, PSO and DE are in general significantly slower than CMA-ES. Only PSO performs similar on separable problems. NEWUOA and BFGS remarkably outperform CMA-ES, if the problem is convex and has a moderate conditioning. On non-convex problems with at least moderate condition number (i.e. ) and on non-separable problems with higher condition number (i.e. ), the CMA-ES breaks even and becomes advantageous with increasing condition number, tested up to. In [] the CMA-ES is recognized as state-of-the-art in CEC-/CEC.htm

3 Given t N {}, m t R D, σ t R +, C t R D D positive definite, p t σ,p t c R D and p t= σ = p t= c =, C t= = I x i m t + σ t N,C t is normally distributed for i =,..., λ () X µ m t+ = m t + c m w i (x i:λ m t ) where f(x :λ ) f(x µ:λ ) f(x µ+:λ )... () p t+ σ h σ = p t+ c C + µ = C µ = i= = ( c σ)p t σ + p c σ( c σ)µ w C t mt+ m t c m σ t ( if p t+ σ < p ( c σ) (t+). + D+ otherwise = ( c c)p t c + h σ p cc( c c)µ w m t+ m t µx i= C t is symmetric with positive eigenvalues such that C t C t is the inverse of C t. If BΛB T = C is an eigendecomposition into an orthogonal matrix B and a diagonal matrix Λ, we have C = BΛ / B T. E N (,I) stalls the update of p c in () when σ increases rapidely c m σ t evolution path for the rank-one update of C t () x i:λ m t `xi:λ m t T w i matrix with rank min(µ, D) for the so-called rank-µ update of C t () σ t σ t µ X w i+y λ i:λ yλ i:λ T i= C t+ = ( c c µ + c α old )C t + c p t+ σ t+ = σ t exp max σ cσ d σ with y λ i:λ = Ct (x λ µ++i:λ m t ) C t (xλ i:λ m t ) c p t+ T c {z } rank one update ««p t+ σ E N (,I) + (c µ + c ( α old)) C + µ () () x λ i:λ m t σ t () {z} rank-µ update c C µ {z } active for E N (,I) = Γ( n+ )/Γ( n ) we use the approximation D ` + (9) D D Figure : Update equations for the state variables in the (µ/µ w, λ)-acma-es with iteration index t =,,,... and a bc+d = min(a, bc+d). The chosen ordering of equations allows to remove the time index in all variables but m t. The symbol x i:λ is the i-th best of the solutions x,...,x λ () real-valued evolutionary optimization. Partly due to the fact that the algorithm is quasi-parameter-free, the CMA-ES is widely applied and has become a quasi-standard. A Further Improvement. The active CMA-ES proposed in [] introduces a negative update of the covariance matrix in the (µ/µ I, λ)-cma-es. The authors investigate essentially unimodal functions. They observe a significant speed-up in particular on the discus function. The speed-up reaches almost a factor of three in dimension (compared to the (µ/µ I, λ)-cma-es) and it increases with increasing dimension (i.e. the update leads to an improved scaling with increasing search space dimension). The negative update can in particular speed-up the adaptation of small variances in a small number of directions. The speed-up is less pronounced with a larger population size λ. This Paper. We combine the technique of negative updates with the (µ/µ w, λ)-cma-es and implement a weighted negative update procedure for the covariance matrix. The update also improves compared to [], in that it remains feasible with a large population size. Consequently, the objective of this paper is threefold. ) Specify a weighted negative update scheme with a parameter setting suited also for a very large population size. ) Reproduce the effects from [], observing significant improvements on functions, where a small number of directions need a small variance (here, we want strategies#covariance_matrix_adaptation_evolution_ Strategies_CMA-ESs to improve over the (µ/µ w, λ)-cma-es). ) Survey the performance of a negative update on a larger number of diverse functions to possibly detect failures of the method. For this survey the BBOB- test environment is used.. THE ALGORITHM The (µ/µ w, λ)-acma-es is summarized in Figure. The initial values m t= R D and σ t= > are user defined and given in the next section. Modifications compared to [, ] are the addition of () and otherwise highlighted in pink (mainly the supplement to ()). The default parameter values are shown in Table. For c =, the (µ/µ w, λ)-cma-es is recovered. Restarts. We apply nine restarts each with maximal + (D + ) / λ iterations. The default termination methods are used, besides that we have set TolHistFun=e-, TolX=e-, StopOnStagnation= on and MaxFunEvals=inf, following []. The population size λ is doubled for each restart, the first value is given in Table.. PARAMETER TUNING AND SETUP The new parameter(s) for acma have been identified with experiments on the sphere function f with various initial covariance matrices. In particular c is chosen such that (a) decreasing c from the default value does not improve the

4 Table : Default parameter values of (µ/µ w, λ)-acma-es, where by definition P µ i= wi = and µ w = P µ i= w i and a b c + d := min(a b, c + d). Only population size λ is possibly left to the users choice (see also [, ]) λ = + ln D population size, number of newly sampled candidate solutions in each iteration µ = λ parent number, number of candidate solution used to update the distribution parameters w i = ` ln λ+ ln i P µ ` j= ln λ+ ln j «recombination weights for i =,..., µ λ/ c m = learning rate for the mean, sometimes interpreted as rescaled mutation with κ = c m µ c σ = w+ D+µ w+ cumulation constant for step-size q µ d σ = + c σ + max, w step-size damping, is usually close to one. This formula might be replaced D+ c c = c = + µ w/d D++ µ w/d αcov min(,λ/) (D+.) +µ w by µ w/λ +. + c σ in the near future cumulation constant for p c, the might be removed in near future covariance matrix learning rate for the rank one update using p c µ c µ = c α w +/µ w cov (D+) +α covµ w/ covariance matrix learning rate for rank-µ update c = c αcov µ min ( cµ) w (D+). +µ w remark that c becomes small, if c µ gets close to one α cov = could be chosen <, e.g. α cov =. for noisy problems with α c min = α ( c c µ)( λ mintarget ) min =, λmintarget =. and λmax(.) is the largest eigenvalue; the min coefficient c λ max(c t / C µ C t / ) min depends in general on the iteration index, but here, due to the setting of α min, no upper bound on c is enforced α old =. is in [, ] and is chosen in the domain middle without deep motives, because it seems quite irrelevant σ = might become in near future, which has only a negligible effect under neutral selection, because the term to the right of the in (9) is approximately cσ d σ N (, /D) under neutral selection max performance, (b) increasing c by a factor of two never leads to a failure, and (c) the condition number of C t remains below in the stationary limit. The remaining parameters are used in their default settings as found in the source code on the WWW or in []. No special attempt has been made to further tune the parameters to the BBOB testbed. The crafting-effort [] of both algorithms is equal to CrE =. Experiments were conducted according to BBOB- [] on the benchmark functions given in [, 9]. On each function, trials are executed (on the first instances) with m uniformly random in [, ] D and σ =. Source code to reproduce the experiment is provided at. CPU Timing Experiment. The complete algorithm was run on f for at least seconds on a Intel Core processor (. GHz) with Linux.. and Matlab Ra. Results for the IPOP-aCMA-ES are.;.;.;.;.;. and. seconds per function evaluation for dimension ; ; ; ; ; and.. RESULTS We show results for two algorithms: the IPOP-aCMA-ES as presented above and denoted as acma, and the IPOP- CMA-ES, the same algorithm where c is set to zero, denoted as CMA. In exploratory experiments comparing both (µ/µ w, λ)-cma-es and (µ/µ w, λ)-acma-es with the result of the (µ/µ I, λ)-variants from [], the (µ/µ w, λ)-variants perform at least as good and usually better: as a rule, (µ/µ w, λ)-acma-es outperforms (µ/µ I, λ)-acma-es that outperforms (µ/µ w, λ)-cma-es that outperforms (µ/µ I, λ)- CMA-ES. Runtime results comparing acma with CMA and with the respective best algorithm from BBOB-9 are presented bbob--results log of ERT loss ratio - - IPOP-CMA-ES CrE = f- log of FEvals / dimension log of ERT loss ratio - - IPOP-aCMA-ES CrE = f- log of FEvals / dimension Figure : Box-Whisker plots of the ERT ratio to the respective best algorithm from BBOB-9 for all functions in dimension in Figures and in Table. The expected running time (ERT), used in the figures and table, depends on a given target function value, f t = f opt+ f, and is computed over all relevant trials as the number of function evaluations executed during each trial while the best function value did not reach f t, summed over all trials and divided by the number of trials that actually reached f t [, ]. Statistical significance is tested with the rank-sum test for a given target f t ( in Figure ) up to the smallest number of function evaluations, b u min, executed in any unsuccessful trial under consideration. The datum from each trial is either the best achieved f-value, or if f t was reached within the budget b u min, the number of needed function evaluations to reach f t (inverted and multiplied by ), In the following, when a performance difference is highlighted on an individual function, the difference is statistically significant. Figure. The figure shows Box-Whisker plots of the ERT ratio compared to the respective best algorith from BBOB-

5 9 for all functions. Both algorithms show a very similar characteristic. Besides for very small budgets, the median (horizontal red line) and average (connected line) ERT ratio are somewhat below ten. In the final stage, acma improves in about % of the functions compared to the best algorithm from 9 (ratio smaller than one). Figure. The figure shows empirical cumulative distributions (a) of the runtime in number of function evaluations and (b) of the runtime ratio between the two algorithms acma/cma. Both algorithms perform very similar while acma appears to be slightly faster. Clearly, the strongest effect in Fig. is observed for the ill-conditioned functions in dimension. The runtime is consistently shorter for acma, almost uniformly by a factor of close to two. The improvement is statistically significant on all functions (see Table ). On the weak structure functions, the observed differences are probably caused by stochastic deviations and not significant. The remaining subgroups show almost identical behavior. Figure. The scatter plots in Fig. visualize the ratio of expected runtime acma/cma for each measurement on each function and each dimension. A slightly improved scaling with acma can be observed on f, f and f. On the Discus function f the effect is most pronounced and the speed-up comes close to a factor of three in dimension. The advantage of CMA on f in dimension is far beyond the maximum number of function evaluations, based on two successes (see Fig. ) and not statistically significant. Additionally, on functions f, f f, and f an advantage by acma, in particular for larger dimension, could be conjectured. On f an advantage of CMA could be conjectured. According to Table, in dimension, statistical significance is established for f and f f. Figure. The questions of a significant performance difference can be pursued in Fig. which plots the ERT ratio versus the target function value. The figure reveals more statistically significant (but sometimes small) differences in performance. A non-negligible and statistically significant performance advantage of acma in some dimensions larger than can be found on f, and f f. No such advantage is detected for CMA. Table. Finally, Table presents the ERT numbers for dimension and in comparison with the respective best algorithm of BBOB-9. Again, acma is significantly better than CMA on f, and f f in dimension as well as in dimension apart from f and f. Compared to the best algorithm from BBOB-9, chosen for each function respectively, the acma can improve the record in dimension on f, f, f, f and f 9. On further four functions acma is visibly better, but the results appear not to be statistically significant. The CMA improves the record on f and f 9.. SUMMARY AND CONCLUSION We have introduced the weighted negative update of the covariance matrix into the (µ/µ w, λ)-cma-es with weighted recombination, denoted as (µ/µ w, λ)-acma-es. The negative update is based in the idea of []. It provides a similar effect for the (µ/µ w, λ)-cma-es as [] for the (µ/µ I, λ)- CMA-ES. The benchmarking of the new IPOP-aCMA-ES with BBOB- reveals no deficiencies or failures compared to the IPOP-CMA-ES on the testbed of functions in any dimension between and. The most prominent effect from acma is observed on ill-conditioned functions. In dimension, the average runtime advantage on the ill-conditioned functions is about. (CMA is. times slower than acma). On three ill-conditioned functions the scaling behavior compared to the CMA improves notably. In no case CMA outperformed acma with statistical significance. Overall, the acma improves the performance over CMA on nine out of essentially unimodal functions significantly at least in dimension, and the advantages are more pronounced in larger dimension. Finally, IPOP-aCMA-ES improves the record on five functions, where it is faster than the respective best algorithm from BBOB-9 in dimension with a sufficient statistical significance. In two of these cases IPOP-CMA-ES is still slightly faster and the record cannot be attributed to the negative covariance matrix update. Acknowledgments NH likes to thank M. Schoenauer for his support of BBOB.. REFERENCES [] A. Auger and N. Hansen. A restart CMA evolution strategy with increasing population size. In Proceedings of the IEEE Congress on Evolutionary Computation (CEC ), pages 9. IEEE Press,. [] A. Auger, N. Hansen, J. Perez Zerpa, R. Ros, and M. Schoenauer. Experimental comparisons of derivative free optimization algorithms. In J. Vahrenhold, editor, th International Symposium on Experimental Algorithms, volume of LNCS, pages. Springer, 9. [] H.-G. Beyer. Evolution strategies. Scholarpedia, ():9,. [] S. Finck, N. Hansen, R. Ros, and A. Auger. Real-parameter black-box optimization benchmarking 9: Presentation of the noiseless functions. Technical Report 9/, Research Center PPE, 9. Updated February. [] S. García, D. Molina, M. Lozano, and F. Herrera. A study on the use of non-parametric tests for analyzing the evolutionary algorithms behaviour: a case study on the CEC special session on real parameter optimization. Journal of Heuristics, ():, 9. [] N. Hansen. Benchmarking a BI-population CMA-ES on the BBOB-9 function testbed. In Rothlauf [], pages 9 9. [] N. Hansen, A. Auger, S. Finck, and R. Ros. Real-parameter black-box optimization benchmarking : Experimental setup. Technical Report RR, INRIA,. [] N. Hansen, A. Auger, R. Ros, S. Finck, and P. Pošík. Comparing results of algorithms from the black-box optimization benchmarking BBOB-9. In J. Branke, editor, GECCO (Companion). ACM,. [9] N. Hansen, S. Finck, R. Ros, and A. Auger. Real-parameter black-box optimization benchmarking 9: Noiseless functions definitions. Technical Report RR9, INRIA, 9. Updated February. [] N. Hansen and S. Kern. Evaluating the CMA evolution strategy on multimodal test functions. In X. Yao et al., editors, Parallel Problem Solving from Nature PPSN VIII, volume of LNCS, pages 9. Springer,. [] N. Hansen, S. Muller, and P. Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evolutionary Computation, ():,. [] N. Hansen and A. Ostermeier. Completely derandomized

6 . + f- -D -D. +: / + f- +: / - -: / - -: 9/9 all functions. - -: / : /. - -: 9/9 : 9/9. log of FEvals / DIM. + f- f log of FEvals(A)/FEvals(A) +: /. log of FEvals / DIM. + f- f log of FEvals(A)/FEvals(A) +: / separable fcts : / -: / : / : / -: / : /. log of FEvals / DIM. + f-9 f- - - log of FEvals(A)/FEvals(A) +: /. log of FEvals / DIM. + f-9 f- - log of FEvals(A)/FEvals(A) +: / - -: / - -: / moderate fcts. - -: / : /. - -: / : / ill-conditioned fcts. log of FEvals / DIM. + f log of FEvals / DIM. + f-9 f log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - log of FEvals(A)/FEvals(A) +: /. log of FEvals / DIM. + f log of FEvals / DIM. + f-9 f-9 - log of FEvals(A)/FEvals(A) +: / -: / -: / : / f- - - log of FEvals(A)/FEvals(A) +: / multi-modal fcts weak structure fcts log of FEvals / DIM. + f log of FEvals / DIM -: / -: / : / f log of FEvals(A)/FEvals(A) +: / -: / -: / : / f log of FEvals(A)/FEvals(A) log of FEvals / DIM. + f log of FEvals / DIM -: / -: / : / f log of FEvals(A)/FEvals(A) +: / -: / -: / : / f log of FEvals(A)/FEvals(A) Figure : Empirical cumulative distributions (ECDF) of run lengths and speed-up ratios in -D (left) and - D (right). Left sub-columns: ECDF of the number of function evaluations divided by dimension D (FEvals/D) to reach a target value f opt + f with f = k, where k {,,, } is given by the first value in the legend, for IPOP-aCMA (solid) and IPOP-CMA (dashed). Light beige lines show the ECDF of FEvals for target value f = of algorithms benchmarked during BBOB-9. Right sub-columns: ECDF of FEval ratios of IPOP-aCMA divided by IPOP-CMA, all trial pairs for each function. Pairs where both trials failed are disregarded, pairs where one trial failed are visible in the limits being > or <. The legends indicate the number of functions that were solved in at least one trial (IPOP-aCMA first).

7 Sphere Ellipsoid separable Rastrigin separable Skew Rastrigin-Bueche Linear slope Attractive sector Step-ellipsoid Rosenbrock original 9 Rosenbrock rotated Ellipsoid Discus Bent cigar Schaffer F, cond. Sharp ridge Schaffer F, cond. Sum of diff. powers Rastrigin 9 Griewank-Rosenbrock Schwefel x*sin(x) Weierstrass Gallagher peaks Gallagher peaks Katsuuras Lunacek bi-rastrigin Figure : Expected running time (ERT in log of number of function evaluations) of IPOP-aCMA versus IPOP-CMA for target values f [, ] in each dimension for functions f f. Markers on the upper or right egde indicate that the target value was never reached by IPOP-aCMA or IPOP-CMA respectively. Markers represent dimension: :+, :, :, :, :, :. Lines indicate the maximum number of function evaluations

8 log(ert/ert) or ~#succ Sphere -D -D -D -D -D -D D D D D D D Ellipsoid separable D D D D D D Rastrigin separable / D log(ert/ert) or ~#succ D D / / Skew Rastrigin-Bueche separable D D / D D D Linear slope Attractive sector Step-ellipsoid Rosenbrock original log(ert/ert) or ~#succ log(ert/ert) or ~#succ D D D D D D log(ert/ert) or ~#succ D D D D D D log(ert/ert) or ~#succ D D D D D D D D D D D D log(delta ftarget) 9 Rosenbrock rotated log(delta ftarget) Ellipsoid log(delta ftarget) Discus log(delta ftarget) Bent cigar log(ert/ert) or ~#succ log(ert/ert) or ~#succ D D D D D D log(ert/ert) or ~#succ D D D D D D log(ert/ert) or ~#succ D D D D D D D D D D D D log(delta ftarget) Sharp ridge log(delta ftarget) Sum of different powers log(delta ftarget) Rastrigin log(delta ftarget) Weierstrass log(ert/ert) or ~#succ log(ert/ert) or ~#succ D D D D D D log(ert/ert) or ~#succ D D D D D D log(ert/ert) or ~#succ D D D D D D D D D D D D log(delta ftarget) Schaffer F, condition log(delta ftarget) Schaffer F, condition log(delta ftarget) 9 Griewank-Rosenbrock FF log(delta ftarget) Schwefel x*sin(x) log(ert/ert) or ~#succ log(ert/ert) or ~#succ D D D D D D log(ert/ert) or ~#succ D D D D D D log(ert/ert) or ~#succ D D D D D D D D D D D D log(delta ftarget) Gallagher peaks log(delta ftarget) Gallagher peaks log(delta ftarget) Katsuuras log(delta ftarget) Lunacek bi-rastrigin log(ert/ert) or ~#succ - log(ert/ert) or ~#succ D / D D D 9/ / / log(ert/ert) or ~#succ 9/ / D D - / / log(ert/ert) or ~#succ / D D D / D D -D -D -D -D -D -D D D D log(delta ftarget) log(delta ftarget) log(delta ftarget) log(delta ftarget) Figure : ERT ratio of IPOP-aCMA divided by IPOP-CMA versus log ( f) for f f in,,,,, -D. Ratios < indicate an advantage of IPOP-aCMA, smaller values are always better. The line gets dashed when for any algorithm the ERT exceeds thrice the median of the trial-wise overall number of f- evaluations for the same algorithm on this function. Symbols indicate the best achieved f-value of one algorithm (ERT gets undefined to the right). The dashed line continues as the fraction of successful trials of the other algorithm, where means % and the y-axis limits mean %, values below zero for IPOP-aCMA. The line ends when no algorithm reaches f anymore. The number of successful trials is given, only if it was in {... 9} for IPOP-aCMA (st number) and non-zero for IPOP-CMA (nd number). Results are significant with p =. for one star and p = # otherwise, with Bonferroni correction within each figure.

9 f e+ e+ e- e- e- e #succ f / : IPOP-CMA.. 9 / : IPOP-aCMA..9 9 / f / : IPOP-CMA 9 / : IPOP-aCMA / f / : IPOP-CMA. 99 / : IPOP-aCMA. 9 / f 9 9 / : IPOP-CMA..e / : IPOP-aCMA..e / f / : IPOP-CMA / : IPOP-aCMA / f / : IPOP-CMA / : IPOP-aCMA / f 9 / : IPOP-CMA / : IPOP-aCMA / f 9 / : IPOP-CMA / : IPOP-aCMA / f 9 9 / : IPOP-CMA..... / : IPOP-aCMA / f 9 9 / : IPOP-CMA / : IPOP-aCMA / f / : IPOP-CMA / : IPOP-aCMA / f 9 / : IPOP-CMA / : IPOP-aCMA / f 9 / : IPOP-CMA / : IPOP-aCMA / f 9 / : IPOP-CMA / : IPOP-aCMA / f / : IPOP-CMA / : IPOP-aCMA / f 9 9 / : IPOP-CMA / : IPOP-aCMA / f / : IPOP-CMA / : IPOP-aCMA / f / : IPOP-CMA / : IPOP-aCMA / f 9.e.e.e / : IPOP-CMA... / : IPOP-aCMA / f / : IPOP-CMA / : IPOP-aCMA / f 9 / : IPOP-CMA.. / : IPOP-aCMA.. / f 9 / : IPOP-CMA / : IPOP-aCMA. 9/ f. 9 / : IPOP-CMA. / : IPOP-aCMA. / f.e.e 9.e.e.e / : IPOP-CMA / : IPOP-aCMA..e / -D -D f e+ e+ e- e- e- e #succ f / : IPOP-CMA. / : IPOP-aCMA.9 / f / : IPOP-CMA / : IPOP-aCMA 9 / f / : IPOP-CMA.9e / : IPOP-aCMA.9e / f.e 9/ : IPOP-CMA.e / : IPOP-aCMA.e / f / : IPOP-CMA / : IPOP-aCMA / f 9 9 / : IPOP-CMA / : IPOP-aCMA / f 9 99 / : IPOP-CMA / : IPOP-aCMA / f 9 9 / : IPOP-CMA / : IPOP-aCMA / f 9 9 / : IPOP-CMA / : IPOP-aCMA / f 9 / : IPOP-CMA / : IPOP-aCMA / f 9 / : IPOP-CMA..... / : IPOP-aCMA / f 9 / : IPOP-CMA / : IPOP-aCMA / f 9 / : IPOP-CMA / : IPOP-aCMA / f 9 9 / : IPOP-CMA / : IPOP-aCMA / f.e.e.e.9e.9e / : IPOP-CMA / : IPOP-aCMA / f.e.9e.e / : IPOP-CMA / : IPOP-aCMA / f / : IPOP-CMA / : IPOP-aCMA / f e.e / : IPOP-CMA / : IPOP-aCMA / f 9.e.e.9e.e / : IPOP-CMA.... / : IPOP-aCMA / f.e.e.9e.e / : IPOP-CMA / : IPOP-aCMA / f 9 / : IPOP-CMA. 9 / : IPOP-aCMA.9 9/ f 9 9.e / : IPOP-CMA.e / : IPOP-aCMA.e / f..9e.e.e / : IPOP-CMA..e / : IPOP-aCMA. 9.e / f.e.e.9e.e.e.e / : IPOP-CMA.e / : IPOP-aCMA..e / Table : Expected running time (ERT in number of function evaluations) divided by the best ERT measured during BBOB-9 (given in the respective first row) for different f values for functions f f. The median number of conducted function evaluations is additionally given in italics, if ERT( ) =. #succ is the number of trials that reached the final target f opt +. Bold entries are statistically significantly better compared to the other algorithm, with p =. or p = k where k > is the number following the symbol, with Bonferroni correction of. self-adaptation in evolution strategies. Evolutionary Computation, 9():9 9,. [] C. Igel, T. Suttorp, and N. Hansen. A computational efficient covariance matrix update and a (+)-CMA for evolution strategies. In Proceedings of the th annual conference on genetic and evolutionary computation GECCO, pages. ACM,. [] G. Jastrebski and D. Arnold. Improving evolution strategies through active covariance matrix adaptation. In IEEE Congress on Evolutionary Computation, CEC, Proceedings, pages. IEEE,. [] S. Kern, S. Müller, N. Hansen, D. Büche, J. Ocenasek, and P. Koumoutsakos. Learning probability distributions in continuous evolutionary algorithms a comparative review. Natural Computing, ():,. [] K. Price. Differential evolution vs. the functions of the second ICEO. In Proc. of the IEEE International Congress on Evolutionary Computation, pages, 99. [] F. Rothlauf, editor. Genetic and Evolutionary Computation Conference, GECCO 9, Proceedings, Montreal, Quعebec, Canada, July -, 9, Companion Material. ACM, 9.