Bio-inspired Continuous Optimization: The Coming of Age
|
|
- Leo Blankenship
- 6 years ago
- Views:
Transcription
1 Bio-inspired Continuous Optimization: The Coming of Age Anne Auger Nikolaus Hansen Nikolas Mauny Raymond Ros Marc Schoenauer TAO Team, INRIA Futurs, FRANCE CEC 27, Singapore, September 27., 27
2 Problem Statement Continuous Domain Search/Optimization The problem Minimize a fitness function (objective function, loss function) in continuous domain f : S R n R, in the Black Box scenario (direct search) x f(x) Hypotheses domain specific knowledge only used within the black box gradients are not available
3 Problem Statement Continuous Domain Search/Optimization The problem Minimize a fitness function (objective function, loss function) in continuous domain f : S R n R, in the Black Box scenario (direct search) x f(x) Typical Examples shape optimization (e.g. using CFD) curve fitting, airfoils model calibration biological, physical parameter identification controller, plants, images
4 Optimization techniques Numerical methods Applied Mathematicians but Heavily rely on theoretical convergence proofs Requires regularity Numerical pitfalls Long history numerical gradient? Bio-inspired algorithms Computer Scientists Recent trendy methods but Computationally heavy No convergence proof mostly from AI field but divided! well, almost see later
5 This talk Goal Empirical comparison on some artificial testbed illustrating typical difficulties of continuous optimization between some bio-inspired algorithms and some (one!) deterministic optimization method(s) in the back-box scenario without specific intensive parameter tuning
6 1 Continuous optimization and stochastic search The algorithms 2 Problem difficulties Ruggedness Ill-Conditionning Non-separability 3 Implementations and parameter settings Algorithm implementations Tuning DE Experiments and results Experimental conditions Outputs and Performance measures 5 Conclusion Further work Conclusions
7 The algorithms Bio-inspired Optimization Algorithms Darwinian Artificial Evolution Repeat (Parent selection Variation Survival selection) Preselection: of CEC 5 Challenge Particle Swarm Optimization Eberhart & Kennedy, 95 Perturb particle velocity best and local best Update best and local best Differential Evolution Rainer and Storn, 95 Add difference vector(s) Uniform crossover Keep best of parent and offspring with proba. 1 CR Covariance Matrix Adaptation-ES Hansen & Ostermeier, 9 Gaussian mutation Keep λ 2 best of λ offspring + Update mutation parameters
8 The algorithms BFGS Gradient-based methods { xt+1 = x t ρ t d t ρ t = Argmin ρ {F(x t ρd t )} Line search Choice of d t, the descent direction? BFGS: a Quasi-Newton method Maintain an approximation Ĥ t of the Hessian of f Solve for d t Ĥ t d t = f (x t ) Compute x t+1 and update Ĥ t Ĥ t+1 Converges if quadratic approximation of F holds around the optimum Reliable and robust on quadratic functions!
9 1 Continuous optimization and stochastic search 2 Problem difficulties Ruggedness Ill-Conditionning Non-separability 3 Implementations and parameter settings Experiments and results 5 Conclusion
10 What makes a problem hard? Non-convexity invalidates most of deterministic theory Ruggedness non-smooth, discontinuous, noisy Multimodality presence of local optima Dimensionality line search is trivial The magnifiscence of high dimensionality Ill-conditioning Non-separability
11 Ruggedness Monotonous transformation invariance Monotonous transformations x 2 i x 2 i ( x 2 i ) 2 Invariance Comparison-based algorithms PSO, DE, CMA-ES,... are invariant w.r.t. monotonous transformations Gradient-based methods BFGS,... are not
12 Ruggedness Multimodality Bio-inspired algorithms BFGS are global search algorithms but performance on multi-modal problems depends on population size has no population :-) but starting point is crucial on multimodal functions Replace population by multiple restarts from uniformly distributed points or from the perturbed final point of previous trial the Hessian is anyway reset to I n
13 Ill-Conditionning Ill-Conditionning The Condition Number (CN) of a positive-definite matrix H is the ratio of its largest and smallest eigenvalues If f is quadratic, f (x) = x T Hx), the CN of f is that of its Hessian H More generally, the CN of f is that of its Hessian wherever it is defined. Graphically, ill-conditioned means squeezed lines of equal function value Increased condition number Issue: The gradient does not point toward the minimum...
14 Ill-Conditionning A priori discussion Bio-inspired algorithms PSO and DE: population can point toward the minimum CMA-ES: covariance matrix can take longer to learn BFGS Numerical gradient can raise numerical problems Hessian matrix can take longer to learn
15 Non-separability Separability Definition (Separable Problem) A function f is separable if ( arg min (x 1,...,x n) f (x 1,..., x n ) = arg min f (x 1,...),..., arg min f (..., x n ) x 1 x n solve n independent 1D optimization problems ) Example: Additively decomposable functions n f (x 1,..., x n ) = f i (x i ) i=1 e.g. Rastrigin function
16 Non-separability Designing Non-Separable Problems Rotating the coordinate system f : x f (x) separable f : x f (Rx) non-separable R rotation matrix Hansen, Ostermeier, & Gawelczyk, 95; Salomon, R
17 Non-separability Rotational invariance Bio-inspired algorithms PSO: is not rotational invariant see next slide DE: Crossover is not rotational invariant Rotational invariance iff CR = 1 CMA-ES: is rotational invariant BFGS Numerical gradient can raise numerical problems Added to ill-conditionning effects
18 Non-separability PSO and rotational invariance. A sample swarm Same swarm, rotated V j i (t+1) = Vj i (t)+ c 1 U j i (, 1)(pj i xj i }{{ (t)) + c 2 Ũ j i } (, 1)(gj i xj i }{{ (t)) } approach the "previous" best approach the "global" best
19 Non-separability PSO and rotational invariance. A sample swarm Same swarm, rotated V j i (t+1) = Vj i (t)+ c 1 U j i (, 1)(pj i xj i }{{ (t)) + c 2 Ũ j i } (, 1)(gj i xj i }{{ (t)) } approach the "previous" best approach the "global" best
20 Non-separability PSO and rotational invariance. A sample swarm Same swarm, rotated V j i (t+1) = Vj i (t)+ c 1 U j i (, 1)(pj i xj i }{{ (t)) + c 2 Ũ j i } (, 1)(gj i xj i }{{ (t)) } approach the "previous" best approach the "global" best
21 Non-separability PSO and rotational invariance. A sample swarm Same swarm, rotated V j i (t+1) = Vj i (t)+ c 1 U j i (, 1)(pj i xj i }{{ (t)) + c 2 Ũ j i } (, 1)(gj i xj i }{{ (t)) } approach the "previous" best approach the "global" best
22 1 Continuous optimization and stochastic search 2 Problem difficulties 3 Implementations and parameter settings Algorithm implementations Tuning DE Experiments and results 5 Conclusion
23 Algorithm implementations The algorithms Default implementations DE: Matlab code from storn/code.html Not really giving default parameters PSO: Std PSO 2, C code from Remark: C code Matlab code here CMA-ES: Matlab code from BFGS: Matlab built-in implementation widely blindely used using numerical gradient + multiple restarts local or global
24 Tuning DE The problem with DE Control parameters NP, F, CR, Stopping Criterion... and strategy to generate difference vector Perturb random or best Number of difference vectors 1 or 2 Slightly mutate perturbation All of the above :-) from public Matlab code 1 DE/rand/1 the basic algorithm 2 DE/local-to-best/1 3 DE/best/1 with jitter DE/rand/1 with per-vector-dither F = rand(f, 1) 5 DE/rand/1 with global-dither either-or-algorithm
25 Tuning DE DE tuning Experimental conditions Rotated ellipsoid, cond. number = Dimension = Stop when f elli < Strat=1 F=.3 F=.5 F=.7 F= CR Strat=3 log(#evals) F=.3 F=.5 F=.7 F=.9 Strat=2.2.. CR 7 Strat= Design of Experiments variants F = {.3,.5,.7,.9} CR = {.3,.5,.7,.9} NP = {3, 5, } dim 3 runs per setting :-( 8 runs F=.3 F=.5 F=.7 F= CR F=.3 F=.5 F=.7 F=.9 Strat= CR Poor DOE, yet totally unrealistic for RWA log(#evals) log(#evals) F=.3 F=.5 F=.7 F= CR 7 5 F=.3 F=.5 F=.7 F=.9 Strat=.2.. CR
26 Tuning DE DE Experimental Setting and Discussion Couldn t decide among 2 variants: DE2 Strategy 2 F =.8 CR = 1. NP = dim DE5 Strategy 5 F =. CR = 1. NP = dim Discussion DE2 very close to recommended parameters Except CR =.9, but Rotational invariance iff CR = 1 no crossover
27 1 Continuous optimization and stochastic search 2 Problem difficulties 3 Implementations and parameter settings Experiments and results Experimental conditions Outputs and Performance measures 5 Conclusion
28 Experimental conditions The parameters Common Parameters Default parameters except for DE MaxEval = 7 Fitness tolerance = 9 Domain: [ 2, 8] d Optimum not at the center 21 runs in each case except BFGS when little success Population size Standard values: for n =, 2, PSO: + floor(2 n) 1, 18, 22 CMA-ES: + floor(3 ln n), 12, 15 DE: n, 2, To be increased for multi-modal functions e.g. Rastrigin
29 Experimental conditions The testbed Test functions Ellipsoid function Rosenbrock function Rastrigin function DiffPow Ellipsoid and DiffPow quadratic both separable and rotated for different condition number almost unimodal both original (non-separable) and rotated for different condition number highly multi-modal both separable and rotated convex, but flat unimodal, but non-convex
30 Outputs and Performance measures Comparisons from CEC 25 challenge on continuous optimization Goals Find the best possible fitness value At minimal cost Number of function evaluations Statistical Measures Must take into account both precision and cost Usual averages and standard deviations of fitness values irrelevant to assess precision Issue: need to impose for obvious practical reasons precision threshold on fitness maximum number of iterations
31 Outputs and Performance measures SP1 measure SP1 Success Performance one Average required effort for success SP1 = avg # evaluations proportion successful runs Effort to reach a given precision on the fitness (success) Same number of total fitness evaluations to allow comparisons Estimated after a fixed number of runs High (unknown) variance in the case of few successes Could also be estimated after a fixed number of successes Would allow to control the variance A single value, insufficient alone anyway
32 Outputs and Performance measures Cumulative distributions Cope with both fitness threshold and bound on # evaluations Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max %run 5 %run evaluation number fitness value % runs vs # eval to reach success threshold (= 9 ) % runs vs fitness value reached before max eval (= 7 ) Rosenbrock function, dim, cond.
33 Outputs and Performance measures Dynamic Behaviors Median, best, worse, 25-, 75-percentiles Rosenbrock : 21 trials, dimension 2, tol 1.E 9, alpha, default size, eval max 12 rosenbrock_de2 rosenbrock_de5 function value 1.e+ 5.e+5 1.e+ 1.5e+ 2.e+ 2.5e+ 3.e+ 3.5e+.e+.5e+ function evaluations All runs of 2 settings Comparing DE2 and DE5 on Rosenbrock(), dim=2
34 Ellipsoid f elli (x) = i 1 n i=1 α n 1 x 2 i = x T H elli x 1... H elli = 1 C A α convex, separable felli rot(x) = f elli(rx) = x T Helli rotx R random rotation Helli rot = R T H ellir convex, non-separable cond(h elli ) = cond(h rot elli ) = α α = 1,..., α = axis ratio of 3, typical for real-world problem
35 Separable Ellipsoid function PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 7 Ellipsoid dimension, 21 trials, tolerance 1e 9, eval max 1e+7 5 SP1 3 2 Ellipsoid BFGS Ellipsoid DE5 Ellipsoid DE2 Ellipsoid PSO Ellipsoid CMAES Sqrt of sqrt of ellipsoid BFGS 1 2 Condition number 8
36 Separable Ellipsoid function PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 2 7 Ellipsoid dimension 2, 21 trials, tolerance 1e 9, eval max 1e+7 5 SP1 3 2 Ellipsoid BFGS Ellipsoid DE5 Ellipsoid DE2 Ellipsoid PSO Ellipsoid CMAES Sqrt of sqrt of ellipsoid BFGS 1 2 Condition number 8
37 Separable Ellipsoid function PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 7 Ellipsoid dimension, 21 trials, tolerance 1e 9, eval max 1e+7 5 SP1 3 2 Ellipsoid BFGS Ellipsoid DE5 Ellipsoid DE2 Ellipsoid PSO Ellipsoid CMAES Sqrt of sqrt of ellipsoid BFGS 1 2 Condition number 8
38 Rotated Ellipsoid function PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 7 Ellipsoid dimension, 21 trials, tolerance 1e 9, eval max 1e+7 5 SP Condition number Rotated Ellipsoid BFGS Rotated Ellipsoid DE5 Rotated Ellipsoid DE2 Rotated Ellipsoid PSO Rotated Ellipsoid CMAES sqrt of sqrt of rotated ellipsoid BFGS 8
39 Rotated Ellipsoid function PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 2 7 Ellipsoid dimension 2, 21 trials, tolerance 1e 9, eval max 1e+7 5 SP Condition number Rotated Ellipsoid BFGS Rotated Ellipsoid DE5 Rotated Ellipsoid DE2 Rotated Ellipsoid PSO Rotated Ellipsoid CMAES sqrt of sqrt of rotated ellipsoid BFGS 8
40 Rotated Ellipsoid function PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 7 Ellipsoid dimension, 21 trials, tolerance 1e 9, eval max 1e+7 5 SP Condition number Rotated Ellipsoid BFGS Rotated Ellipsoid DE5 Rotated Ellipsoid DE2 Rotated Ellipsoid PSO Rotated Ellipsoid CMAES sqrt of sqrt of rotated ellipsoid BFGS 8
41 Separable Ellipsoid function PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 7 Ellipsoid dimension, 21 trials, tolerance 1e 9, eval max 1e+7 5 SP1 3 2 Ellipsoid BFGS Ellipsoid DE5 Ellipsoid DE2 Ellipsoid PSO Ellipsoid CMAES Sqrt of sqrt of ellipsoid BFGS 1 2 Condition number 8
42 Ellipsoid: Discussion Bio-inspired algorithms Separable case: PSO and DE insensitive to conditionning... but PSO rapidly fails to solve the rotated version... while CMA-ES and DE (CR = 1) are rotation invariant DE scales poorly with dimension d 2.5 compared to d 1.5 for PSO and CMA-ES and BFGS... vs BFGS BFGS fails to solve ill-conditionned cases Matlab message: Roundoff error is stalling convergence CMA-ES only 7 times slower Line search couldn t find an acceptable point in the current search direction on quadratic functions!
43 Rosenbrock function (Banana) f rosen (x) = n 1 [ i=1 (1 xi ) 2 + α (x i+1 xi 2)2] Non-separable, but... also ran rotated version α = 2, classical Rosenbrock function α = 1,..., 8 Multi-modal for dimension >
44 Rosenbrock functions PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 8 dimension, 21 trials, tolerance 1e 9, eval max 1e+7 7 SP beta 8
45 Rosenbrock functions PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 2 8 dimension 2, 21 trials, tolerance 1e 9, eval max 1e+7 7 SP beta 8
46 Rosenbrock functions PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 8 dimension, 21 trials, tolerance 1e 9, eval max 1e+7 7 SP beta 8
47 Rosenbrock function Dim Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS α 1 Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha 1, default size, eval max Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha 1, default size, eval max %run 5 %run evaluation number fitness value % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
48 Rosenbrock function Dim Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS α Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max %run 5 %run evaluation number fitness value % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
49 Rosenbrock function Dim Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS α Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max %run 5 %run evaluation number fitness value % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
50 Rosenbrock function Dim Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS α 3 Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha 3, default size, eval max Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha 3, default size, eval max %run 5 %run evaluation number fitness value % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
51 Rosenbrock function Dim Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS α Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max %run 5 %run evaluation number fitness value % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
52 Rosenbrock function Dim Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS α Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max Rosenbrock : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max %run 5 %run evaluation number fitness value % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
53 Rosenbrock: Discussion Bio-inspired algorithms PSO sensitive to non-separability DE still scales badly with dimension... vs BFGS Numerical premature convergence on ill-condition problems Both local and global restarts improve the results
54 Rastrigin function f rast (x) = n + n i=1 x2 i cos(2πx i ) 3 2 separable multi-modal f rot rast(x) = f rast (Rx) 3 R random rotation non-separable multimodal
55 Rastrigin function - SP1 vs fitness value PSO, DE2, DE5, CMA-ES, and BFGS - PopSize Rastrigin: dimension, NP = 9 7 CMAES Rastrigin PSO Rastrigin DE2 Rastrigin DE5 Rastrigin BFGS Rastrigin CMAES Rotated Rastrigin PSO Rotated Rastrigin DE2 Rotated Rastrigin DE5 Rotated Rastrigin BFGS Rotated Rastrigin SP Function Value To Reach
56 Rastrigin function - SP1 vs fitness value PSO, DE2, DE5, CMA-ES, and BFGS - PopSize 1 Rastrigin: dimension, NP = CMAES Rastrigin PSO Rastrigin DE2 Rastrigin DE5 Rastrigin BFGS Rastrigin CMAES Rotated Rastrigin PSO Rotated Rastrigin DE2 Rotated Rastrigin DE5 Rotated Rastrigin BFGS Rotated Rastrigin SP Function Value To Reach
57 Rastrigin function - SP1 vs fitness value PSO, DE2, DE5, CMA-ES, and BFGS - PopSize 3 Rastrigin: dimension, NP = CMAES Rastrigin PSO Rastrigin DE2 Rastrigin DE5 Rastrigin BFGS Rastrigin CMAES Rotated Rastrigin PSO Rotated Rastrigin DE2 Rotated Rastrigin DE5 Rotated Rastrigin BFGS Rotated Rastrigin SP Function Value To Reach
58 Rastrigin function - SP1 vs fitness value PSO, DE2, DE5, CMA-ES, and BFGS - PopSize Rastrigin: dimension, NP = 9 7 CMAES Rastrigin PSO Rastrigin DE2 Rastrigin DE5 Rastrigin BFGS Rastrigin CMAES Rotated Rastrigin PSO Rotated Rastrigin DE2 Rotated Rastrigin DE5 Rotated Rastrigin BFGS Rotated Rastrigin SP Function Value To Reach
59 Rastrigin function - SP1 vs fitness value PSO, DE2, DE5, CMA-ES, and BFGS - PopSize 3 Rastrigin: dimension, NP = CMAES Rastrigin PSO Rastrigin DE2 Rastrigin DE5 Rastrigin BFGS Rastrigin CMAES Rotated Rastrigin PSO Rotated Rastrigin DE2 Rotated Rastrigin DE5 Rotated Rastrigin BFGS Rotated Rastrigin SP Function Value To Reach
60 Rastrigin function - SP1 vs fitness value PSO, DE2, DE5, CMA-ES, and BFGS - PopSize Rastrigin: dimension, NP = 9 7 CMAES Rastrigin PSO Rastrigin DE2 Rastrigin DE5 Rastrigin BFGS Rastrigin CMAES Rotated Rastrigin PSO Rotated Rastrigin DE2 Rotated Rastrigin DE5 Rotated Rastrigin BFGS Rotated Rastrigin SP Function Value To Reach
61 Rastrigin function - Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS - PopSize Rastrigin : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max Rastrigin : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max %run %run evaluation number fitness value 3 % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
62 Rastrigin function - Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS - PopSize 1 Rastrigin : 21 trials, dimension, tol 1.E 9, alpha 1, default size, eval max Rastrigin : 21 trials, dimension, tol 1.E 9, alpha 1, default size, eval max %run %run evaluation number fitness value 3 % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
63 Rastrigin function - Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS - PopSize 3 Rastrigin : 21 trials, dimension, tol 1.E 9, alpha 3, default size, eval max Rastrigin : 21 trials, dimension, tol 1.E 9, alpha 3, default size, eval max %run 5 %run evaluation number fitness value % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
64 Rastrigin function - Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS - PopSize Rastrigin : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max Rastrigin : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max %run 5 %run evaluation number fitness value % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
65 Rastrigin function - Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS - PopSize 3 Rastrigin : 21 trials, dimension, tol 1.E 9, alpha 3, default size, eval max Rastrigin : 21 trials, dimension, tol 1.E 9, alpha 3, default size, eval max %run 5 %run evaluation number fitness value % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
66 Rastrigin function - Cumulative distributions PSO, DE2, DE5, CMA-ES, and BFGS - PopSize Rastrigin : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max Rastrigin : 21 trials, dimension, tol 1.E 9, alpha, default size, eval max %run 5 %run evaluation number fitness value % success vs # eval to reach success threshold (= 9 ) % success vs fitness value reached before max eval (= 7 )
67 Rastrigin: Discussion Bio-inspired algorithms Increasing population size improves the results Optimal size is algorithm-dependent CMA-ES and PSO solve separable case PSO about times slower Only CMA-ES solves the rotated Rastrigin reliably requires popsize 3... vs BFGS Gets stuck in local optima Whatever the restart strategies No numerical premature convergence
68 Away from quadraticity Non-linear scaling invariance Comparison-based algorithms are insensitive to monotonous transformations True for DE, PSO and all ESs BFGS is not and convergence results do depend on convexity Other test functions Simple transformation of ellispoid f SSE (x) = felli (x) The DiffPow function and DiffPow f DiffPow (x) = ( x i 2+( i) )
69 DiffPow SP1 vs Fitness Values PSO, DE2, DE5, CMA-ES, and BFGS - Dimension 5 Sum of diff. powers: dimension SP1 3 BFGS diffpow DE5 NP= diffpow DE2 NP= diffpow PSO NP= diffpow CMAES diffpow BFGS sqrt(diffpow) DE5 NP= sqrt(diffpow) DE2 NP= sqrt(diffpow) PSO NP= sqrt(diffpow) CMAES sqrt(diffpow) Diff Pow Value 3
70 Sqrt and DiffPow: Discussion Bio-inspired algorithms Invariant PSO performs best as expected! DiffPow is separable... vs BFGS Worse on Ellipsoid than on Ellispoid Better on DiffPow than on DiffPow closer to quadraticity? Premature numerical convergence for high CN... fixed by the local restart strategy
71 1 Continuous optimization and stochastic search 2 Problem difficulties 3 Implementations and parameter settings Experiments and results Experimental conditions Outputs and Performance measures 5 Conclusion
72 Further work What is missing? The algorithms Other deterministic methods e.g. Scilab procedures PCX crossover operator Specific Evolution Engine The testbed Noisy functions Constrained functions The statistics Confidence bounds for SP1 and other precision/cost measures Real-world functions Which ones??? Do complete experiments
73 Conclusions Summary Bio-inspired algorithms BFGS All are monotonous-transformation invariant PSO very good... only on separable (easy) functions DE poorly scales up with the dimension CMA-ES is a clear best choice Sensitive to non-separability when CR < 1 Redo experiments with parameter-less restart version Optimal choice for quasi-quadratic functions but can suffer from numerical premature convergence for high condition number Even with restart procedures, fails on multimodal problem dim only here
74 Conclusions The coming of age The message to our Applied Maths colleagues Bio-inspired vs BFGS but CMA-ES only 7 times slower than BFGS on (quasi-)quadratic functions. is less hindered by high conditionning, is monotonous-transformation invariant, is a global search method! Moreover, Theoretical results are catching up Linear convergence for SA-ES Auger, 5 with bound on the CV speed On-going work for CMA-ES
Experimental Comparisons of Derivative Free Optimization Algorithms
Experimental Comparisons of Derivative Free Optimization Algorithms Anne Auger Nikolaus Hansen J. M. Perez Zerpa Raymond Ros Marc Schoenauer TAO Project-Team, INRIA Saclay Île-de-France, and Microsoft-INRIA
More informationIntroduction to Black-Box Optimization in Continuous Search Spaces. Definitions, Examples, Difficulties
1 Introduction to Black-Box Optimization in Continuous Search Spaces Definitions, Examples, Difficulties Tutorial: Evolution Strategies and CMA-ES (Covariance Matrix Adaptation) Anne Auger & Nikolaus Hansen
More informationEmpirical comparisons of several derivative free optimization algorithms
Empirical comparisons of several derivative free optimization algorithms A. Auger,, N. Hansen,, J. M. Perez Zerpa, R. Ros, M. Schoenauer, TAO Project-Team, INRIA Saclay Ile-de-France LRI, Bat 90 Univ.
More informationStochastic Methods for Continuous Optimization
Stochastic Methods for Continuous Optimization Anne Auger et Dimo Brockhoff Paris-Saclay Master - Master 2 Informatique - Parcours Apprentissage, Information et Contenu (AIC) 2016 Slides taken from Auger,
More informationTutorial CMA-ES Evolution Strategies and Covariance Matrix Adaptation
Tutorial CMA-ES Evolution Strategies and Covariance Matrix Adaptation Anne Auger & Nikolaus Hansen INRIA Research Centre Saclay Île-de-France Project team TAO University Paris-Sud, LRI (UMR 8623), Bat.
More informationAdvanced Optimization
Advanced Optimization Lecture 3: 1: Randomized Algorithms for for Continuous Discrete Problems Problems November 22, 2016 Master AIC Université Paris-Saclay, Orsay, France Anne Auger INRIA Saclay Ile-de-France
More informationStochastic optimization and a variable metric approach
The challenges for stochastic optimization and a variable metric approach Microsoft Research INRIA Joint Centre, INRIA Saclay April 6, 2009 Content 1 Introduction 2 The Challenges 3 Stochastic Search 4
More informationIntroduction to Randomized Black-Box Numerical Optimization and CMA-ES
Introduction to Randomized Black-Box Numerical Optimization and CMA-ES July 3, 2017 CEA/EDF/Inria summer school "Numerical Analysis" Université Pierre-et-Marie-Curie, Paris, France Anne Auger, Asma Atamna,
More informationProblem Statement Continuous Domain Search/Optimization. Tutorial Evolution Strategies and Related Estimation of Distribution Algorithms.
Tutorial Evolution Strategies and Related Estimation of Distribution Algorithms Anne Auger & Nikolaus Hansen INRIA Saclay - Ile-de-France, project team TAO Universite Paris-Sud, LRI, Bat. 49 945 ORSAY
More informationEvolution Strategies and Covariance Matrix Adaptation
Evolution Strategies and Covariance Matrix Adaptation Cours Contrôle Avancé - Ecole Centrale Paris Anne Auger January 2014 INRIA Research Centre Saclay Île-de-France University Paris-Sud, LRI (UMR 8623),
More informationOptimisation numérique par algorithmes stochastiques adaptatifs
Optimisation numérique par algorithmes stochastiques adaptatifs Anne Auger M2R: Apprentissage et Optimisation avancés et Applications anne.auger@inria.fr INRIA Saclay - Ile-de-France, project team TAO
More informationNumerical optimization Typical di culties in optimization. (considerably) larger than three. dependencies between the objective variables
100 90 80 70 60 50 40 30 20 10 4 3 2 1 0 1 2 3 4 0 What Makes a Function Di cult to Solve? ruggedness non-smooth, discontinuous, multimodal, and/or noisy function dimensionality non-separability (considerably)
More informationAddressing Numerical Black-Box Optimization: CMA-ES (Tutorial)
Addressing Numerical Black-Box Optimization: CMA-ES (Tutorial) Anne Auger & Nikolaus Hansen INRIA Research Centre Saclay Île-de-France Project team TAO University Paris-Sud, LRI (UMR 8623), Bat. 490 91405
More informationA Restart CMA Evolution Strategy With Increasing Population Size
Anne Auger and Nikolaus Hansen A Restart CMA Evolution Strategy ith Increasing Population Size Proceedings of the IEEE Congress on Evolutionary Computation, CEC 2005 c IEEE A Restart CMA Evolution Strategy
More informationNumerical Optimization: Basic Concepts and Algorithms
May 27th 2015 Numerical Optimization: Basic Concepts and Algorithms R. Duvigneau R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 1 Outline Some basic concepts in optimization Some
More informationTutorial CMA-ES Evolution Strategies and Covariance Matrix Adaptation
Tutorial CMA-ES Evolution Strategies and Covariance Matrix Adaptation Anne Auger & Nikolaus Hansen INRIA Research Centre Saclay Île-de-France Project team TAO University Paris-Sud, LRI (UMR 8623), Bat.
More informationMirrored Variants of the (1,4)-CMA-ES Compared on the Noiseless BBOB-2010 Testbed
Author manuscript, published in "GECCO workshop on Black-Box Optimization Benchmarking (BBOB') () 9-" DOI :./8.8 Mirrored Variants of the (,)-CMA-ES Compared on the Noiseless BBOB- Testbed [Black-Box Optimization
More informationBI-population CMA-ES Algorithms with Surrogate Models and Line Searches
BI-population CMA-ES Algorithms with Surrogate Models and Line Searches Ilya Loshchilov 1, Marc Schoenauer 2 and Michèle Sebag 2 1 LIS, École Polytechnique Fédérale de Lausanne 2 TAO, INRIA CNRS Université
More informationThree Steps toward Tuning the Coordinate Systems in Nature-Inspired Optimization Algorithms
Three Steps toward Tuning the Coordinate Systems in Nature-Inspired Optimization Algorithms Yong Wang and Zhi-Zhong Liu School of Information Science and Engineering Central South University ywang@csu.edu.cn
More informationThree Steps toward Tuning the Coordinate Systems in Nature-Inspired Optimization Algorithms
Three Steps toward Tuning the Coordinate Systems in Nature-Inspired Optimization Algorithms Yong Wang and Zhi-Zhong Liu School of Information Science and Engineering Central South University ywang@csu.edu.cn
More informationAdaptive Coordinate Descent
Adaptive Coordinate Descent Ilya Loshchilov 1,2, Marc Schoenauer 1,2, Michèle Sebag 2,1 1 TAO Project-team, INRIA Saclay - Île-de-France 2 and Laboratoire de Recherche en Informatique (UMR CNRS 8623) Université
More informationSurrogate models for Single and Multi-Objective Stochastic Optimization: Integrating Support Vector Machines and Covariance-Matrix Adaptation-ES
Covariance Matrix Adaptation-Evolution Strategy Surrogate models for Single and Multi-Objective Stochastic Optimization: Integrating and Covariance-Matrix Adaptation-ES Ilya Loshchilov, Marc Schoenauer,
More informationDecomposition and Metaoptimization of Mutation Operator in Differential Evolution
Decomposition and Metaoptimization of Mutation Operator in Differential Evolution Karol Opara 1 and Jaros law Arabas 2 1 Systems Research Institute, Polish Academy of Sciences 2 Institute of Electronic
More information2 Differential Evolution and its Control Parameters
COMPETITIVE DIFFERENTIAL EVOLUTION AND GENETIC ALGORITHM IN GA-DS TOOLBOX J. Tvrdík University of Ostrava 1 Introduction The global optimization problem with box constrains is formed as follows: for a
More informationarxiv: v1 [cs.ne] 9 May 2016
Anytime Bi-Objective Optimization with a Hybrid Multi-Objective CMA-ES (HMO-CMA-ES) arxiv:1605.02720v1 [cs.ne] 9 May 2016 ABSTRACT Ilya Loshchilov University of Freiburg Freiburg, Germany ilya.loshchilov@gmail.com
More informationIntroduction to Optimization
Introduction to Optimization Blackbox Optimization Marc Toussaint U Stuttgart Blackbox Optimization The term is not really well defined I use it to express that only f(x) can be evaluated f(x) or 2 f(x)
More informationBounding the Population Size of IPOP-CMA-ES on the Noiseless BBOB Testbed
Bounding the Population Size of OP-CMA-ES on the Noiseless BBOB Testbed Tianjun Liao IRIDIA, CoDE, Université Libre de Bruxelles (ULB), Brussels, Belgium tliao@ulb.ac.be Thomas Stützle IRIDIA, CoDE, Université
More informationA0M33EOA: EAs for Real-Parameter Optimization. Differential Evolution. CMA-ES.
A0M33EOA: EAs for Real-Parameter Optimization. Differential Evolution. CMA-ES. Petr Pošík Czech Technical University in Prague Faculty of Electrical Engineering Department of Cybernetics Many parts adapted
More informationCovariance Matrix Adaptation in Multiobjective Optimization
Covariance Matrix Adaptation in Multiobjective Optimization Dimo Brockhoff INRIA Lille Nord Europe October 30, 2014 PGMO-COPI 2014, Ecole Polytechnique, France Mastertitelformat Scenario: Multiobjective
More informationBenchmarking a BI-Population CMA-ES on the BBOB-2009 Function Testbed
Benchmarking a BI-Population CMA-ES on the BBOB- Function Testbed Nikolaus Hansen Microsoft Research INRIA Joint Centre rue Jean Rostand Orsay Cedex, France Nikolaus.Hansen@inria.fr ABSTRACT We propose
More informationBenchmarking a Weighted Negative Covariance Matrix Update on the BBOB-2010 Noiseless Testbed
Benchmarking a Weighted Negative Covariance Matrix Update on the BBOB- Noiseless Testbed Nikolaus Hansen, Raymond Ros To cite this version: Nikolaus Hansen, Raymond Ros. Benchmarking a Weighted Negative
More informationBenchmarking the (1+1)-CMA-ES on the BBOB-2009 Function Testbed
Benchmarking the (+)-CMA-ES on the BBOB-9 Function Testbed Anne Auger, Nikolaus Hansen To cite this version: Anne Auger, Nikolaus Hansen. Benchmarking the (+)-CMA-ES on the BBOB-9 Function Testbed. ACM-GECCO
More informationComparison-Based Optimizers Need Comparison-Based Surrogates
Comparison-Based Optimizers Need Comparison-Based Surrogates Ilya Loshchilov 1,2, Marc Schoenauer 1,2, and Michèle Sebag 2,1 1 TAO Project-team, INRIA Saclay - Île-de-France 2 Laboratoire de Recherche
More informationAdaptive Differential Evolution and Exponential Crossover
Proceedings of the International Multiconference on Computer Science and Information Technology pp. 927 931 ISBN 978-83-60810-14-9 ISSN 1896-7094 Adaptive Differential Evolution and Exponential Crossover
More informationInvestigating the Local-Meta-Model CMA-ES for Large Population Sizes
Investigating the Local-Meta-Model CMA-ES for Large Population Sizes Zyed Bouzarkouna, Anne Auger, Didier Yu Ding To cite this version: Zyed Bouzarkouna, Anne Auger, Didier Yu Ding. Investigating the Local-Meta-Model
More informationBeta Damping Quantum Behaved Particle Swarm Optimization
Beta Damping Quantum Behaved Particle Swarm Optimization Tarek M. Elbarbary, Hesham A. Hefny, Atef abel Moneim Institute of Statistical Studies and Research, Cairo University, Giza, Egypt tareqbarbary@yahoo.com,
More informationNatural Evolution Strategies for Direct Search
Tobias Glasmachers Natural Evolution Strategies for Direct Search 1 Natural Evolution Strategies for Direct Search PGMO-COPI 2014 Recent Advances on Continuous Randomized black-box optimization Thursday
More informationMulti-start JADE with knowledge transfer for numerical optimization
Multi-start JADE with knowledge transfer for numerical optimization Fei Peng, Ke Tang,Guoliang Chen and Xin Yao Abstract JADE is a recent variant of Differential Evolution (DE) for numerical optimization,
More informationInvestigation of Mutation Strategies in Differential Evolution for Solving Global Optimization Problems
Investigation of Mutation Strategies in Differential Evolution for Solving Global Optimization Problems Miguel Leon Ortiz and Ning Xiong Mälardalen University, Västerås, SWEDEN Abstract. Differential evolution
More informationThe particle swarm optimization algorithm: convergence analysis and parameter selection
Information Processing Letters 85 (2003) 317 325 www.elsevier.com/locate/ipl The particle swarm optimization algorithm: convergence analysis and parameter selection Ioan Cristian Trelea INA P-G, UMR Génie
More informationComputational Intelligence Winter Term 2017/18
Computational Intelligence Winter Term 2017/18 Prof. Dr. Günter Rudolph Lehrstuhl für Algorithm Engineering (LS 11) Fakultät für Informatik TU Dortmund mutation: Y = X + Z Z ~ N(0, C) multinormal distribution
More informationIEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION 1
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION 1 Population-Based Algorithm Portfolios for Numerical Optimization Fei Peng, Student Member, IEEE, Ke Tang, Member, IEEE, Guoliang Chen, and Xin Yao, Fellow,
More informationEfficient Covariance Matrix Update for Variable Metric Evolution Strategies
Efficient Covariance Matrix Update for Variable Metric Evolution Strategies Thorsten Suttorp, Nikolaus Hansen, Christian Igel To cite this version: Thorsten Suttorp, Nikolaus Hansen, Christian Igel. Efficient
More informationReal-Parameter Black-Box Optimization Benchmarking 2010: Presentation of the Noiseless Functions
Real-Parameter Black-Box Optimization Benchmarking 2010: Presentation of the Noiseless Functions Steffen Finck, Nikolaus Hansen, Raymond Ros and Anne Auger Report 2009/20, Research Center PPE, re-compiled
More informationCMA-ES a Stochastic Second-Order Method for Function-Value Free Numerical Optimization
CMA-ES a Stochastic Second-Order Method for Function-Value Free Numerical Optimization Nikolaus Hansen INRIA, Research Centre Saclay Machine Learning and Optimization Team, TAO Univ. Paris-Sud, LRI MSRC
More informationLECTURE 22: SWARM INTELLIGENCE 3 / CLASSICAL OPTIMIZATION
15-382 COLLECTIVE INTELLIGENCE - S19 LECTURE 22: SWARM INTELLIGENCE 3 / CLASSICAL OPTIMIZATION TEACHER: GIANNI A. DI CARO WHAT IF WE HAVE ONE SINGLE AGENT PSO leverages the presence of a swarm: the outcome
More informationResearch Article A Novel Differential Evolution Invasive Weed Optimization Algorithm for Solving Nonlinear Equations Systems
Journal of Applied Mathematics Volume 2013, Article ID 757391, 18 pages http://dx.doi.org/10.1155/2013/757391 Research Article A Novel Differential Evolution Invasive Weed Optimization for Solving Nonlinear
More informationBenchmarking Natural Evolution Strategies with Adaptation Sampling on the Noiseless and Noisy Black-box Optimization Testbeds
Benchmarking Natural Evolution Strategies with Adaptation Sampling on the Noiseless and Noisy Black-box Optimization Testbeds Tom Schaul Courant Institute of Mathematical Sciences, New York University
More informationFALL 2018 MATH 4211/6211 Optimization Homework 4
FALL 2018 MATH 4211/6211 Optimization Homework 4 This homework assignment is open to textbook, reference books, slides, and online resources, excluding any direct solution to the problem (such as solution
More informationBlack-Box Optimization Benchmarking the IPOP-CMA-ES on the Noisy Testbed
Black-Box Optimization Benchmarking the IPOP-CMA-ES on the Noisy Testbed Raymond Ros To cite this version: Raymond Ros. Black-Box Optimization Benchmarking the IPOP-CMA-ES on the Noisy Testbed. Genetic
More informationZero-Order Methods for the Optimization of Noisy Functions. Jorge Nocedal
Zero-Order Methods for the Optimization of Noisy Functions Jorge Nocedal Northwestern University Simons Institute, October 2017 1 Collaborators Albert Berahas Northwestern University Richard Byrd University
More informationHow information theory sheds new light on black-box optimization
How information theory sheds new light on black-box optimization Anne Auger Colloque Théorie de l information : nouvelles frontières dans le cadre du centenaire de Claude Shannon IHP, 26, 27 and 28 October,
More informationImproving the Convergence of Back-Propogation Learning with Second Order Methods
the of Back-Propogation Learning with Second Order Methods Sue Becker and Yann le Cun, Sept 1988 Kasey Bray, October 2017 Table of Contents 1 with Back-Propagation 2 the of BP 3 A Computationally Feasible
More informationScientific Computing: Optimization
Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture
More informationSub-Sampled Newton Methods
Sub-Sampled Newton Methods F. Roosta-Khorasani and M. W. Mahoney ICSI and Dept of Statistics, UC Berkeley February 2016 F. Roosta-Khorasani and M. W. Mahoney (UCB) Sub-Sampled Newton Methods Feb 2016 1
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationA (1+1)-CMA-ES for Constrained Optimisation
A (1+1)-CMA-ES for Constrained Optimisation Dirk Arnold, Nikolaus Hansen To cite this version: Dirk Arnold, Nikolaus Hansen. A (1+1)-CMA-ES for Constrained Optimisation. Terence Soule and Jason H. Moore.
More informationThe CMA Evolution Strategy: A Tutorial
The CMA Evolution Strategy: A Tutorial Nikolaus Hansen November 6, 205 Contents Nomenclature 3 0 Preliminaries 4 0. Eigendecomposition of a Positive Definite Matrix... 5 0.2 The Multivariate Normal Distribution...
More informationA comparative introduction to two optimization classics: the Nelder-Mead and the CMA-ES algorithms
A comparative introduction to two optimization classics: the Nelder-Mead and the CMA-ES algorithms Rodolphe Le Riche 1,2 1 Ecole des Mines de Saint Etienne, France 2 CNRS LIMOS France March 2018 MEXICO
More informationDifferential Evolution: Competitive Setting of Control Parameters
Proceedings of the International Multiconference on Computer Science and Information Technology pp. 207 213 ISSN 1896-7094 c 2006 PIPS Differential Evolution: Competitive Setting of Control Parameters
More informationarxiv: v1 [cs.ne] 29 Jul 2014
A CUDA-Based Real Parameter Optimization Benchmark Ke Ding and Ying Tan School of Electronics Engineering and Computer Science, Peking University arxiv:1407.7737v1 [cs.ne] 29 Jul 2014 Abstract. Benchmarking
More informationUnconstrained Multivariate Optimization
Unconstrained Multivariate Optimization Multivariate optimization means optimization of a scalar function of a several variables: and has the general form: y = () min ( ) where () is a nonlinear scalar-valued
More informationA Comparative Study of Differential Evolution, Particle Swarm Optimization, and Evolutionary Algorithms on Numerical Benchmark Problems
A Comparative Study of Differential Evolution, Particle Swarm Optimization, and Evolutionary Algorithms on Numerical Benchmark Problems Jakob Vesterstrøm BiRC - Bioinformatics Research Center University
More informationStochastic Search using the Natural Gradient
Keywords: stochastic search, natural gradient, evolution strategies Sun Yi Daan Wierstra Tom Schaul Jürgen Schmidhuber IDSIA, Galleria 2, Manno 6928, Switzerland yi@idsiach daan@idsiach tom@idsiach juergen@idsiach
More informationComparison of NEWUOA with Different Numbers of Interpolation Points on the BBOB Noiseless Testbed
Comparison of NEWUOA with Different Numbers of Interpolation Points on the BBOB Noiseless Testbed Raymond Ros To cite this version: Raymond Ros. Comparison of NEWUOA with Different Numbers of Interpolation
More informationStochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions
International Journal of Control Vol. 00, No. 00, January 2007, 1 10 Stochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions I-JENG WANG and JAMES C.
More informationLocal Search & Optimization
Local Search & Optimization CE417: Introduction to Artificial Intelligence Sharif University of Technology Spring 2018 Soleymani Artificial Intelligence: A Modern Approach, 3 rd Edition, Chapter 4 Some
More informationOptimization using derivative-free and metaheuristic methods
Charles University in Prague Faculty of Mathematics and Physics MASTER THESIS Kateřina Márová Optimization using derivative-free and metaheuristic methods Department of Numerical Mathematics Supervisor
More informationResearch Article A Hybrid Backtracking Search Optimization Algorithm with Differential Evolution
Mathematical Problems in Engineering Volume 2015, Article ID 769245, 16 pages http://dx.doi.org/10.1155/2015/769245 Research Article A Hybrid Backtracking Search Optimization Algorithm with Differential
More informationImproving on the Kalman Swarm
Improving on the Kalman Swarm Extracting Its Essential Characteristics Christopher K. Monson and Kevin D. Seppi Brigham Young University, Provo UT 84602, USA {c,kseppi}@cs.byu.edu Abstract. The Kalman
More informationDifferential Evolution Based Particle Swarm Optimization
Differential Evolution Based Particle Swarm Optimization Mahamed G.H. Omran Department of Computer Science Gulf University of Science and Technology Kuwait mjomran@gmail.com Andries P. Engelbrecht Department
More informationGradient Descent. Dr. Xiaowei Huang
Gradient Descent Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Three machine learning algorithms: decision tree learning k-nn linear regression only optimization objectives are discussed,
More informationPrincipled Design of Continuous Stochastic Search: From Theory to
Contents Principled Design of Continuous Stochastic Search: From Theory to Practice....................................................... 3 Nikolaus Hansen and Anne Auger 1 Introduction: Top Down Versus
More informationParticle Swarm Optimization with Velocity Adaptation
In Proceedings of the International Conference on Adaptive and Intelligent Systems (ICAIS 2009), pp. 146 151, 2009. c 2009 IEEE Particle Swarm Optimization with Velocity Adaptation Sabine Helwig, Frank
More informationLandscapes & Algorithms for Quantum Control
Dept of Applied Maths & Theoretical Physics University of Cambridge, UK April 15, 211 Control Landscapes: What do we really know? Kinematic control landscapes generally universally nice 1 Pure-state transfer
More informationBioinformatics: Network Analysis
Bioinformatics: Network Analysis Model Fitting COMP 572 (BIOS 572 / BIOE 564) - Fall 2013 Luay Nakhleh, Rice University 1 Outline Parameter estimation Model selection 2 Parameter Estimation 3 Generally
More informationUNCONSTRAINED OPTIMIZATION
UNCONSTRAINED OPTIMIZATION 6. MATHEMATICAL BASIS Given a function f : R n R, and x R n such that f(x ) < f(x) for all x R n then x is called a minimizer of f and f(x ) is the minimum(value) of f. We wish
More informationEfficient Natural Evolution Strategies
Efficient Natural Evolution Strategies Evolution Strategies and Evolutionary Programming Track ABSTRACT Yi Sun Manno 698, Switzerland yi@idsiach Tom Schaul Manno 698, Switzerland tom@idsiach Efficient
More informationToward Effective Initialization for Large-Scale Search Spaces
Toward Effective Initialization for Large-Scale Search Spaces Shahryar Rahnamayan University of Ontario Institute of Technology (UOIT) Faculty of Engineering and Applied Science 000 Simcoe Street North
More informationCenter-based initialization for large-scale blackbox
See discussions, stats, and author profiles for this publication at: http://www.researchgate.net/publication/903587 Center-based initialization for large-scale blackbox problems ARTICLE FEBRUARY 009 READS
More informationBenchmarking Projection-Based Real Coded Genetic Algorithm on BBOB-2013 Noiseless Function Testbed
Benchmarking Projection-Based Real Coded Genetic Algorithm on BBOB-2013 Noiseless Function Testbed Babatunde Sawyerr 1 Aderemi Adewumi 2 Montaz Ali 3 1 University of Lagos, Lagos, Nigeria 2 University
More informationQuality and Robustness Improvement for Real World Industrial Systems using a Fuzzy Particle Swarm Optimization
ENGINEERING APPLICATIONS OF ARTIFICIAL INTELLIGENCE, VOL. X, NO. X, 2015 1 Quality and Robustness Improvement for Real World Industrial Systems using a Fuzzy Particle Swarm Optimization Sai Ho Ling, Senior
More informationBenchmarking Gaussian Processes and Random Forests Surrogate Models on the BBOB Noiseless Testbed
Benchmarking Gaussian Processes and Random Forests Surrogate Models on the BBOB Noiseless Testbed Lukáš Bajer Institute of Computer Science Academy of Sciences of the Czech Republic and Faculty of Mathematics
More information4 damped (modified) Newton methods
4 damped (modified) Newton methods 4.1 damped Newton method Exercise 4.1 Determine with the damped Newton method the unique real zero x of the real valued function of one variable f(x) = x 3 +x 2 using
More informationLocal Search & Optimization
Local Search & Optimization CE417: Introduction to Artificial Intelligence Sharif University of Technology Spring 2017 Soleymani Artificial Intelligence: A Modern Approach, 3 rd Edition, Chapter 4 Outline
More informationDeep Learning. Authors: I. Goodfellow, Y. Bengio, A. Courville. Chapter 4: Numerical Computation. Lecture slides edited by C. Yim. C.
Chapter 4: Numerical Computation Deep Learning Authors: I. Goodfellow, Y. Bengio, A. Courville Lecture slides edited by 1 Chapter 4: Numerical Computation 4.1 Overflow and Underflow 4.2 Poor Conditioning
More informationAn Algorithm for Unconstrained Quadratically Penalized Convex Optimization (post conference version)
7/28/ 10 UseR! The R User Conference 2010 An Algorithm for Unconstrained Quadratically Penalized Convex Optimization (post conference version) Steven P. Ellis New York State Psychiatric Institute at Columbia
More information1 Numerical optimization
Contents 1 Numerical optimization 5 1.1 Optimization of single-variable functions............ 5 1.1.1 Golden Section Search................... 6 1.1. Fibonacci Search...................... 8 1. Algorithms
More informationBBOB-Benchmarking Two Variants of the Line-Search Algorithm
BBOB-Benchmarking Two Variants of the Line-Search Algorithm Petr Pošík Czech Technical University in Prague, Faculty of Electrical Engineering, Dept. of Cybernetics Technická, Prague posik@labe.felk.cvut.cz
More informationCrossover and the Different Faces of Differential Evolution Searches
WCCI 21 IEEE World Congress on Computational Intelligence July, 18-23, 21 - CCIB, Barcelona, Spain CEC IEEE Crossover and the Different Faces of Differential Evolution Searches James Montgomery Abstract
More informationSelected Topics in Optimization. Some slides borrowed from
Selected Topics in Optimization Some slides borrowed from http://www.stat.cmu.edu/~ryantibs/convexopt/ Overview Optimization problems are almost everywhere in statistics and machine learning. Input Model
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More informationMethods for Unconstrained Optimization Numerical Optimization Lectures 1-2
Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Coralia Cartis, University of Oxford INFOMM CDT: Modelling, Analysis and Computation of Continuous Real-World Problems Methods
More informationVerification of a hypothesis about unification and simplification for position updating formulas in particle swarm optimization.
nd Workshop on Advanced Research and Technology in Industry Applications (WARTIA ) Verification of a hypothesis about unification and simplification for position updating formulas in particle swarm optimization
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More informationIntroduction to Optimization
Introduction to Optimization Konstantin Tretyakov (kt@ut.ee) MTAT.03.227 Machine Learning So far Machine learning is important and interesting The general concept: Fitting models to data So far Machine
More informationThe Story So Far... The central problem of this course: Smartness( X ) arg max X. Possibly with some constraints on X.
Heuristic Search The Story So Far... The central problem of this course: arg max X Smartness( X ) Possibly with some constraints on X. (Alternatively: arg min Stupidness(X ) ) X Properties of Smartness(X)
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationA CMA-ES with Multiplicative Covariance Matrix Updates
A with Multiplicative Covariance Matrix Updates Oswin Krause Department of Computer Science University of Copenhagen Copenhagen,Denmark oswin.krause@di.ku.dk Tobias Glasmachers Institut für Neuroinformatik
More informationCLASSICAL gradient methods and evolutionary algorithms
IEEE TRANSACTIONS ON EVOLUTIONARY COMPUTATION, VOL. 2, NO. 2, JULY 1998 45 Evolutionary Algorithms and Gradient Search: Similarities and Differences Ralf Salomon Abstract Classical gradient methods and
More information