Problem Statement Continuous Domain Search/Optimization. Tutorial Evolution Strategies and Related Estimation of Distribution Algorithms.

Transcription

1 Tutorial Evolution Strategies and Related Estimation of Distribution Algorithms Anne Auger & Nikolaus Hansen INRIA Saclay - Ile-de-France, project team TAO Universite Paris-Sud, LRI, Bat ORSAY Cedex, France Continuous Domain Search/Optimization Task: minimize a objective function (fitness function, loss function) in continuous domain f : X R n R, x f(x) Black Box scenario (direct search scenario) x f(x) Copyright is held by the author/owner(s). GECCO 8, July, 8, Atlanta, Georgia, USA. ACM /8/ /8/7 gradients are not available or not useful problem domain specific knowledge is used only within the black box, e.g. within an appropriate encoding Search costs: number of function evaluations Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 Content Continuous Domain Search/Optimization Non-Separable Problems Ill-Conditioned Problems Evolution Strategies and EDAs A Search Template Why One-Fifth Success Rule Self-Adaptation Path Length Control 4 Estimation of Distribution 5 Conclusion Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 77 Goal fast convergence to the global optimum... or to a robust solution x solution x with small function value with least search cost there are two conflicting objectives Typical Examples shape optimization (e.g. using CFD) model calibration parameter calibration Problems exhaustive search is infeasible naive random search takes too long deterministic search is not successful / takes too long Approach: stochastic search, Evolutionary Algorithms curve fitting, airfoils biological, physical controller, plants, images Anne Auger & Nikolaus Hansen () Evolution Strategies 4 / 5

2 Metaphors Evolutionary Computation Optimization individual, offspring, parent candidate solution decision variables design variables object variables population set of candidate solutions fitness function objective function loss function cost function generation iteration What Makes a Function Difficult to Solve? Why stochastic search? ruggedness non-smooth, discontinuous, multimodal, and/or noisy function dimensionality non-separability (considerably) larger than three dependencies between the objective variables ill-conditioning cut from -D example, solvable with CMA-ES a narrow ridge... function properties Anne Auger & Nikolaus Hansen () Evolution Strategies 5 / 5 Anne Auger & Nikolaus Hansen () Evolution Strategies 7 / 5 Objective Function Properties We assume f : X R n R to have at least moderate dimensionality, say n, and to be non-linear, non-convex, and non-separable. Additionally, f can be multimodal non-smooth discontinuous ill-conditioned noisy... there are eventually many local optima derivatives do not exist Goal : cope with any of these function properties they are related to real-world problems Separable Problems Definition (Separable Problem) A function f is separable if arg min (x,...,x n) f(x,...,x n ) = ( Non-Separable Problems ) arg min f(x,...),...,arg min f(...,x n ) x x n it follows that f can be optimized in a sequence of n independent -D optimization processes Example: Additively decomposable functions n f(x,...,x n ) = f i (x i ) i= Rastrigin function Anne Auger & Nikolaus Hansen () Evolution Strategies 6 / 5 78 Anne Auger & Nikolaus Hansen () Evolution Strategies 8 / 5

3 Non-Separable Problems Non-Separable Problems Building a non-separable problem from a separable one Ill-Conditioned Problems What Makes a Function Difficult to Solve?... and what can be done Rotating the coordinate system f : x f(x) separable f : x f(rx) non-separable R rotation matrix The Problem Ruggedness What can be done non-local policy, large sampling width (step-size) as large as possible while preserving a reasonable convergence speed R Hansen, Ostermeier, Gawelczyk (995). On the adaptation of arbitrary normal mutation distributions in evolution strategies: The generating set adaptation. Sixth ICGA, pp , Morgan Kaufmann Salomon (996). Reevaluating Genetic Algorithm Performance under Coordinate Rotation of Benchmark Functions; A survey of some theoretical and practical aspects of genetic algorithms. BioSystems, 9():6-78 Anne Auger & Nikolaus Hansen () Evolution Strategies 9 / 5 Dimensionality, Non-Separability Ill-conditioning stochastic, non-elitistic, population-based method recombination operator serves as repair mechanism exploiting the problem structure locality, neighborhood, encoding second order approach changes the neighborhood metric Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 Ill-Conditioned Problems Ill-Conditioned Problems If f is quadratic, f : x x T Hx, ill-conditioned means a high condition number of Hessian Matrix H ill-conditioned means squeezed lines of equal function value Evolution Strategies and EDAs Non-Separable Problems Ill-Conditioned Problems Evolution Strategies and EDAs A Search Template Increased condition number consider the curvature of iso-fitness lines Why One-Fifth Success Rule Self-Adaptation Path Length Control 4 Estimation of Distribution 5 Conclusion Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 79 Anne Auger & Nikolaus Hansen () Evolution Strategies / 5

4 Evolution Strategies and EDAs A Search Template Evolution Strategies and EDAs Stochastic Search Normal Distribution Isotropic Case A black box search template to minimize f : R n R Initialize distribution parameters θ, set population size λ N While not terminate Sample distribution P (x θ) x,...,x λ R n Evaluate x,...,x λ on f Update parameters θ F θ (θ, x,...,x λ, f(x ),...,f(x λ )) probability density.4... Standard Normal Distribution 4 4 D Normal Distribution probability density of -D standard normal distribution.8 Everything depends on the definition of P and F θ deterministic algorithms are covered as well In Evolutionary Algorithms the distribution P is often implicitly defined via operators on a population, in particular, selection, recombination and mutation Natural template for Estimation of Distribution Algorithms Anne Auger & Nikolaus Hansen () Evolution Strategies / D Anne Auger & Nikolaus Hansen () Evolution Strategies 5 / 5 Evolution Strategies and EDAs A Search Template Evolution Strategies and Normal Estimation of Distribution Algorithms New search points are sampled normally distributed as perturbations of m where x i m + σ N i (, C) for i =,...,λ where x i, m R n, σ R +, and C R n n the mean vector m R n represents the favorite solution the so-called step-size σ R + controls the step length the covariance matrix C R n n determines the shape of the distribution ellipsoid The question remains how to update m, C, and σ. Evolution Strategies and EDAs The Multi-Variate (n-dimensional) Normal Distribution Any multi-variate normal distribution N (m, C) is uniquely determined by its mean value m R n and its symmetric positive definite n n covariance matrix C. The mean value m determines the displacement (translation) is the value with the largest density (modal value) the distribution is symmetric about the distribution mean 4 4 probability density.4... Standard Normal Distribution Anne Auger & Nikolaus Hansen () Evolution Strategies 4 / 5 7 Anne Auger & Nikolaus Hansen () Evolution Strategies 6 / 5

5 Evolution Strategies and EDAs The covariance matrix C determines the shape. It has a valuable geometrical interpretation: any covariance matrix can be uniquely identified with the iso-density ellipsoid {x R n x T C x = } Lines of Equal Density Evolution Strategies and EDAs The (µ/µ, λ)-es Non-elitist selection and intermediate (weighted) recombination Given the i-th solution point x i = m + σ N i (, C) }{{} =: y i = m + σ y i Let x i:λ the i-th ranked solution point, such that f(x :λ ) f(x λ:λ ). The new mean reads m µ w i x i:λ = m + σ i= µ w i y i:λ i= } {{ } =: y w N `m, σ I m + σn (, I) one degree of freedom σ components of N (, I) are independent standard normally distributed N `m, D m + D N (, I) N (m, C) m + C N (, I) n degrees of freedom (n + n)/ degrees of freedom components are components are independent, scaled correlated... CMA where w w µ >, µ i= w i = The best µ points are selected from the new solutions (non-elitistic) and weighted intermediate recombination is applied. Anne Auger & Nikolaus Hansen () Evolution Strategies 7 / 5 Anne Auger & Nikolaus Hansen () Evolution Strategies 9 / 5 Evolution Strategies and EDAs Why Evolution Strategies Why? (µ +, λ) µ: # parents, λ: # offspring + selection in {parents} {offspring}, selection in {offspring} ( + )-ES Sample one offspring from parent m x = m + σ N (, C) If x better than m select f(x) = n i= x i in [.,.8] n for n = m x... why? Anne Auger & Nikolaus Hansen () Evolution Strategies 8 / 5 7 Anne Auger & Nikolaus Hansen () Evolution Strategies / 5

6 Why Why Why? Why? f(x) = n i= x i f(x) = n i= x i in [.,.8] n for n = in [.,.8] n for n = Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 Why Why Why? Why?. f(x) = n i= x i in [.,.8] n for n = normalized progress.5..5 normalized step size evolution window for the step-size on the sphere function evolution window refers to the step-size interval where reasonable performance is observed Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 7 Anne Auger & Nikolaus Hansen () Evolution Strategies 4 / 5

7 Methods for Why One-fifth success rule One-Fifth Success Rule /5-th success rule ab, often applied with + -selection increase step-size if more than % of the new solutions are successful, decrease otherwise σ-self-adaptation c, applied with, -selection mutation is applied to the step-size and the better one, according to the objective function value, is selected simplified global self-adaptation path length control d (Cumulative Step-size Adaptation, CSA) e, applied with, -selection a Rechenberg 97, Evolutionsstrategie, Optimierung technischer Systeme nach Prinzipien der biologischen Evolution, Frommann-Holzboog b Schumer and Steiglitz 968. Adaptive step size random search. IEEE TAC c Schwefel 98, Numerical Optimization of Computer Models, Wiley d Hansen & Ostermeier, Completely Derandomized Self-Adaptation in Evolution Strategies, Evol. Comput. 9() Proba of success (p s ) / /5 Proba of success (p s ) too small e Ostermeier et al 994, Step-size adaptation based on non-local use of selection information, PPSN IV Anne Auger & Nikolaus Hansen () Evolution Strategies 5 / 5 Anne Auger & Nikolaus Hansen () Evolution Strategies 7 / 5 One-Fifth Success Rule One-Fifth Success Rule One-fifth success rule One-fifth success rule increase σ decrease σ Let p s : # of successful offspring / generation ( σ σ exp p ) s p target Increase σ if p s > p target p target Decrease σ if p s < p target ( + )-ES p target = /5 IF offspring better parent p s = ELSE p s = Anne Auger & Nikolaus Hansen () Evolution Strategies 6 / 5 7 Anne Auger & Nikolaus Hansen () Evolution Strategies 8 / 5

8 Self-Adaptation Path Length Control Self-adaptation in a (, λ)-es Path Length Control The Equations MUTATEfor i =,...λ step-size σ i σ exp(τ N(, )) parent x i x + σ i N(, I) EVALUATE SELECT Best offspring x with its step-size σ Rationale Unadapted step-size won t produce successive good individuals The step-size are adjusted by the evolution itself Initialize m R n, σ R +, evolution path p σ =, set c σ 4/n, d σ. m m + σy w where y w = µ i= w i y i:λ update mean p σ ( c σ ) p σ + ( c σ ) µw y w }{{}}{{} accounts for c σ accounts for w ( ( )) i cσ p σ σ σ exp d σ E N (, I) update step-size }{{} > p σ is greater than its expectation... CMA in a nutshell Anne Auger & Nikolaus Hansen () Evolution Strategies 9 / 5 Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 Path Length Control The Concept Path Length Control Measure the length of the evolution path the pathway of the mean vector m in the generation sequence x i = m + σ y i m m + σy w Evolution Strategies and EDAs decrease σ increase σ 4 Estimation of Distribution 5 Conclusion loosely speaking steps are perpendicular under random selection (in expectation) perpendicular in the desired situation (to be most efficient) Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 74 Anne Auger & Nikolaus Hansen () Evolution Strategies / 5

9 Rank-One Update m m + σy w, y w = µ i= w i y i:λ, y i N i (, C) covariance matrix adaptation C ( c cov)c + c covµ wy wy T w learns all pairwise dependencies between variables off-diagonal entries in the covariance matrix reflect the dependencies conducts a principle component analysis (PCA) of steps y w, sequentially in time and space eigenvectors of the covariance matrix C are the principle components / the principle axes of the mutation ellipsoid new distribution, C.8 C +. y w y T w the ruling principle: the adaptation increases the probability of successful steps, y w, to appear again learns a new, rotated problem representation and a new metric (Mahalanobis) components are independent (only) in the new representation approximates the inverse Hessian on quadratic functions overwhelming empirical evidence, proof is in progress... equations... cumulation, rank-µ, step-size control Anne Auger & Nikolaus Hansen () Evolution Strategies / 5 Anne Auger & Nikolaus Hansen () Evolution Strategies 5 / 5 Rank-One Update Initialize m R n, and C = I, set σ =, learning rate c cov /n While not terminate x i = m + σ y i, y i N i (, C), µ m m + σy w where y w = w i y i:λ i= C ( c cov )C + c cov µ w y w y T w }{{} rank-one where µ w = µ i= w i Evolution Strategies and EDAs 4 Estimation of Distribution 5 Conclusion Anne Auger & Nikolaus Hansen () Evolution Strategies 4 / 5 75 Anne Auger & Nikolaus Hansen () Evolution Strategies 6 / 5

10 Cumulation The Evolution Path Cumulation Utilizing the Evolution Path Evolution Path Conceptually, the evolution path is the path the strategy takes over a number of generation steps. It can be expressed as a sum of consecutive steps of the mean m. An exponentially weighted sum of steps y w is used p c The recursive construction of the evolution path (cumulation): p c ( c c) {z } decay factor gx i= p c + p ( c c) µ w y w {z } {z} normalization factor ( c c) g i {z } exponentially fading weights input, m m old σ where µ w = P wi, c c. History information is accumulated in the evolution path. Anne Auger & Nikolaus Hansen () Evolution Strategies 7 / 5 y (i) w We used y wy T w for updating C. Because y wy T w = y w( y w) T the sign of y w is neglected. The sign information is (re-)introduced by using the evolution path. where µ w = P wi, c c. p c ( c c) p c + p ( c c) {z } µ w {z } decay factor normalization factor y w... equations Anne Auger & Nikolaus Hansen () Evolution Strategies 9 / 5 Cumulation is a widely used technique and also know as exponential smoothing in time series, forecasting exponentially weighted mooving average iterate averaging in stochastic approximation momentum in the back-propagation algorithm for ANNs... Using an evolution path for the rank-one update of the covariance matrix reduces the number of function evaluations to adapt to a straight ridge from O(n ) to O(n). a a Hansen, Müller and Koumoutsakos. Reducing the Time Complexity of the Derandomized Evolution Strategy with (CMA-ES). Evolutionary Computation, (), pp. -8 The overall model complexity is n but important parts of the model can be learned in time of order n... why?... rank µ update Anne Auger & Nikolaus Hansen () Evolution Strategies 8 / 5 76 Anne Auger & Nikolaus Hansen () Evolution Strategies 4 / 5

11 Rank-µ Update x i = m + σ y i, y i N i (, C), m m + σy w y w = P µ i= w i y i:λ The rank-µ update extends the update rule for large population sizes λ using µ > vectors to update C at each generation step. The matrix µ C µ = w i y i:λ y T i:λ i= computes a weighted mean of the outer products of the best µ steps and has rank min(µ, n) with probability one. The rank-µ update then reads where c cov µ w /n and c cov. C ( c cov ) C + c cov C µ Anne Auger & Nikolaus Hansen () Evolution Strategies 4 / 5 The rank-µ update increases the possible learning rate in large populations roughly from /n to µ w/n can reduce the number of necessary generations roughly from O(n ) to O(n) given µ w λ n Therefore the rank-µ update is the primary mechanism whenever a large population size is used say λ n + The rank-one update uses the evolution path and reduces the number of necessary function evaluations to learn straight ridges from O(n ) to O(n). Rank-one update and rank-µ update can be combined... Hansen, Müller, and Koumoutsakos. Reducing the Time Complexity of the Derandomized Evolution Strategy with (CMA-ES). Evolutionary Computation, (), pp. -8 Anne Auger & Nikolaus Hansen () Evolution Strategies 4 / 5 Estimation of Distribution Estimation of Distribution Algorithms Estimate a distribution that (re-)samples the parental population. All parameters of the distribution θ are estimated from the given population. x i = m + σ y i, y i N (, C) sampling of λ = 5 solutions where C = I and σ = C µ = P yi:λ µ y T i:λ C ( ) C + C µ calculating C where µ = 5, w = = w µ = µ, and c cov = m new m + µ P yi:λ new distribution Example: EMNA (Estimation of Multi-variate Normal Algorithm Initialize m R n, and C = I While not terminate x i = m + y i, y i N i (, C), for i =,...,λ µ C (x i:λ m)(x i:λ m) T m i= µ i= x i:λ Larrañaga and Lozano. Estimation of Distribution Algorithms Anne Auger & Nikolaus Hansen () Evolution Strategies 4 / 5 77 Anne Auger & Nikolaus Hansen () Evolution Strategies 44 / 5

12 Estimation of Distribution Estimation of Multivariate Normal Algorithm EMNA global versus rank-µ CMA 4 What did we achieve? Conclusion x i = m old + y i, y i N (, C) C µ P (xi:λ m new)(x i:λ m new) T m new = m old + µ P yi:λ EMNA global conducts a PCA of points Covariance matrix adaptation: reduce any convex quadratic function f(x) = x T Hx to the sphere model f(x) = x T x x i = m old + y i, y i N (, C) C µ P (xi:λ m old )(x i:λ m old ) T m new = m old + µ P yi:λ rank-µ CMA conducts a PCA of steps sampling of λ = 5 calculating C from µ = 5 new distribution solutions (dots) solutions The CMA-update yields a larger variance in particular in gradient direction, because m new is the minimizer for the variances when calculating C 4 Hansen, N. (6). The CMA Evolution Strategy: A Comparing Review. In J.A. Lozano, P. Larranga, I. Inza and E. Bengoetxea (Eds.). Towards a new evolutionary computation. Advances in estimation of distribution algorithms. pp. 75- Anne Auger & Nikolaus Hansen () Evolution Strategies 45 / 5 without use of derivatives lines of equal density align with lines of equal fitness C H Step-size control: converge log-linearly on the sphere Rank-based selection: the same holds for any g(f(x)) = g ( x T Hx ) g : R R stricly monotonic (order preserving) Anne Auger & Nikolaus Hansen () Evolution Strategies 47 / 5 Conclusion Conclusion Experimentum Crucis () f convex quadratic, separable Evolution Strategies and EDAs 4 5 Conclusion Anne Auger & Nikolaus Hansen () Evolution Strategies 46 / 5 78 abs(f) (blue), f min(f) (cyan), Sigma (green), Axis Ratio (red) f= e ( e ) Standard Deviations of All Variables function evaluations f(x) = i n i= α n x i, α = 6 Object Variables (9D) Scaling (All Main Axes) 4 6 function evaluations Anne Auger & Nikolaus Hansen () Evolution Strategies 48 /

13 Conclusion Experimentum Crucis () f convex quadratic, as before but non-separable (rotated) abs(f) (blue), f min(f) (cyan), Sigma (green), Axis Ratio (red) f= e ( e ) Standard Deviations of All Variables function evaluations Object Variables (9D) Scaling (All Main Axes) 4 6 function evaluations f(x) = g ( x T Hx ), g : R R stricly monotonic C H for all g, H... internal parameters Anne Auger & Nikolaus Hansen () Evolution Strategies 49 / 5 Conclusion Invariance Under Strictly Monotonically Increasing Functions Rank-based algorithms Selection based on the rank: f(x :λ ) f(x :λ )... f(x λ:λ ) Update of all parameters uses only the rank g(f(x :λ )) g(f(x :λ ))... g(f(x λ:λ )) Anne Auger & Nikolaus Hansen () Evolution Strategies g preserves rank 5 / 5 79