Evolutionary Computation: introduction

Transcription

1 Evolutionary Computation: introduction Dirk Thierens Universiteit Utrecht The Netherlands Dirk Thierens (Universiteit Utrecht) EC Introduction 1 / 42 What? Evolutionary Computation Evolutionary Computation = Population-based, stochastic search algorithms based on the mechanisms of natural evolution Evolution viewed as search algorithm or problem solver. EC part of Computational Intelligence. Note: Natural evolution only used as metaphor for designing computational systems. No model building of natural evolution ($ evolutionary biology). Dirk Thierens (Universiteit Utrecht) EC Introduction 2 / 42

2 Evolutionary Computation Darwinian process characteristics 5 Key requirements of a Darwinian system 1 Structures 2 Structures are copied 3 Copies partially vary from the original 4 Structures are competing for a limited resource 5 Relative reproductive success depends on the environment Dirk Thierens (Universiteit Utrecht) EC Introduction 3 / 42 Evolutionary Computation Darwinian process characteristics in EC 1 Structures ) e.g. binary strings, real-valued vectors, programs,... 2 Structures are copied ) selection algorithm: e.g. tournament selection,... 3 Copies partially vary from the original ) mutation & crossover operators 4 Structures are competing for a limited resource ) selecting fixed sized parent pool 5 Reproductive success depends on environment ) user defined fitness function Dirk Thierens (Universiteit Utrecht) EC Introduction 4 / 42

3 Evolutionary Computation Dirk Thierens (Universiteit Utrecht) EC Introduction 5 / 42 Evolutionary Computation Dirk Thierens (Universiteit Utrecht) EC Introduction 6 / 42

4 Genetic Algorithm Genetic Algorithms: structures are discrete representations. Neo-Darwinism organism *...AUUCGCCAAU... Genetic Algorithm f: < * * user: string representation and function f * GA: string manipulation I selection: copy better strings I variation: generate new strings Dirk Thierens (Universiteit Utrecht) EC Introduction 7 / 42 Genetic Algorithm selection: copy better strings I tournament selection I truncation selection I proportionate selection variation: generate new strings 1 crossover point crossover: ) uniform crossover: ) mutation { ) { Dirk Thierens (Universiteit Utrecht) EC Introduction 8 / 42

5 Toy example Genetic Algorithm x [0, 31] : f (x) =x 2 binary integer representation: x i {0, 1} x = x x x x x Initial Random Population: : 18 2 = : 12 2 = : 9 2 = : 20 2 = : 8 2 = : 7 2 = 49 population mean fitness f (0) = 177 Dirk Thierens (Universiteit Utrecht) EC Introduction 9 / 42 Genetic Algorithm Generation 1: tournament selection, 1-point crossover, mutation Parents Fitness Offspring Fitness 100! ! ! ! ! ! Parent population mean fitness f (1) =383 Dirk Thierens (Universiteit Utrecht) EC Introduction 10 / 42

6 Genetic Algorithm Generation 3: Parents Fitness Offspring Fitness 1! ! ! ! ! ! Parent population mean fitness f (3) =762 Dirk Thierens (Universiteit Utrecht) EC Introduction 11 / 42 Permutation Representation Permutation problems Representations are permutations of a set of elements. Need suitable representations and genetic operators for permutation or sequencing problems Examples I scheduling I vehicle routing I queueing I... Dirk Thierens (Universiteit Utrecht) EC Introduction 12 / 42

7 Permutation Representation Traveling salesman problem Find the shortest route while visiting all cities exactly once. Dirk Thierens (Universiteit Utrecht) EC Introduction 13 / 42 Permutation Representation Permutation problems Tour representation: list of cities? I p1 = I p2 = I simple crossover ) illegal tours I c1 = I c2 = alternative search space representation alternative genetic operators different aspects in different ordering problems: I adjacency I relative order I absolute order ) whole set of permutation crossover operators proposed! Dirk Thierens (Universiteit Utrecht) EC Introduction 14 / 42

8 Cycle crossover Permutation Representation p1: A B C D E F G H I p2: f c d a e b h i g cy: ch: A B C D E F h i g 1 mark cycles 2 cross full cycles ) emphasizes absolute position above adjacency or relative order Dirk Thierens (Universiteit Utrecht) EC Introduction 15 / 42 Edge recombination Permutation Representation Edge map: Table of all cities with their direct neighbors in both parents. edge map: Two parent tours [ABCDEF] & [BDCAEF] city A B C D E F edges BFCE ACDF BDA CEB DFA AEB Dirk Thierens (Universiteit Utrecht) EC Introduction 16 / 42

9 Permutation Representation Edge recombination algorithm: 1 choose initial city from one parent 2 remove current city from edge map 3 if current city has remaining edges goto step 4 else goto step 5 4 choose current city edge with fewest remaining edges 5 if still remaining cities, choose one with fewest remaining cities Dirk Thierens (Universiteit Utrecht) EC Introduction 17 / 42 Permutation Representation 1 random choice ) B 2 next candidates: A C D F choose from C D F (same edge number) ) C 3 next candidates: A D (edgelist D < edgelist A) ) D 4 next candidate: E ) E 5 next candidates: A F tie breaking ) A 6 next candidate: F ) F Resulting tour = [BCDEAF] All edges in the child tour are inherited from one of the parent tours [ABCDEF] & [BDCAEF]! Dirk Thierens (Universiteit Utrecht) EC Introduction 18 / 42

10 Permutation Representation Fitness correlation coefficients What variation operator is most suitable for a given problem? Genetic operators should preserve useful fitness characteristics between parents and offspring = inheritance. Calculate the fitness correlation coefficient to quantify this. K-ary operator: generate n sets of k parents. Apply operator to each set to create children. Compute fitness of all individuals. {f (p g1 ), f (p g2 ),...,f (p gn } {f (c g1 ), f (c g2 ),...,f (c gn } Dirk Thierens (Universiteit Utrecht) EC Introduction 19 / 42 Permutation Representation Fitness correlation coefficients F p : mean fitness of the parents F c : mean fitness of the children (F p ) = standard deviation of fitness parents (F c ) = standard deviation of fitness children cov(f p, F c )= P n (f (p gi ) F p )(f (c gi ) F c ) i=1 n covariance between fitness parents and fitness children Operator fitness correlation coefficient op : op = cov(f p, F c ) (F p ) (F c ) Dirk Thierens (Universiteit Utrecht) EC Introduction 20 / 42

11 Permutation Representation Traveling Salesman problem: crossover operators various crossover operators applicable: I cycle crossover (CX) I partially matched crossover (PMX) I order crossover (OX) I edge crossover (EX) performance: EX > OX > PMX > CX crossover correlation coefficients cross : EX 0.90 OX 0.72 PMX 0.61 CX 0.57 Dirk Thierens (Universiteit Utrecht) EC Introduction 21 / 42 Dirk Thierens (Universiteit Utrecht) EC Introduction 22 / 42

12 Dirk Thierens (Universiteit Utrecht) EC Introduction 23 / 42 Dirk Thierens (Universiteit Utrecht) EC Introduction 24 / 42

13 Evolving a Checkers Player Can we build intelligent systems to learn to play checkers? No expert knowledge provided to the learning system. Programs simply have to play against themselves, and figure out how to play. Evolutionary computation feasible approach? Dirk Thierens (Universiteit Utrecht) EC Introduction 25 / 42 Game playing Board representation Move search: minimax algorithm Traditional game playing programs: board evaluation functions are extensively knowledge based 1 weighted feature function 2 opening games 3 end games table look-up ) they do not learn by themselves! Here: Evolving Neural Networks to Play Checkers Dirk Thierens (Universiteit Utrecht) EC Introduction 26 / 42

14 Board representation Output 2 [ ] -1: loss positions +1: win positions! closer to +1 ) better evaluations Input: vector of 32 possible positions, 5 possible values 1 - K : king opponent 2-1 : checker opponent 3 0 : empty : checker self 5 + K : king self K 2 [1...3] : exact value evolved Dirk Thierens (Universiteit Utrecht) EC Introduction 27 / 42 Mini-Max algorithm Dirk Thierens (Universiteit Utrecht) EC Introduction 28 / 42

15 Game lookahead The further we can lookahead the better. Computational restrictions: number of possible board positions grows very fast with increasing number of lookahead levels. Deep Blue when defeating chess champion Garry Kasparov made 200 million chess board evaluations per second! Here only lookahead search of 2 moves each side when evolving. When testing against players on Internet: lookahead search of 3 moves each side. Dirk Thierens (Universiteit Utrecht) EC Introduction 29 / 42 Board Evaluation Evaluation function represented by an artificial neural network Dirk Thierens (Universiteit Utrecht) EC Introduction 30 / 42

16 Neural Network Architecture Input Layer: 32 inputs First hidden layer: 40 neurons Second hidden layer: 10 neurons Direct input-output connections with weight 1.0 Total number of neural network weights (incl. bias term) = (32 + 1) x 40 + (40 + 1) x 10 + (10 + 1) x 1 = 1741 Need to determine values for the 1741 weights + Use evolutionary algorithm to co-evolve the weights Dirk Thierens (Universiteit Utrecht) EC Introduction 31 / 42 Evolutionary search for network weights Mutate neural network weights by adding a small random, Gaussian distributed number to each weight. Each weight has its own Gaussian distribution (different widths or standard deviations). The width or variance of each Gaussian distribution also evolves by mutation. For each neural network NN i all the N w (= 1741) neural network weights w i (j) (j = 1...N w ) gets associated with the corresponding standard deviation i (j) (j = 1...N w ) of the Gaussian mutation distribution (mean value is always zero). Dirk Thierens (Universiteit Utrecht) EC Introduction 32 / 42

17 Self-adaptive Mutation First mutate the N w widths of the Gaussian distributions, then mutate the N w weights (j = 1...N w ): Self-adaptive mutation of the mutation step-size: 0 i (j) = i(j) exp( RandNorm j(0,1) p2 p N w ) Mutation of the neural network weights: w 0 i (j) =w i(j)+ 0 i (j)randnorm j(0, 1) Dirk Thierens (Universiteit Utrecht) EC Introduction 33 / 42 Evolutionary Cycle Dirk Thierens (Universiteit Utrecht) EC Introduction 34 / 42

18 Fitness evaluation 15 parents + 15 offspring neural networks Each NN competes against 5 randomly chosen NN Score: win : + 1; draw : 0; loss : -2 Fitness: sum of scores Dirk Thierens (Universiteit Utrecht) EC Introduction 35 / 42 Darwinian system? 1 Structures?! neural networks 2 Structures are copied?! 15 parents ) 15 offspring 3 Copies partially vary from the original! weights Gaussian mutated 4 Structures are competing for a limited resource! fixed population size each generation 5 Reproductive success depends on environment! winning strategies survive Dirk Thierens (Universiteit Utrecht) EC Introduction 36 / 42

19 Experiment total of 250 generations evolved 15 neural networks each generation ) 15 x 250 = 3750 neural networks created fitness evaluation: 15 parents + 15 offspring: each competing against 5 others ) 30 x 5 x 250 = games played Dirk Thierens (Universiteit Utrecht) EC Introduction 37 / 42 Checkers Rating Dirk Thierens (Universiteit Utrecht) EC Introduction 38 / 42

20 Evolved neural network rating: 1902 Dirk Thierens (Universiteit Utrecht) EC Introduction 39 / 42 1 draw against player rated 2207, ie. master level, ranked 18 out of listed players Dirk Thierens (Universiteit Utrecht) EC Introduction 40 / 42

21 Discussion Chinook: opening games, end games table look-up, hand-crafted evaluation function Deep Blue: 200 million board evaluations per second vs here Payoff: summed score over 5 games! no immediate feedback about winning or losing a single game Input to neural networks do not give spatial information: only 1 x 32 vector of {-K,-1,0,1,K} Allen Newell: It is extremely doubtful whether there is enough information in win, lose, or draw when referred to the whole play of the game to permit any learning at all over available time scales Dirk Thierens (Universiteit Utrecht) EC Introduction 41 / 42 Conclusion Building intelligent systems by evolutionary computing. Learn to play checkers at high level by competing against themselves. No tedious domain knowledge extraction. Learning by evolution: feasible approach. Dirk Thierens (Universiteit Utrecht) EC Introduction 42 / 42