Competitive Co-evolution Robert Lowe Motivations for studying Competitive co-evolution Evolve interesting behavioural strategies - fitness relative to opponent (zero sum game) Observe dynamics of evolving populations (evolutionary stability, e.g. evolutionary game theory). Motivations for studying Competitive co-evolution Behavioural Strategies: increasing complexity in behavioural strategies greater variety of strategies Dynamics of co-evolving populations: oscillatory dynamics helps to avoid local minima may be a powerful means for discovering optimized solutions stable strategies vs stable ratios (Maynard Smith, 1982 Evolution and the Theory of Games ) 1
Co-evolution for discovery of interesting behavioural strategies Karl Sims Creatures (1994): Evolution of competitive strategies regarding occupation of a block. Co-evolution of behavioural strategies Sims (1994) All contests are 1 on 1. A number of different ways of pairing competitors of different populations are used: see figure 2 (from Sims (1994)) Co-evolution of behavioural strategies - Sims (1994) 2
Dynamics of Competitive coevolution - The Red Queen Effect The Red Queen was a living chess piece in Lewis Carroll s Through the looking glass, who ran perpetually without getting very far because the landscape kept up with her (Cliff & Miller 1995) Fitness co-dependence and Arms races competitive co-evolution (CCE) ~ the evolution of two or more competing populations with coupled fitness e.g. predator - prey may enhance the power of artificial evolution evolutionary arms races ~ competing populations can drive each other to increasing levels of behavioral complexity a pedagogical series of challenges that gradually require more complex solutions hypothesized by biologists to be one of the main sources of evolutionary innovation and adaptation BUT CCE can create trivial oscillations between simple behaviours Incremental Evolution Vs CCE Incremental Evolution is a gradual modification of the fitness function or the environment but: requires careful attention and planning by the experimenter Natural (and possibly artificial) CCE, on the other hand, The environment contains another network that is also evolving So the task effectively faced by each species can become increasingly more complex Throughout evolution, agents face opponents that use different strategies Therefore they must themselves develop different counterstrategies or more general abilities 3
The Red Queen Difficult to measure/assess progress of the co-evolutionary process any solution/strategy found in one generation could be no longer valid in later ones the fitness landscape changes all the time (!) difficult to monitor progress with conventional indicators such as best and average fitness values Monitoring CCE Techniques that have been suggested CIAO (Current Individual vs. Ancestral Opponents) (Cliff & Miller, 1995) - testing the performance of the best (elite) individual of each generation against the best competing ancestors from each previous generation an Evolutionarily Stable Strategy (ESS)? Master tournament (F & N, 1997) - testing against all best opponents, even future ones (i.e. only possible after the evolutionary process) Hall of fame tournament (Rosin & Belew 1997) adopt elitism to encourage arms races. Predator-Prey Experiments 2 Khepera robots predator has vision prey is twice as fast 64 pixel linear scanner with 36 o field of view. Reduced to a 5 bit visual field 4
Predator-Prey Experiments Fitness Measures Fitness graphs show oscillatory trends, but do not inform us of: Evolutionary progress An appropriate optimization approach Frequency dependency strategies can account for oscillatory patterns Predators discover and rediscover two classes of strategy (all replications): A1 - track the prey and try to catch it A2 - track the prey while remaining in one area and attacking only on promising occasions Prey cycles between: B1 - stay still or hide near a wall; try to escape when detecting the predator B2 - move fast, avoiding both predator and walls A1 > B1 (late detection); B2 > A1 (faster prey); A2 > B2 (waiting for too fast prey); B1 > A2 (let the predator wait ) 5
Monitoring Co-Evolution In these experiments agents repeatedly switched between two opposing strategy sets and evolution did not really progress but It is claimed that co-evolution should be able to enhance artificial evolution by adding incrementally through arms races promoting generality Changing landscapes reduce the local minima problem 6
Master Tournament Like CAIO but elites compete against future elites too. (filling in the CAIO square) This allows us to plot the actual fitness relative to past and future agents Hall of Fame Every individual of this generation competes against the elite of previous generations to ascertain their fitness for reproduction Master Tournament plot for the Individuals generated by a hall of fame selection strategy CCE vs Individual Population Evolution: Evolving One Species Only 7
Co-Evolution and Learning Evolutionary Adaptation: No learning. Ontogenetic Adaptation: Genotype encodes sign, learning rate, learning rule but weights are always initialized to small random values in all runs predators show higher average and best fitness values Co-Evolution and Learning plastic predators adapt their strategies during their lifetime (e.g. center) almost all of them can adapt to B1 and B2 the prey usually do not (or cannot because of sensorimotor limitations) because they have less time and less information for learning when learning was optional predators evolved to learn, prey did not (!) Summary Advantages of Competitive Co-Evolution (CCE): allows the study of adaptation in a changing environment, this sometimes promotes a greater diversity of behavioural strategies. under some conditions, may produce non-supervised incremental evolution (when a general solution exists). Disadvantage of CCE: May find limit cycles. Possible solution to cycling problem: Elitism but may limit evolutionary possibilities constrained by evolutionary history, biologically unrealistic CCE can drive towards general strategies exploited further by learning. Caveat: Constraints imposed by design or by body sensor morphology have a strong effect on the behaviours evolved 8
Group Discussion We want to evolve general strategies that work well in a variety of situations. This may require search and refinement of agent strategies rather than have the agents constantly switch between different strategies How can we promote this? What other factors might aid co-evolution? How would you extend this work? (think about nature) References Nolfi & Floreano (2000) Evolutionary Robotics. Cambridge, MA: MIT Press. - chapter 8 Cliff D. &Miller G.F (1995) Tracking the Red Queen: Measurements of adaptive progress in co-evolutionary simulations. In F. Moran, A. Moreno, JJ. Merelo, and P. Chacon (Eds.) Advances in Artificial Life: Proceedings of the third european conference on artificial life. Berlin: Springer Verlag. Floreano & Mondada (1998). Evolutionary neurocontrollers for autonomous mobile robots. Neural Networks, 11(7-8). Predator/Prey videos: http://gral.ip.rm.cnr.it/nolfi/predprey/ K.Sims, Artificial Life IV Proceedings, ed.by Brooks & Maes, MIT Press, 1994, pp.28-39. K.Sims video: http://www.archive.org/details/sims_evolved_virtual_creatures_1994 9