Evolutionary branching under slow directional evolution

Transcription

1 International Institute for Applied Sstems Analsis Schlossplatz A- Laxenburg, Austria el: + Fax: + publications@iiasa.ac.at Web: Interim Report IR-- Evolutionar branching under slow directional evolution Hiroshi C. Ito Ulf Dieckmann (dieckmann@iiasa.ac.at) Approved b Pavel Kabat Director General and Chief Executive Officer June Interim Reports on work of the International Institute for Applied Sstems Analsis receive onl limited review. Views or opinions expressed herein do not necessaril represent those of the Institute, its National Member Organizations, or other organizations supporting the work.

2 Evolutionar branching under slow directional evolution Hiroshi C. Ito and Ulf Dieckmann Evolution and Ecolog Program, International Institute for Applied Sstems Analsis, Schlossplatz, A- Laxenburg, Austria Department of Evolutionar Studies of Biosstems, he Graduate Universit for Advanced Studies (Sokendai), Haama -, Kanagawa, Japan Corresponding author: Hiroshi C. Ito Phone + Fax + addresses: H. C. Ito, hiroshibeetle@gmail.com, U. Dieckmann, dieckmann@iiasa.ac.at Highlights + We derive conditions for evolutionar branching in directionall evolving populations. + he derived conditions extend those for univariate trait spaces to bivariate trait spaces. + Numerical analses demonstrate their robustness. + Our conditions are further extended to multivariate trait spaces. Page of

3 Abstract Evolutionar branching is the process b which ecological interactions induce evolutionar diversification. In asexual populations with sufficientl rare mutations, evolutionar branching occurs through trait-substitution sequences caused b the sequential invasion of successful mutants. A necessar and sufficient condition for evolutionar branching of univariate traits is the existence of a convergence stable trait value at which selection is locall disruptive. Real populations, however, undergo simultaneous evolution in multiple traits. Here we extend conditions for evolutionar branching to bivariate trait spaces in which the response to disruptive selection on one trait can be suppressed b directional selection on another trait. o obtain analtical results, we stud trait-substitution sequences formed b invasions that possess maximum likelihood. B deriving a sufficient condition for evolutionar branching of bivariate traits along such maximum-likelihood-invasion paths (Ps), we demonstrate the existence of a threshold ratio specifing how much disruptive selection in one trait direction is needed to overcome the obstruction of evolutionar branching caused b directional selection in the other trait direction. Generalizing this finding, we show that evolutionar branching of bivariate traits can occur along evolutionarbranching lines on which residual directional selection is sufficientl weak. We then present numerical analses showing that our generalized condition for evolutionar branching is a good indicator of branching likelihood even when trait-substitution sequences do not follow Ps and when mutations are not rare. Finall, we extend the derived conditions for evolutionar branching to multivariate trait spaces. Kewords frequenc-dependent selection, speciation, adaptive dnamics, two-dimensional traits, multi- dimensional traits Page of

4 Introduction Real populations have undergone evolution in man quantitative traits. Even when such populations experience contemporar selection pressures, their selection response will usuall be highl multivariate. However, not all responding adaptive traits evolve at the same speed: in nature, such evolutionar speeds exhibit a large variation (Hendr and Kinnison, ; Kinnison and Hendr, ).). Past speciation processes ma have been driven mainl b traits undergoing fast evolution (Schluter, ), while gradual evolutionar differentiation among species, genera, and families ma derive from traits undergoing slow evolution. hese differences in evolutionar speed can have two fundamentall different causes. First, the ma be due to less genetic variation being available for evolution to act on: in asexual populations this occurs when mutation rates and/or magnitudes are smaller in some traits than in others, while in sexual populations this occurs when standing genetic variation is smaller in some traits than in others. Second, differences in evolutionar speed are also expected when fitness is much less sensitive to changes in some traits than to changes in others. For conveniencebrevit, we refer to the slowl evolving and the fastrapidl evolving traits as slow traits and fast traits, respectivel. If the slow traits are sufficientl slow, it is tempting to neglect their effects on the evolution of the fast traits. As far as evolutionar responses to directional selection are concerned, this simplification is usuall unproblematic: directional evolution, i.e.,the directional trend of evolution (Rice, ) resulting from such selection, which we can refer to as directional evolution,, is described effectivel b ordinar differen- o facilitate the reviewing process, we adopt the author-date stle for citations. Naturall, we will immediatel change these citations to numbers when it is suggested or once our manuscript is accepted. Page of

5 tial equations or difference equations focusing onl on those fast traits (Price ; Lande, ; Dieckmann and Law ; Rice, ).). On the other hand, such a simplification ma not be safe where more complex evolutionar dnamics are involved. A tpical example is adaptive speciation, i.e., evolutionar diversification driven b ecological interactions (Dieckmann et al., ; Rundle and Nosil, ). Ito and Dieckmann () have shown numericall that when populations undergo disruptive selection in thea fast trait, their evolutionar diversification can be suppressed b directional evolution of a slowanother trait, even if the latter is slow. Conversel, if the slow directional evolution is sufficientl slow, disruptive selection in the fast trait can drive evolutionar diversification, both in asexual populations and in sexual populations (Ito and Dieckmann, ). he suppression of evolutionar diversification can occur even when the slow and fast traits are mutationall and ecologicall independent of each other. hus, in a multivariate trait space, evolutionar diversification in one trait can be suppressed b slow directional evolution in just one of the man other traits. Moreover, such slow directional evolution ma never cease, as the environments of populations are alwas changing, at least slowl, due to changes in abiotic components (e.g., climatic change) or biotic components (i.e., evolution in other species of the considered biological communit). It is therefore important to improve the theoretical understanding of this phenomenon b deriving conditions for evolutionar diversification under slow directional evolution. As a starting point to this end, we can consider the special situation in which there is onl a single fast trait, while all other traits of the considered population are evolving extremel slowl, such that the are completel negligible. In this case, the question whether the selection on the fast trait favors its evolutionar diversification can be examined through Page of

6 conditions that have been derived for the evolutionar branching of univariate traits (Metz et al., ; Geritz et al.,, ). In general, evolutionar branching is the process through which a unimodal phenotpe distribution of a population becomes bimodal in response to frequenc-dependent disruptive selection (Metz et al., ; Geritz et al.,, ; Dieckmann et al., ), which can occur through all fundamental tpes of ecological interaction, including competition, predator-pre interactionexploitation, mutualism, and cooperation (Doebeli and Dieckmann, ; Doebeli et al., ). his kind of diversifing evolution provides ecological underpinning for the smpatric or parapatric speciation of sexual populations (e.g., Doebeli, ; Dieckmann and Doebeli, ; Kisdi and Geritz, ; Doebeli and Dieckmann, ; Dieckmann et al., ; Claessen et al., ; Durinx and Van Dooren, ; Heinz et al., ; Pane et al., ). Moreover, evolutionar branching ma lead to selection pressures that favor further evolutionar branching, inducing recurrent adaptive radiations and extinctions (e.g., Ito and Dieckmann, ), and thus communit assembl (e.g., Jansen and Mulder, ; Bonsall et al., ; Johansson and Dieckmann, ; Brännström et al., ) and food-web formation (e.g., Loeuille and Loreau, ; Ito et al., ; Brännström et al., ; akahashi et al., in press). herefore, evolutionar branching ma be one of the important mechanisms underling the evolutionar diversification of biological communities. Conditions for the evolutionar branching of univariate traits can be extended to bivariate trait spaces, if all traits considered evolve at comparable speeds (Bolnick and Doebeli, ; Vukics et al., ; Ackermann and Doebeli, ; Van Dooren et al., ; Egas et al., ; Leimar, ; Van Dooren, ; Ito and Shimada, ; Ravigné et al., ). However, if their evolutionar speeds are significantl different, the resultant conditions for bivariate traits can fail to predict evolutionar branching observed in numerical analses (Ito and Dieckmann, ; Ito et al.,, Ito and Dieckmann, ). In the present stud, we therefore derive conditions for a population s evolutionar branching in a fast trait when, at the same time, such a population is directionall evolving in one or more slow traits. he resultant conditions reveal when slow directional evolution either prevents or permits evolutionar branching. Page of

7 his paper is structured as follows. Section explains heuristicall how the likelihood of evolutionar branching in asexual populations depends on selection pressures and mutational step sizes. Section derives a normal form for strong disruptive selection and weak directional selection in a bivariate trait space and explains when arbitrar bivariate fitness functions can be mapped onto this normal form. Section introduces the concept of maximum-likelihood-invasion paths, formed b mutants with maximum likelihood of invasion. On that basis, Section derives sufficient conditions for evolutionar branching. Section numericall examines the robustness of these conditions when the simplifing assumptions underling our derivation are relaxed. Section summarizes all conditions needed for identifing evolutionar-branching lines and extends these conditions to multivariate trait spaces. Section discusses how our results generalize previousl derived conditions for evolutionar branching that ignored slow directional evolution, and how our maximum-likelihood-invasion paths are related to existing methods for determining evolutionar dnamics or reconstructing evolutionar histories. Heuristics We start b describing, in a heuristic wa, how disruptive selection in one direction, directional selection in the otheranother direction, and mutational step sizes ma affect the likelihood of evolutionar branching. We then explain the analses required for deriving the conditions for evolutionar branching, which are conducted in the subsequent sections. When a population undergoes disruptive selection in trait x, as well as directional selec- tion in trait, its fitness landscape resembles that illustrated in Fig. a. he strength of dis- ruptive selection in x is given b the fitness landscape s curvature (i.e., second derivative) along x, denoted b D xx, while the strength of directional selection in is given b the fitness landscape s steepnessslope (i.e., first derivative) along, denoted b G. For sim- plicit, we assume that the population is monomorphic with a resident phenotpe ( x,, ) in- dicated b a small black circle in Fig. a, and that mutational step sizes are identical in all directions. In this case, possible mutants are located on a circle around the resident phenotpe, Page of

8 as shown in Fig. b-g. hen, small G means slow evolution in. Roughl speaking, the direction of evolution favored b selection is indicated b the mutants possessing maximum fitness (small white circles in Fig. b-g). hese mutants are located where the circle of con- sidered mutants is tangential to the fitness contours. From this simple setting, we can alread draw the following geometricall evident conclusions. If G is large compared to D xx, which results in low curvatures for the fitness contours (Fig. b), the mutant having the maximum has maximum fitness, in which case directional evolution along is expected. On the other hand, if compared to G is sufficientl small D xx (Fig. c, d), the high curvatures mean that two different mutants are sharing the same maximum fitness. In this case, evolutionar diversification in x ma be expected. In addition, we can easil see (Fig. e-g) that the smaller the mutational step size, the smaller the G and/or larger the maximum fitness (Fig. e-g). D xx required for two different mutants jointl having It turns out that these qualitative and heuristic insights can be corroborated b formal analsis (Sections -). For this, two things have to be done properl. First, we have to clarif the conditions under which a population undergoes disruptive selection in one direction and sufficientl weak directional selection in the other direction. o compare the strengths of selection among different directions, trait spaces have to be normalized so that mutation becomes isotropic in all directions, as in Fig. b-g. Second, because the existence of disruptive selection is a necessar but not a sufficient condition for evolutionar branching (Metz et al., ; Geritz et al.,, ), the emergence of an initial dimorphism and the subsequent process of divergent evolution have to be analzed. Conducting these analses in the subsequent sections, we end up being able quantitativel to predict the likelihood of evolutionar branching in terms of G, D xx, and mutational step sizes. Page of

9 Normal form for bivariate invasion-fitness functions causing slow directional evolution In this section, we first derive a normal form that applies when evolution is slow in one direction. As mentioned before, this ma occur when mutational steps or fitness sensitivities are strongl asmmetrical. Second, we explain the evolutionar dnamics that are expected under this normal form. hird, we outline the fundamental ideas underling our subsequent analses.. InvasionBivariate invasion-fitness functions causing slow directional evolution We start b considering arbitrar bivariate trait spaces; accordingl, each phenotpe S ( XY, ) comprises two scalar traits X and Y. We assume an asexual population with a large population size and sufficientl small mutation rates. he latter assumption has two consequences. First, the population dnamics have sufficient time to relax toward their equilibrium after a new mutant emerges. Second, as long as the population experiences directional selection, onl the phenotpe with the highest fitness among the existing phenotpes survives as a result of selection. hus, the population is essentiall alwas close to equilibrium and monomorphic. his allows its directional evolution to be translated into a trait-substitution sequence based on the invasion fitness of a mutant phenotpe S arising from a resident phe- notpe S (Metz et al.,, ; Dieckmann and Law, ). he invasion fitness of S under S, denoted b F( S; S ), is defined as the initial per capi- ta growth rate of S in the monomorphic population of S at its equilibrium population size. he function F( S; S ) can be treated as a fitness landscape in S, whose shape depends on S. When a mutant emerges, which occurs with probabilit per birth, we assume that its phenotpe follows a mutation probabilit distribution denoted b M ( S ), where SS S. he distribution is assumed to be smmetric, unimodal, and smooth. As long as the mutation- al step sizes are sufficientl small, such that M ( S ) has sufficientl narrow width, the distri- bution is well characterized b its variance-covariance matrix, Page of

10 VXX VXY M( )dx dy, () VXY VYY where V X M X Y, XX ( )d d V Y M X Y, and YY ( )d d VXY XY M( )dx dy. he standard deviation of mutational step sizes along each direction is thus described b an ellipse, S S, as shown in Fig. a. Since is smmetric, its two eigenvectors are orthogonal, and these vectors determine the directions of the long and short principal axes of this ellipse. hrough a coordinate rotation, we align the axes of the coordinate sstem with the eigenvectors of. In the rotated coordinate sstem, is diagonal, V XY, and we can choose the axes such that X Y, where X VXX and Y VYY (Fig. b). hen, stretching the trait space in the Y -direction b X / Y gives an isotropic mutation distribution with standard deviation c). his is achieved b introducing a new coordinate sstem X Y in all directions (Fig. X s ( x, ), where x X and ( / ) Y. We assume that is small, such that. In this normalized trait space, the invasion-fitness function can be expanded in s and s around a base point as s ( x, ) f ( s; s) Gs( ss) Cs s Ds O( ), (a) where ss s O( ) and ss O( ) are assumed. he G, C, and D are given b Cxx Cx Dxx Dx G Gx Gfm, C fmm frm, D fmm, Cx C Dx D f f f f f f f f f f fx fsss x x x xx x xx x m, mm, rr, rm fx f fx f fx f s s s s s s s s s (b) where the subscripts m and r refer to mutants and residents, respectivel, and where for x,, x, and f for, x,, x, denote the first and second derivatives of f f ( s; s ), respectivel. See Appendix A for the derivation of Eqs. (). Notice that G, C, and D are functions of the base point s. he vector G describes the fitness gradient at s when s s. he matrix C describes how the fitness gradient at s changes as s deviates Page of

11 from s. he matrix D describes the curvature (i.e., second derivative) of the fitness landscape, which is approximatel constant as long as the third-order terms can be neglected. If Y is much smaller than X, such that O( ), the stretching of the trait space in the Y -direction for the normalization makes derivatives with respect to ver small, in the sense G O( ), Cx O( ), Cx O( ), Dx O( ), C his simplifies Eq. (a), Y X O( ), and D O( ). f( s; s) GxxCxx ( xx ) x Dxxx G O( ), () where terms with O( ) C x, C x, C, D x, and D are subsumed in the higher-order terms. For a derivation of Eq. (), see Appendix B. Notice that on the right-hand side onl the first term is of order O( ), while the other terms, including G, are of order his means that this normal form describes fitness functions with significantl weak directional selection along compared to that along x as long as G X and G Y in the original trait space have similar magnitudes. We call Eq. () the normal form for invasion- fitness functions with significant sensitivit difference. O( ). A comparison of Eqs. (a) and () shows that the latter can be obtained even for Y X, provided the sensitivit of the fitness function to variation in trait Y is significantl weaker than that in X, so that satisfing G, C x, C x, C, D x, and D are all relativel small, G C C C D D x x x G C D x xx xx O( ). () For a derivation of Eq. (), see Appendix B. Notice that the assumption needed for the derivation of Eq. (), i.e., Y O( X), also satisfies Eq. (). hus, the condition for a significant difference between mutational step sizes in the original trait space S can naturall be integrated with the condition for a significant sensitivit difference of the invasion-fitness function in the normalized trait space s. Based on Eq. (), we therefore define the condition for significant sensitivit difference as follows. Page of

12 Significant sensitivit difference: After normalization to make mutation isotropic, the invasion-fitness function can be made to satisf Eq. () b rotating the x - and - axes.. Evolutionar dnamics expected under normal form We now consider the expected evolutionar dnamics induced b the normal form in Eq. (). For this purpose we first recap expectations for the simpler case in which G O( ). In that case, G O( ) G is so small that is negligible, so that vanishes from Eq. (), and the evolutionar dnamics therefore become univariate in x, so that phenotpes are characterized b that trait value alone. In this simpler case, conditions for evolutionar branching are easier to understand (Metz et al., ; Geritz et al.,, ), as follows. Suppose that the base point x of the expansion of f ( s; s ) can be chosen such that Gx. Such a point, which we denote b x b, is called an evolutionaril singular point (or simpl a singular point or an evolutionar singularit) because a resident located at experiences no directional selection. In contrast, a resident located close to directional selection along x, x b x b experiences f ( s; s) x xx C ( xx ). () xx b If C xx is negative, the fitness gradient is positive for x xb and negative for x xb, which means that it attracts a monomorphic population through directional evolution. In other words, the singular point is then convergence stable (Christiansen, ). When a population comes close to x b, it ma become possible for a mutant s to coexist with a resident s. Mutual invasibilit between s and s, which gives rise to protected di- morphism (Prout, ), is defined b f ( s; s ) and f (; ss ), which requires D C. Following the emergence of a protected dimorphism of trait values denoted b xx xx s and s, the resultant fitness landscape f ( s; s, s ) maintains approximatel the same curvature (i.e., second derivative) D xx along x. If this curvature is positive, i.e., this point is not a local evolutionaril stable strateg, or ESS (Manard Smith and Price, ), the two Page of

13 subpopulations evolve in opposite directions, keeping their coexistence, in a process called dimorphic divergence. When dimorphic divergence occurs in univariate trait spaces, it can never collapse (if it is assumed that mutual invasibilit among phenotpes ensures their coex- istence). his is because onl mutants outside of the interval between the two residents can invade, and such an invading mutant then alwas excludes the closer resident and is mutuall invasible with the other more distant resident, resulting in a new protected dimorphism with a larger phenotpic distance (Geritz et al., ). As we will see below, such collapses, howev- er, become crucial when analzing dimorphic divergence in bivariate trait spaces. he evolutionar process described above is called evolutionar branching. It requires monomorphic convergence ( Cxx ), mutual invasibilit ( Dxx C xx ), and dimorphic divergence ( Dxx ). We therefore see that the necessar and sufficient conditions for univariate evolutionar branching are given b the existence of a point x x satisfing b Gx, (a) Cxx, (b) Dxx, (c) with x xb. When trait is also taken into account, the point x xb forms a line in the bivariate trait space. hus, the aforementioned conditions for univariate evolutionar branching can be translated into the following statement. Bivariate translation of conditions for univariate evolutionar branching: When directional selection in is ver weak, such that populations around a line branching, if and onl if Eqs. () are all satisfied. b G O( ), monomorphic x x converge to that line and bring about evolutionar Using these simple results as a baseline for comparison, we now consider the case that G is of order, which, according to Eq. (), implies that the evolution in can affect the evolution in x. When the population is not close to the singular line x x, directional evolution in x dominates the effects of. hus, the singular line still attracts monomorphic b Page of

14 populations if and onl if Cxx. We thus call such a singular line a convergence-stable line. When a population is in the neighborhood of a convergence-stable line and evolves toward it, directional selection in x inevitabl becomes small, such that evolutionar dnamics. G ma affect the When x is close to x b, a dimorphism in x ma emerge, but invasion b a mutant in with higher fitness ma exclude both of the coexisting resident phenotpes and thereb abort the incipient evolutionar branching. Such an abortion is especiall likel when hus, the larger G is large. G becomes, the more difficult evolutionar branching is expected to be. Below, we examine how the resultant likelihood of evolutionar branching can be estimated.. Motivation for further analses In principle, bivariate evolutionar branching is possible even for ver large G, as long as trait-substitution sequences comprise invasions onl in x for an adequatel large number of substitutions after the inception of dimorphism. However, for large G, sequential invasions of this kind are unlikel, because the fitness advantage of mutants in then is large, which favors their invasion, which in turn easil destros an initial dimorphisms. hus, the average number of invasions required for evolutionar branching is expected to be quite large in this case. We can thus measure the likelihood of evolutionar branching as the probabilit of its successful completion within a given number of invasions. It is difficult to calculate this probabilit directl, and thus to determine its dependence on the parameters G, C xx, and D xx of the normal form. o avoid this difficult, we focus on invasions that individuall have maximum likelihood for a given composition of residents. We can loosel interpret the successions of residents formed b such invasions as describing tpical evolutionar paths. Because of their special construction, it is possible analticall to derive sufficient conditions for evolutionar branching along these paths. It is expected that the conditions thus obtained can serve as useful indicators for the probabilit of evolutionar branching along the more general evolutionar paths formed b arbitrar stochastic invasions. Notice that when we refer to stochastic invasions, we refer to the stochasticit of mutations and to the stochasticit of the initial survival of rare mutants, but not to the effects resulting Page of

15 from small resident population size, which occasionall allow mutants to invade even when the have negative fitness. In formal terms, these clarifications are implied b our assumption of sufficientl large resident population size. In our analses below, we assume that the conditions for evolutionar branching in univariate trait spaces, Eqs. (), are satisfied. Our goal is to determine how conditions for bivariate evolutionar branching in x under weak directional selection in differ from Eqs. (). Maximum-likelihood-invasion paths In this section, we define evolutionar paths formed b sequential invasions each of which has maximum likelihood. Among all possible evolutionar paths formed b arbitrar stochastic invasions, these paths have high likelihood and ma therefore be regarded as tpical. Our reason for introducing these maximum-likelihood-invasion paths (Ps) is that we can derive, in Section, conditions for evolutionar branching along those tpical paths in Section.. Definition of oligomorphic stochastic invasion paths We start b explaining how probabilities of invasion events are formall defined. We consider a monomorphic population with phenotpe s, as a trait vector with an arbitrar dimension, at equilibrium population size ˆn that is uniquel determined b s. he birth and death rates (i.e., the number of birth and death events per individual per unit time, respectivel) of a rare mutant phenotpe s are denoted b b( s; s ) and d( s; s ), where b(;) ss and d(;) ss denote the birth and death rates of the resident s, which must satisf f(;) ss b(;) ss d(;) ss, because the resident is at population dnamical equilibrium. he invasion fitness of the mutant b s in the environment determined b the resident s is given f( s; s) b( s; s) d( s; s ). () Page of

16 Once a mutant s has arisen, the probabilit of its successful invasion in a population of resident s is approximatel given b f( s; s) / b( s; s ) (Dieckmann and Law, ). Here, the subscript + denotes conversion of negative values to zero. he probabilit (densit per unit time) for the emergence of a successfull invading mutant s in a population of residents s is given b multipling the numberdensit nb ˆ (;) ssm( s s ) of mutants emerging per unit time with their probabilit of successful invasion, f ( s; s) E( s; s) nb ˆ ( s; s) M( ss) b( s; s) nm ˆ ( ss) f ( s; s), where is the mutation probabilit per birth event, and M ( s s ) is the mutation (a) probabilit distribution. he above approximation applies in the leading order of s s, when is sufficientl small such that b( s; s) b( s; s ) is much smaller than b( s; s ) (i.e., [(;) b s s b(;)]/ s s b(;) s s O( )) (see Appendix C for the derivation). he expected waiting time for the next invasion event is given b / E( s; s)ds. When an invasion event occurs, the successfull invading mutants s follow the invasion-event probabilit densit P( s; s) E( s; s ). (b) For a polmorphism with resident phenotpes sr ( s,..., s N ), the probabilit densit per unit time of successful invasion b a mutant phenotpe s originating from the resident given b an approximation analogous to the monomorphic case, Eq. (a), i N i i N si is E ( s; s,..., s ) nm ˆ ( ss ) f ( s; s,..., s ), (a) where n ˆi is the equilibrium population size of the resident define an invasion-event probabilit densit i N i N s i. As in Eq. (b), we can also P( s; s,..., s ) E ( s; s,..., s ), (b) N where / E ( s; s,..., s )ds i i N is the expected waiting time for the next invasion event. Notice that the invasion event is identified b the combination ( s, i ) of the mutant phenotpe s and its parental resident s i. Consequentl, the invasion event probabilit den- Page of

17 N sities are normalized according to P ( s; s,..., s )ds i i N. When onl a single resident exists, Eqs. (a) and (b) are identical to Eqs. (a) and (b), respectivel. he invasion b a mutant leads the communit to a new population dnamical equilibrium. In most cases, the mutant replaces onl its parental resident, while under certain conditions the coexistence of both, extinction of both, or extinction of other residents ma occur. A se- quence of such invasions specifies a succession dnamics of resident phenotpes, which is called a trait-substitution sequence (Metz et al., ). If the invasion event is calculated stochasticall according to Eqs. (b) and (b), the resultant trait substitution is called an oligomorphic stochastic process (Ito and Dieckmann, ). When considering an initial monomorphic resident s a and a mutant-invasion sequence I ( s(),..., s( k),..., s( K )), (a) where s ( k) is the mutant that invades in the k th invasion event, the trait-substitution sequence is denoted b RIs ( ; ) ( s (),..., s ( k),..., s ( K )), (b) a R R R where sr( k) ( s ( k),..., s N( k) ( k)) is an Nk-dimensional ( ) vector composed of the Nk ( ) resident phenotpes that coexist after the invasion of s ( k). his kind of trait-substitution sequence constitutes an evolutionar path, which we call an oligomorphic stochastic invasion path (OSIP). If the k th invasion event leads to the extinction of the entire communit, no further inva- sions occur. In this case, the lengths of I and RIs (; a ) are limited b k. In this stud, we condition all analses on the absence of complete communit extinction.. Definition of maximum-likelihood-invasion paths We now introduce the concept of maximum-likelihood invasion. Specificall, we define a maximum-likelihood-invasion event as the combination of the mutant s and its parental resident s i with i i that maximizes the invasion-event probabilit densit, Eq. (b), across all s and i for a given set of residents, Page of

18 s, i argmax P ( s; s,..., s ), () ( s, i) i N where we refer to s and s i as the mutant and resident, respectivel, and denote s i b s for convenience. A maximum-likelihood-invasion path (P) is a trait-substitution sequence formed b events, denoted b I ( s (),..., s ( k)..., s ( K )). he P, which is expressed as RI ( ; sa ) with the for an initial monomorphic resident s a, is included in the set of all corresponding possible OSIPs RIs (; a ). Note that the mutational steps s s are bounded b, if invasion-fitness functions are approximated b quadratic forms of s (e.g., Eqs. ) and if mutation probabilit distributions are approximated b multivariate Gaussian functions (Appendix F). Also note that the P does not give the maximum-likelihood OSIP, which would require maximization of the likelihood at the level of the mutant-invasion sequence rather than at the level of individual mutant-invasion events. Although such sequence-level maximization would be more appropriate for our purpose, it seems analticall intractable. On the other hand, the event-level maximization defined b Ps is analticall tractable, and the P is still expected to have a relativel large likelihood among corresponding OSIPs. Likewise, as illustrated b our numerical results in Section, when an P RI ( ; s a ) exhibits evolu- tionar branching, then a large fraction of the corresponding OSIPs evolutionar branching. RIs (; a ) also exhibit Conditions for evolutionar branching along Ps In this section, we derive sufficient conditions for evolutionar branching along Ps, in terms of the properties of the normal form for invasion-fitness functions with significant sensitivit difference, Eq. ().. Further rescaling Here we assume that the base point of expansion s ( x, ) is on a convergence-stable line x x that satisfies univariate conditions for evolutionar branching, Eqs. (). o b Page of

19 simplif the analsis, we adjust the trait space as follows, without loss of generalit. First, we shift the origin of the trait space to the base point s so that s and b (,) x. Second, we rescale the trait space so that. (In this case, magnitude differences among s O( ), s O( ) with are transformed into those among the corresponding derivative coefficients G, C, and D, while the magnitudes of s, s themselves become similar to each other.) hird, we rescale time and potentiall flip the direction of the -axis so that G. For simplicit, we consider the first- and second-order terms onl. Consequentl, f ( s; s ) is given b s s, (a) f( ; ) Dx Cxx where C C G xx (b) and D D xx, (c) G with before the rescaling (i.e., ). In the simplified normal form in Eq. (a), onl two dimensionless parameters D and C determine the geometr of the fitness landscape. his geometr not onl determines the fitness landscape s shapes ( D ), but also how the landscape changes when the resident phenotpe is varied ( C ). Eq. (a) then shows that an possible fitness landscape f ( s; s ) can be obtained from parallel shift, i.e., f s s s s with ( ; ) f( w;(,) ) his means that the contour curve f ( s; s ), given b f( ;(,) ) Dx s b a s ( x, ) ( C x / D, Cx/ D). w w w, D( x x xw) w alwas has a constant parabolic shape specified b D, so that the position of this curve determines the fitness landscape (Fig. a). In the next two subsections, we derive conditions on D and C for evolutionar branching along Ps. We first obtain conditions on mutants, s, for evolutionar Page of

20 branching. hen we analze these conditions considering the dependence of s on D and C, which provides conditions on these two parameters for evolutionar branching.. Conditions on mutants for evolutionar branching Here we obtain conditions on s for evolutionar branching. he process of evolutionar branching can be decomposed into two steps: emergence of protected dimorphism (dimorphic emergence) and directional evolution of these two morphs in opposite directions (dimorphic divergence). First, sufficient conditions for dimorphic emergence and specific evolutionar dnamics ensured b these conditions are expressed as follows (see Appendix G for the derivation). Lemma : Suppose the conditions for dimorphic emergence below hold. hen, for an arbitrar initial resident s a, repeated invasions b s first induce directional evolution of the population toward the convergence-stable line x xb, and then bring about protected dimorphism after sufficient convergence. Conditions for dimorphic emergence: An monomorphic resident s satisfies s ( x, ) under an arbitrar x x for x x x for x, (a) and D x x ( ). (b) he set of mutants satisfing inequalities () are illustrated as ais indicated b the white region in Fig. c. Clearl, inequalities (a) ensure that the mutant is alwas closer than the resident to the convergence-stable line, resulting in directional evolution toward this line as long as the mutant replaces the resident. Inequalit (b) restricts the deviation of the mutant from the resident along the -axis, and thus ensures that a protected dimorphism (with f ( s; s ) and f (; ss ) ) emerges after sufficient convergence to the line. Page of

21 After emergence of an initial protected dimorphism, we denote the coexisting phenotpes b s and s, with x x, without loss of generalit (Fig. b). A sufficient condition for dimorphic divergence and specific evolutionar dnamics ensured b these conditions are expressed as follows (see Appendix H for the derivation). Lemma : Suppose the conditions for dimorphic divergence below hold. hen, for an initial protected dimorphism of s and s emerged under the conditions for dimorphic emergence, subsequent invasions b s continue directional evolution of the two morphs in opposite directions in x without collapse. Conditions for dimorphic divergence: An s satisfies x x and D( x x ) (a) or x x and D( x x ), (b) where x x is assumed without loss of generalit. he set mutants satisfing inequalities () are illustrated asis indicated b the white regions in Fig. d. In each invasion step, s replaces onl its parental resident, so the divergence of the new dimorphism in x is larger than that of s and s. Clearl, if conditions for dimorphic emergence and that for dimorphic divergence both hold, evolutionar branching along Ps inevitabl occurs for an arbitrar initial resident s a.. P condition As s is a function of D and C, substituting this function into the conditions for dimorphic emergence and divergence above and solving those for D and C gives conditions on these parameters for evolutionar branching. Page of

22 o derive s as a function of D and C, we explicitl define the mutation distribution. For analtical tractabilit, we assume that the mutation distribution is approximated b a two- dimensional Gaussian distribution, which is expressed in the normalized and rescaled trait space s ( x, ) as M ( s) exp s, () where the standard deviation of mutational step sizes is scaled to. Under monomorphism with phenotpe s, the mutant s, which maximizes the invasion-event probabilit densit, is given b the s that maximizes Eq. (b), P( s; s) nm ˆ ( s) f ( s; s) nˆ exp [ x ] Dx Cxx. () We first focus on the special case that s is located exactl on the convergence-stable line, i.e., s (, ) with arbitrar. In this case, s is given b, s for D s D, s for D. D D () Substitution of Eq. () and emergence ields s (, ) into the inequalit condition (b) for dimorphic D. () We do not need to examine inequalities (a), which are of interest onl for x. Even if the resident is not located on the convergence-stable line, i.e., s ( x, ) with s ( x, ) with x, inequalities () still hold under D /, as shown in Appendix D. herefore, conditions for dimorphic emergence are satisfied if D / holds. In this case, s is alwas located inside of the dark gra rectangle in Fig. e. Moreover, under D /, the derivation in Appendix D derivesshows that s alwas satisfies the condition for dimorphic divergence. herefore, inequalit (), D /, is a Page of

23 sufficient condition for evolutionar branching along Ps starting from an arbitrar initial monomorphic resident. More specificall, we have heorem : Suppose a normalized and rescaled invasion-fitness function having significant sensitivit difference, which is expressed in a form of Eq. () with, satisfies both C and inequalit (), D /. hen, an P starting from an arbitrar initial monomorphic resident monotonicall converges toward the convergence-stable line, x, and brings about protected dimorphism, which leads to dimorphic divergence without collapse, i.e., evolutionar branching. We thus call inequalit (), D /, the P condition for evolutionar branching, and refer to a convergence-stable line satisfing this condition as an evolutionar-branching line.. Directional evolution sufficient for evolutionar branching Under the P condition, dimorphism with before the population directionall has evolved b x x x * for arbitrar * * a a a * x emerges L x, x x x /, () where the second equalit defines the function L * x, x a, and s ( x, ) is the initial a a a monomorphic resident (see Appendix J for the derivation). he is the mean value of, given b for monomorphism or b ( nˆ ˆ ˆ ˆ n)/( n n) for dimorphism, where ˆn and ˆn are the equilibrium population sizes of s and s, respectivel. Numerical examination of P condition In this section we investigate how the P condition is related to the likelihood of evolutionar branching in numericall calculated Ps, OSIPs, and polmorphic stochastic invasion paths (PSIPs) in which mutation rates are not small. See Appendices K, M, and L for details on the calculation algorithms and initial settingsconditions. When deriving the P condition, we assumed the bivariate Gaussian mutation distribution defined in Eq. (), called bivariate Gaussian here.). he resultant P Page of

24 condition ma also be applicable to other tpes of mutation distributions. o examine this kind of robustness, below we investigate onan additional three different mutation distributions for the calculation of OSIPs and PSIPs. Ahe bivariate fixed-step distribution has possible mutations that are bounded on a circle (Fig. b). Ahe univariate Gaussian distribution applies when mutations in x and occur separatel, each following a one-dimensional Gaussian distribution (Fig. c). Ahe univariate fixed-step distribution also limits possible mutations to affect either x or, but with fixed step sizes (Fig. d). See Appendix L for details on these mutation distributions. he cumulative likelihood of evolutionar branching is measured as a probabilit pl ( Lˆ ), where L is the length of directional evolution in along Ps, OSIPs, or PSIPs until evolutionar branching has occurred, while ˆL, calculated withthrough Eq. (), is the length of directional evolution in along Ps sufficient for the occurrence of evolutionar branching (see Appendix K for details on ˆL ). hus, pl ( ˆ L) measuresgives the cumulative probabilit of evolutionar branching beforewhen the population has directionall evolved in b L, beond what is implied b the P condition ( ˆL ) L L ˆ is the additionall needed directional evolution in, relative to what is implied b the P condition. In the case of Ps, pl ( ˆ L) clearl holds for L ˆ L. In the case of OSIPs and PSIPs, when values of pl ( ˆ L) for L ˆ L are close to, this indicates that the P condition is working well also under such relaxed conditions. However, pl ( Lˆ ) never reaches in OSIPs, differentl from Ps. One reason is that even under ver large D there are non-zero probabilities for repeated mutant invasions onl in the - direction, providingcausing directional evolution in the -direction. Another reason is that even after the emergence of a protected dimorphism, thethis dimorphism ma collapse b subsequent mutant invasions in the case of OSIPs. When a dimorphism has collapsed, leaving behind a monomorphic resident, b the definition of OSIPs, the information about the col- lapse itself is lost, and it is onl the remaining resident that determines the likelihood of evo- lutionar branching in the next trial. A sufficientl large D is expected to induce evolu- tionar branching within a few trials, keeping the total directional evolution in the - direction short, which results in a high value of pl ( ˆ L) for L ˆ L and vice versa.. Page of

25 . Sufficient vs. necessar conditions: Ps Fig. a shows the branching likelihood in Ps under the bivariate Gaussian mutation distribution for varing C and D : the contour curves indicate where a % cumulative probabilit of Fig. a shows the occurrence of evolutionar branching in Ps is reached for L ˆ L, L ˆ L, and L ˆ L (i.e., p()., p()., and p(). ), at various values for C and D under bivariate Gaussian mutation. For D /, Ps quickl undergo evolutionar branching in the gra area in Fig. a, while the do not undergo evolutionar branching in the white area in Fig. a. Examples of branching and non-branching Ps are shown as gra curves in Fig. b and Fig. c,d, respectivel. Importantl, the threshold D / provided b the P condition and indicated b the black dashed line in Fig. a characterizes ver well the area that ensures the occurrence of evolutionar branching. In particular, the P condition D / seems to give a necessar and sufficient condition as C converges to.. Robustness of P condition: OSIPs When the P condition D / holds, OSIPs tend to undergo immediate evolutionar branching (black curves in Fig. b). On the other hand, even for D /, OSIPs ma still undergo evolutionar branching (black curves in Fig. c). In this case, however, the required L ˆ L becomes large as D is decreased. As D is decreased further, evolutionar branching ma not be observed even for ver large L ˆ L (black curves in Fig. d). Fig. a shows the branching likelihood in OSIPs under the bivariate Gaussian mutation dis- tribution for varing C and D : t. he contour curves indicate where a % likelihood cumulative probabilit of evolutionar branching is reached for L ˆ L,, and (i.e., p()., p()., and p(). ). We see that more than % branching likelihood is attained for L ˆ L, as expected b the P condition. Similarl, more than % branching likelihood is attained for L ˆ L for each of the three oth- er mutation distributions (Fig. b-d), as long as the mutation rate in is not ver small com- pared to that in x (i.e., / x. ) for the univariate Gaussian and univariate fixed-step Page of

26 mutation distributions. hus, for the examined OSIPs, the P condition turns out to be ro- bust (at a likelihood level of %) as an almost sufficient condition for evolutionar branch- ing; it is also robust against variations in mutation distributions.. Robustness of P condition: PSIPs For PSIPs assume that, mutation rates areneed not be low. In this case, evolutionar dnamics are no longer, in contrast with OSIPs, given b trait-substitution sequences (as for OSIPs), but b gradual changes of polmorphic phenotpe distributions. Population dnamics of PSIPs are calculated based on the stochastic sequence of individual births and deaths (Dieckmann and Law, ). he stochastic effects become large when fitness gradients and curvatures are both weak and/or population sizes are small. In this case, the likelihood of evolutionar branching in PSIPs, in contrast with OSIPs, ma be affected not onl b C and D, but also b other parameters, such as the mutational step size, the mutation rate, and the carring capacit along the evolutionar-branching line, K. We have numericall confirmed that the P condition is still useful for characterizing evolutionar branching in PSIPs across a certain range of parameter values. For example, D / provides p(). under all four mutation distributions for.., K, and., with K (results not shown). Fig. e-h show the branching likelihood in PSIPs for varing C and D, with., K, and. :. the contour curves indicate where a % likelihood cumulative probabilit of evolutionar branching is reached for L ˆ L,, and (i.e., p()., p()., and p(). ). We see that more than % branching likelihood is attained for L ˆ L under all four mutation distributions, as long as the mutation rate in is not ver small compared to that in x (i.e., / x. ) for the uni- variate Gaussian and univariate fixed-step mutation distributions. hus, for the examined PSIPs, the P condition turns out to be robust as a good indicator for evolutionar branch- ing, even when mutation rates are not small and/or mutation distributions other than bivariate Gaussian are considered. Page of

27 Conditions for evolutionar-branching lines In this section, we first summarize the conditions for evolutionar-branching lines in bivariate trait spaces. Second, we extend these conditions to multivariate trait spaces. hird, we explain how to find evolutionar-branching lines or manifolds in arbitrar trait spaces with arbitrar dimensionalit.. Conditions for evolutionar-branching lines in bivariate trait spaces B an appropriate affine transformation, arbitrar trait spaces S ( XY, ) can be normalized into a representation s ( x, ) with isotropic mutation with standard deviation. In this normalized trait space, the invasion-fitness function can be expanded around s as shown in Eq. (a), f ( s; s) Gs( ss) Cs s Ds O( ), where s(, ) s s. If the x -and -axes can be adjusted such that Eq. () holds, x G C C C D D x x x G C D x xx xx O( ), then f ( s; s ) is significantl less sensitive to trait than to trait x. In this case, Eq. (a) is transformed into Eq. (), f( s; s) GxxCxx ( xx ) x Dxxx G O( ). If Eq. (a) holds, Gx, then there exists a singular line x x denoted b x b. his line is convergence stable if Eq. (b) holds, Cxx (i.e., if C ). he convergence-stable line causes evolutionar branching along Ps (maximum-likelihood-invasion paths), if inequalit (), i.e., the P condition, holds, Page of

28 D D xx. G In this case, an P starting from a monomorphic resident s a ( xa, a) with sa s O( ) inevitabl converges to the line and brings about evolutionar branching. Here we refer to Eqs. (), (a), (b), and () as the conditions for evolutionar-branching lines, which is summarized as follows. heorem : Suppose that s a point s in a normalized trait space s satisfies the conditions for evolutionar-branching lines below. hen, there exists an evolutionar- branching line passing through s and parallel with the -axis of the trait space. In this case, an P starting from a monomorphic resident s a ( xa, a) with sa s O( ) monotonicall converges to the line and brings about a protected dimorphism, which leads to dimorphic divergence in x without collapse, as long as theirthe deviations from s are O( ). Conditions for evolutionar-branching lines along Ps in normalized bivariate trait spaces: he x -and -axes can be adjusted b rotation such that the first and second derivatives of the invasion-fitness function at s satisf all of Eqs. (), (a), (b), and (). Rescaling trait spaces such that and appling theorem proves this theorem. If these conditions for evolutionar-branching lines hold, then evolutionar branching oc- curs with high likelihood in evolutionar paths even under relaxed assumptions (i.e., in OSIPs and PSIPs, as shown in Section ). As the sensitivit difference goes to infinit, which means that the right-hand side of Eq. () converges to zero, the conditions for evolutionar- branching lines converge to the univariate conditions for evolutionar branching, given b Eqs. (). Notice that the P condition requires that is not infinitesimall small, but finite; oth- erwise, satisfing this inequalit is impossible. hus, as long as the population is directionall evolving, its evolutionar branching requires finite mutational step sizes. Conversel, can have large magnitudes, as long as approximation ofapproximating the invasion-fitness func- Page of