Evolutionary Stability for Interactions Among Kin Under Quantitative Inheritance

Transcription

1 Copyright by the Genetics Society of America Evolutionary Stability for Interactions Among Kin Under Quantitative Inheritance Alan B. Harper Department of Zoology, University of Washington, Seattle, Washington Manuscript received August 1, 1987 Accepted for publication December 17, 1988 ABSTRACT The theory of evolutionarily stable strategies (ESS) predicts the long-term evolutionary outcome of frequency-dependent selection by making a number of simplifying assumptions about the genetic basis of inheritance. I use a symmetrized multilocus model of quantitative inheritance without mutation to analyze the results of interactions between pairs of related individuals compare the equilibria to thosefound by ESS analysis. It is assumed that thefitnesschangesduetointeractions canbe approximated by the exponential of a quadratic surface. The major results are the following. (1) The evolutionarily stable phenotypes found by ESS analysis are always equilibria of the model studied here. (2) When relatives interact, one of the two conditions for stability of equilibria differs between the two models; this can be accounted by for positing that the inclusive fitness function for quantitative characters is slightly different from the inclusive fitness function for characters determined by a single locus. (3) The inclusion of environmental variance will in general change the equilibrium phenotype, but the equilibria of ESS analysis are changed to the same extent by environmental variance. (4) A class of genetically polymorphic equilibria occur, which in the present model are always unstable. These results exp the range of conditions under which one can validly predict the evolution of pairwise interactions using ESS analysis. HE theory of evolutionarily stable strategies T (ESSs) has been proposed as a method of determining the evolutionary outcome of frequency-dependent selection (MAYNARD SMITH PRICE 1973; MAYNARD SMITH 1982). An ESS of a game is a state in which no rare strategy can do better than the common strategy in a population. One particular form of frequency-dependent selection is the evolution of interactions among relatives (MICHOD 1982; KARLIN MATESSI 1983). In this paper I will examine a genetic model of pairwise interaction between relatives involving a single phenotypic character. Models of such interactions have been analyzed using ESS theory (MAYNARD SMITH 1982), but such analyses leave in doubt the actual dynamic evolution of the phenotype. In this paper I will show that the stable equilibria of a particular model of polygenic inheritance are similar, often identical, to ESSs will offer a graphical interpretation of the nature of the two sorts of equilibria. Let w(x, y) be the viability of an individual of phenotype x after it interacts with an individual of phenotype y; x y are unidimensional real variables. If individuals interact at rom, an ESS (MAYNARD SMITH PRICE 1973) is defined as a phenotype x* for which w(x*, x*) 2 w(x, x*); (14 Present address: Box 654, 1916 Pike Place #12, Seattle, Washington in the case of equality in (la), then w(x*, x) > w(x, x) (1b) for all x # x*. When conditions (la, b) hold, then no rare mutant affecting just this phenotype can invade an infinite panmictic population with haploid inheritance that is fixed for phenotype x*, nor can a nonrecessive mutation invade a diploid population. Because of the close correspondence of the conditions for an ESS the uninvadability condition for panmictic populations, I will use the terms ESS uninvadable phenotype interchangeably. If a single genotype results in a probability distribution of phenotypes, then one needs to replace w in (la, b) with 6, where 6 is defined as the expected fitness of genotype x when it interacts with an individual of genotype y, averaged over the joint distribution of the phenotypes produced by x y. This is the method used to analyze the evolution of the probabilities of adopting one of two or more strategies in an interaction (see MAYNARD SMITH 1982, Appendix D; THOMAS 1984). When the phenotype is a continuous character, such as a morphological trait, one might expect the phenotype to be determined by both genetic environmental effects (FALCONER 198 l). By integrating over the distribution of environmental deviations one can convert the phenotypic fitness function w to the genotypic function 6 find ESSs for such characters. The criterion (1) has often been modified when Genetics 121: (April, 1989)

2 878 A. B. Harper phenotypes can vary continuously over a range of values. When the phenotypes can vary continuously, determining the validity of criterion (1) requires knowledge of the global properties of w. It is often easier to find ESSs that are determined by only the local properties of w in the neighborhood of x * (MAY- NARD SMITH 1982). If dw(x, y)/dx Ix=y=x* = 0 (24 d2w(x, y)/dx I < 0, (2b) then rare mutations of small effect cannot invade a population fixed for x*. Conditions (2a, b) are not exactly the same as conditions (l), but conditions (2a, b) guarantee that w(x*, x*) > W(X* + t, x*) for t small nonzero. When the phenotypes can vary continuously, it is not necessarily true that the invasion of rare mutants of small effect will take the population toward the ESS (ESHEL MOTRO 1981; ESHEL 1983). In a sense, this means that not all ESSs are attainable; for some ESSs if the population starts near the ESS the continued invasion of favored mutations will cause the population to evolve away from the ESS, even though if the population were exactly at the ESS, no mutation could invade. ESHEL MOTRO have shown that when the populations is fixed for a gene with phenotype x* + t then mutations with phenotypic effect of the order t2 that vary in the direction of the ESS can invade just when d w(x, y)/dx2 + d w(x, y)/dxdyi x=y=x* < 0. (2c) ESHEL (1983) has called phenotypes that meet conditions (2a-c) continuously stable strategies, but here 1 will incorporate condition (2c) into the definition of an ESS to avoid complexity of terminology. The foregoing methods are used when individuals interact in romly formed pairs. When individuals are more likely to interact with relatives, the invasion conditions must be modified to reflect this. If an individual with a rare genotype which produces phenotype x is in a population in which the common phenotype is y then the fitness of this individual will be W(r,(X, y) = (1 - r)w(x, y) + TW(X, x), (3) where r is the probability that the rare genotype interacts with another of the same genotype. In a diploid population without inbreeding, T is just the average relatedness of interactants. The conditions for an ESS for interactions among relatives are (1) with w(?) substituted for w. Following the same reasoning as above, it can be seen that the conditions for uninvadability by mutations of small effect for continuous stability can be obtained by substituting qr) for w in (1) (2) (see also MAYNARD SMITH 1982, Appendix F; GRAFEN 1979). The conditions for an ESS among relatives are then dw(,)(x, y)/dx = wx + my = 0, (44 d2w(,)(x, r)/dx2 = wxx + 2mXy + myy < 0, (4b) d2w(?)(x, y)/dx + d2w& y)/dxdy = wxx + (1 + T)WV + myy < 0. (44 The subscripts x y on w indicate partial derivatives of w with respect to the subscript variable, evaluated atx=y=x*. A number of investigations have considered the question: when can the equilibria of other genetic models be identified with the ESS phenotypes found by invasion analysis? The models used are generally of two types, it is assumed either that there are a number of discrete phenotypes possible (called here n-strategy games ) or that there is a continuous, onedimensional, range of phenotypes ( continuous-strategy games ). Most of these investigations have looked at the case where individuals interact at rom. In the n-strategy rom-encounter game it is clear that at any ESS the fitness of individuals adopting each of the various strategies must be equal, otherwise a phenotype that adopted only the most fit strategy could invade. SLATKIN (1978, see also LESSARD 1984) has shown in the n-strategy rom-encounter game that if there are a sufficient variety of mutations in a diploid population, evolution will tend to equilibrate the fitnesses of the phenotypes, leading to the same result as predicted by ESS analysis. MAYNARD SMITH (198 1) has shown that when there are just two phenotypes, diploid inheritance, two alleles affecting the probability of having each phenotype, then it is almost true to say that the population will evolve to an ESS if it can, if it cannot it will approach the ESS ascloselyas the genetic system will allow (p. 1016). ESHEL (1982, see also HINES 1987a) has shown that under diploid single-locus multiallele inheritance, under multilocus additive inheritance, every achievable ESS is a stable equilibrium of gene frequencies. ESHEL FELDMAN (1984) have found that the initial increase conditions in a two-locus diploid n- strategy model are a function of the difference of the population mean phenotype the ESS of the effects of epistasis recombination. These results together indicate that the predictions of ESS theory in n-strategy, rom encounter models are quite robust to variations in the inheritance of the strategies, except when there are epistatic interactions among the alleles at two or more loci. All of these models consider only the case when the phenotypes interact at rom, do not consider the effects of population subdivisions such as interactions within families.

3 Evolutionary Quantitative Stability 879 SLATKIN (1979) considered three models for the evolution of a normally-distributed character subject to frequency- density-dependent selection. Two of the models concern us here. In a continuous-genotype model, similar to the model used below, two sorts of equilibria can occur: either all heritable variation is lost from the population, or heritable variation remains certain average fitness effects exactly cancel. The second equilibrium is characterized by the two conditions where p(z It,.I) is a normal density function with specified mean variance, w(z, i, u:) is the fitness of phenotype z when the population is distributed according to p. SLATKIN shows that conditions (5a, b) also characterize phenotypes that are uninvadable by mutations that change t a:. Conditions (5a, b) are equivalent to the ESS condition (2a) for the two characters the mean value of z the variance in expression of z. The extensive literature on kin selection (reviewed in KARLIN MATESSI 1983 MICHOD 1982) can be seen, in part, as an examination of whether w (~) can be used as a summary measure of selection involving interactions among relatives. Many models of kin selection assume that there are two strategies that an individual can assume in an interaction, altruistic or non-altruistic, that an individual s genotype affects its probability of being altruistic. This is equivalent to assuming that w (~) is linear. Let x y be the probabilities of being altruistic for one individual its relative. Then the fitness of an individual after an interaction is w(x, y) = 1 + yp - XY, (6) where y represents the amount that the individual s fitness is decreased by being altruistic, P represents the amount an altruistic relative increases the individual s fitness. If w(~) were an adequate fitness measure, then under (6) one would expect larger values of the phenotype to have higher fitness just when rp > y. When w is linear, it is found that the effects of dominance (in the probability of being altruistic) among alleles at a single locus (CAVALLI-SFORZA FELDMAN 1978; UYENOYAMA FELDMAN 1981; UYENOYAMA, FELDMAN MUELLER 198 l), the effect of recombination between two loci (MUELLER FELDMAN 1985) can lead to a class of interior equilibria ( structural equilibria ) that cannot be predicted by the naive use of w(r) as a measure of inclusive fitness. In many single-locus models, the interior equilibria can be simply described by considering the average effect of a substitution of alleles (UYENOYAMA FELDMAN 1981; UYENOYAMA 1984). The structural equilibria only occur when heterozygotes are not approximately intermediate in phenotype between the homozygotes with which they share alleles; for nearly additive genes, w(~) is an adequate measure of the direction of selection when w is linear (TORO et al. 1982). Quantitative genetic models with additive alleles in which it is assumed that either w is linear or log w is linear also have behavior consistent with the predictions of wb) (YOKOYAMA FELSENSTEIN 1978; AOKI 1982; CROW AOKI 1982; ENCELS 1983). However, when w is linear, there can be no stable phenotypic equilibria, either altruism or selfishness is increased to the limits of heritable variation. These results indicate that w (~) is an adequate measure of inclusive fitness when genes are nearly additive the fitness function is linear, i.e., interactions among relatives with fitness function w evolve in the same direction as interactions among nonrelatives with fitness function w(+ [BOYD RICHERSON S (1980) analysis of a nonlinear fitness function using a quantitative genetic model is discussed later.] I will investigate the equilibria found in a continuous-genotype quantitative-genetic model in which the fitness of an individual is determined by a nonlinear function of its own phenotype the phenotype of other individuals with which it interacts. Interactions between relatives of an arbitrary degree of relatedness are considered. It will be seen that ESSs are always equilibria of the model, but that the stability conditions for the equilibria are not exactly the same as characterize ESSs. I examine nonlinear fitness functions since there are a number of interesting situations in which the fitness function may not be linear. Kin selection theory concentrates on the linear fitness function (6) only because this is the simplest mathe- matical form that an altruistic act can take. ESS theory (recently reviewed in HINES 1987b) often analyzes a linear form for w, both for historical reasons (BROWN VINCENT 1987) because of the powerful mathematical techniques such a form allows (for ex- ample, SCHUSTER et al. 1981). A linear form for w is appropriate if there are discrete strategies the fitness effect of interactions depends only on the strategies the interactants adopt (ie., if the payoff matrix is constant). If a continuum of strategies is available, or if the fitness effects of interactions are frequency dependent, w will usually be curved. For instance, models of siblicide generally allow just two strategies-siblicidal or not- consider a linear form for w (DICKENS CLARK 1987; GODFRAY 1987), while models of begging behavior, the intensity of which can vary continuously, assume a curved form for w (PARKER MACNAIR 1979; HARPER 1986).

4 880 A. B. Harper THE MODEL that I find below depend only on the sum of the variance within covariance among loci, suggesting In this paper I usefelsenstein S (1979) symme- that a nonsymmetric apportionment of variance will trized version of LANDE S (1975) model of the inher- not affect the position of the equilibria. However, the itance of a one-dimensional quantitative character. only forms of instability of equilibria that can be found The assumptions limitations of this model have with this model are when an equilibrium is unstable been discussed in detail elsewhere (FELSENSTEIN 1977; to the same perturbation in all variances or all covar- TURELLI 1984) are only briefly reviewed here. iances. This model cannot identify instabilities in LANDE S original unsymmetrized model assumes that which, say, at some loci the variance grows at the inheritance of the character is determined by a others it shrinks. As far as I know, this type of instafinite number, n, of loci in an infinite population. The bility has not been reported in a quantitative genetic contribution of the effect of one allele is a real num- model. ber, which contributes additively to the phenotype, so that the phenotype of an individual is the sum of the RECURSION EQUATIONS effects of all 2n alleles, plus any environmental deviation. It is assumed that the distribution of allelic effects FELSENSTEIN (1979) has derived the recursion equais multivariate normal, so that the population genetic tions for the symmetrized LANDE model with norstructure can be specified by the mean allelic effects optimizing selection. Before selection, the variance of at each locus, the variances of allelic effects within allelic effects at each locus is V,/(2n) at generation t, the covariances between loci. This last assumption the covariance between any two loci is Cl/(2n(n - l)), requires that a number of approximations be made. the average effect at any locus is Mt/(2n). The If the fitness function is not normal, the distribution breeding value of an individual is then the sum of 2n of allelic effects cannot be normal both before rom values normally distributed with these moafter selection. Recombination adds the problem that ments, implying that the breeding value is normally the population after recombination consists of a mixdistributed with mean MI variance C, + V,. The nor-optimal fitness function for breeding value ( is ture of recombinant nonrecombinant classes. usually written as Each class may be multivariate normal, but in general a mixture of normal distributions is not itself normal. Mutation leads to the same sort of approximation as recombination, since the population consists of a mixture of mutant nonmutant classes. All these ap- (I will use i for a genotypic fitness function, w proximations are achieved by just keeping track of the for a phenotypic fitness function, ( to represent a first second moments of allelic effects as the breeding value, z a phenotype.) The curvature of population proceeds through the steps of mating, the fitness function is S the optimal phenotype selection, recombination, mutation, treating (or pessimal phenotype if S < 0) is P. When Equation the population at each step as multivariate normal 7a changes from optimizing to disruptive selection S with the proper moments. The import of TURELLI S passes through m, so I will reparameterize (7a) as (1984) suggestion that mutational rates may be low enough, mutational effects large enough, that the i(0 cz exp(ytas-2 + Bo, (7b) distribution of allelic effects will be non-normal is left where S = -1/A P = -B/A. It turns out that Vl to the discussion below. Simulations reported in FEL- Ct enter into the recursion equations only as a SENSTEIN (1 979) suggest that the approximations for sum, so it will be convenient to define G, = V, + C, as the effects of recombination selection are generally appropriate, even in the case of wildly non-normal fitness functions. the total genetic variance in generation t. Reparameterizing FELSENSTEIN S (1979) recursion equations for V,, C,, M,: FELSENSTEIN S symmetrized version of LANDE S model assumes that the recombination rate T is the same between all pairs of loci, that the mean variance at every locus is the same, that the covariance between every pair of loci is the same. The symmetric recombination rates cannot be realized by any biological linkage map unless r = 0 or T =?h. The equilibrium results that I find depend only on the presence of recombination between loci, not on the magnitude of it, suggesting that differing recombination rates among pairs of loci would not change the results. Perhaps more troubling is the assumption of symmetric variance covariances. The equilibria 1 AG: Vt+l = vt + - 2n 1 - AG; n- 1 AG: 2n 1 - AG, c1+, = C,(1 - T) + - (8b) PAIRWISE INTERACTIONS Nowwe will obtain the values of A B when there are pairwise interactions among individuals.

5 Stability Evolutionary Quantitative 881 This will allow us to find the equilibria of Equations 8 compare these equilibria to conditions (4). Conditions (4) result from the assumption that individuals form pairs, interact, have viability fitness w as a result of the interaction. I will allow the possibility of a correlation of phenotypes in interacting pairs either through a correlation of breeding values or through a correlation of environmental effects. The possibility of phenotypic assortative formation of pairs is not considered here, as such assortment could affect both the genetic phenotypic correlations together in a complicated way. If the population is panmictic, the covariance among loci, Cf, the variance within loci, V,, contribute proportionally to the correlation among the breeding values of two individuals. This can be seen by the following argument. The only way for the genotypes of two individuals to covary in a panmictic population is for the individuals to share one or more ancestors. If, say, individuals a b share an ancestor, the covariance between a b due to any one locus is the probability of sharing an allele from this ancestor times the variance at that locus in the ancestor s generation. The covariance between locus 1 in a locus 2 in b due to this common ancestor is the probability of a b both having alleles from this ancestor times the covariance between the loci in the ancestor. The recombination between the two loci does not enter into this calculation. This means that if the average proportion of alleles a b which are shared due to common descent from generation t - i is PC, then the covariance between a b due to these ancestors is pc(vf-; + Cf-i) = p&-i. The value of PC for any type of relation can be determined by stard methods (FALCONER 198 1). Unless the population has reached an equilibrium, the values of V,-i Cf-i will differ for each generation. However, I will make the approximation that the covariance between the breeding values of two individuals can be represented by pc(vi + C,) = pggf ignoring the variation among generations. This will be exactly true at any equilibrium. If the correlation among interacting individuals is due to the encounters of kin in a panmictic population, this approximation will not affect the equilibria1 results reported below. This is because kin are related through sharing common ancestors a fixed number of generations before the present generation (1 generation back in the case of siblings, 2 generations back in the case of cousins, etc.), violations of the approximation will only add second-order effects to the perturbation analyses. However, if the population is inbreeding, then individuals can be related through genes shared by a common ancestor any number of generations before, this approximation would fail. This approximation will also fail far away from an equilibrium. The correlation among the phenotypes of breeding individuals may also have an environmental component. I assume that the phenotype of an individual is the sum of its breeding value an independent normally distributed environmental deviation: z, = 5; + ex, where = 0, Var{e,) = E. The environmental deviations of a pair of individuals may covary: Cov( e, cy) = PEE. The covariance between the phenotypes of two interacting individuals is then the sum of the phenotypic genotypic covariances: Cov(zx, zy) = PGG, + pee. In order to derive the genotypic fitness function, we will need the covariance between the phenotypes of two interacting individuals conditioned upon the breeding value of one of the pair: Cov(zx, zy I S,). Given an individual s breeding value {, the breeding value of the individual it interacts with, ry, is normally distributed with moments - ry = (1 - m)mf + P CL Var(S;l = (1 - pzp)g, (9) for relatives of any fixed value of PC. The distribution of the phenotypes of an interacting pair, conditioned upon a certain breeding value for one member of the pair, is a bivariate normal distribution (MORRISON 1976). Let p(..) be a generic probability density. Stard theory tells us that p(zx, zy I 5;) has the following moments: - -.zx = rx, zy = (1 - PC)M + Pcs;, (10) Varlz,) = E, Var(zy) = (1 - pz)c, + E, Cov(z,, zy) = PcGt + PEE. I will treat the joint distribution of the breeding values of interacting relatives as if it were a bivariate normal distribution. This approximation is approached only when the number of loci affecting the trait, n, is large (BULMER 1980). When n = 1 the distribution of phenotypes of interacting pairs of kin would consist of a mixture of three classes, sharing 0, 1, or 2 alleles. The average fitness of a genotype is a function of both the distribution of interacting pairs the effects of the alleles, if the trait were determined by one or a few loci the average fitness could not be obtained analytically. When mixtures of relatives of varying relatedness interact (e.g., families consisting of full half siblings) the distribution of pairs of breeding values will no longer be normal since a mixture of binormally distributed pairs of relatives is generally not normal. Such a mixture would consist of distributions of breeding values, with the same variances, GI, but with different covariances depending on PC. In the spirit of the approximations we have already made, we will assume that this mixture really is normal proceed using the average value of pc to describe the pairs of relatives. This should add no further inaccuracies than our treatment of recombi-

6 882 A. B. Harper nation has, which itself has some support from simulations (FELSENSTEIN 1979). We now have, from (lo), the joint distribution of phenotypes of interacting relatives given the breeding value of one of the relatives. If we then assume that the phenotypic fitness function describing the fitness of an individual after an interaction can be approximated by an exponential of a quadratic function of the two phenotypes, ie., that the fitness function has the shape of a bivariate normal surface, then we can obtain the genotypic fitness function, in the form of (7b). The genotypic fitness function the recursion Equations 8 completely specify the evolution of the population. The derivation of the genotypic fitness function is shown in APPENDIX A, the results of which are sketched here. As in (l), let w(z,, zy) be the fitness of an individual of phenotype z, which interacts with an individual of phenotype zy. The fitness of breeding value 3; is then the average fitness resulting from the distribution of phenotypes produced by 5;: It is easier to underst (1 1) if we decompose the phenotypes z, zy into a breeding value an independent environmental deviation: The approximation in (13) allows us to complete the integral in (1 2), this gives us the genotypic fitness function needed to complete the recursion equations of the model. The value of (1 2) is shown in (A6) of APPENDIX A. EQUILIBRIA We can now find the equilibria stability conditions of the recursion equations. It turns out that the recursion equations for the genetic variance covariance, V, Cf, are independent of the mean phenotype, Mt. Following FELSENSTEIN (1979), take (n- 1) times (Sa) minus (Sb), substitute t = t + 1 = w, find that C, = 0. Since then G, = V,, we see from (Sa) that the only equilibria are A = 0 or G = 0. From (8c) we see that M can be at an equilibrium either when G = 0, or when A = 0 B = 0. This analysis implies that there are two classes of equilibria for Equations 8: either there is no heritable variation, G = 0, or the genotypic fitness function is flat there is no selection, A = B = 0. This is the same behavior as described by SLATKIN (1979). The stability of these two classes of equilibria is analyzed in APPENDIX B. It turns out that there are no stable equilibria with nonzero genetic variance in this model. Of course, when the approximations made in this model are violated, especially the assumptions that the phenotype distribution is normal that the fitness function can be approximated as in (13), kin selection could maintain genetic variation. COMPARISON OF THE G = 0 EQUILIBRIUM WITH ESS ANALYSIS = J w *p(ry I r xkqy x 9 fy y I 3;)dr.Y. Here $({,, ly) is the expected fitness of an individual with breeding value rx when interacting with an individual of breeding value S; averaged over the distribution of phenotypes that can be produced by these two breeding values. Let us approximate the logarithm of the phenotypic fitness function w with a second order Taylor series exped about a certain phenotype, LO. Let let v?, v,, vxy, vd be the other first- secondorder partials of log w evaluated at (zo, zo), let voo = log w(z0, zo), resulting in 4x7 Y) = exp(ww(x9 Y) = exp(vo0 + v,(x - LO) + vyo) - 20) + 1/2vxy(x - zo)2 + V,(X - z0)o) - Io) + %tldo) - zoy). (1 3) Remember that the ESS method finds an ESS by requiring that the partial derivatives of the fitness surface satisfy certain conditions when the partials are evaluated at the ESS. The equivalent solution for the present model is the conditions on the partial derivatives of (13) necessary for a phenotype to be a stable equilibrium of the model when the equilibrium is the point zo around which w was exped in (13). These conditions are derived in APPENDIX B, are 3, + PC;, = 0, ( 1 44 Vxx + 2pcsxy + p:v, 0, ( 14b) 3, + (1 + PC)kY + PCV, 0. (1 44 Here the parameters, G,, etc. are the partial derivatives of the logarithm of the genotypic fitness function seen in (12), ie., I - d log w x, Cy) vx = ai-, I =<= y 20 Remember that the genotypic fitness function is the expected fitness for a breeding value averaged over

7 Quantitative Evolutionary Stability 883 the distribution of environmental deviations, will differ from the phenotypic fitness function in (12) if there is any environmental variance in the expression of the phenotype. The values of Gx etc. are shown in Equations A2 of APPENDIX A. The conditions (14) can be compared to the conditions for an ESS by applying the ESS conditions (4a-c) to the present model. The ESS conditions are obtained by applying conditions (4) to the genotypic fitness function i(rx, ly), see AP- PENDIX B for the details. It turns out that that (14a) (14c) are the same as the conditions (4a) (4c), respectively, but that (4b) becomes sxx + 2PC3Xy + pcsyy + PC( 1 - PC)s: < 0, (15) which will generally differ from (14b) unless PC is 0 or 1. To recapitulate, the present model finds two classes of equilibria when relatives interact. At equilibria of one class, there is positive genetic variance maintained by selection. Such equilibria are always unstable. At the second class, genetic variance is eliminated by selection, these equilibria are identical to the equilibria found by ESS analysis (4a). But the stability conditions for the second class of equilibria are generally different than the ESS conditions (4b, c). The difference in the stability conditions can be understood by consideration of the assumptions that have been made about how pairs of interacting relatives are formed. In ESS (or invasion) analysis it is assumed that only two genotypes are segregating in the population, so that three types of interacting pairs can be formed using the wild type the rare mutant. Under this assumption, Equation 3 describes the inclusive fitness of a rare genotype. Under the assumptions of polygenic inheritance, it is assumed that the breeding value of an individual is determined by a large number of loci that rom sampling will cause relatives to share on average PC of the alleles at these loci. This suggests that the equation that should describe the inclusive fitness of individuals when relatives interact is W(P)(X, y) = w(x, (1 - Pc)y + PC%). (1 6) If we then apply the ESS conditions (2a-c) to w(+ we get the following conditions: h(,,~(x, y)/dx = wx + pcwr = 0, (17a) d2w(p)(x, y)/ax = wxx + 2 pcwxy + p h, < 0, (17b) a2w(p)(x? r)/d3cey + a2w (P)(X, r)/ax (174 = w xx + (1 + PC)WXy + pcw, < 0. Conditions (1 7) are the same as (1 4) as can be seen by substituting i for w in (16). It is interesting to note that the conditions for a stable equilibrium under polygenic inheritance have the same algebraic form whether the partials of the fitness function or of the logarithm of the fitness function are used. This is not true of the ESS conditions (4a-c). There is a simple graphical interpretation of the difference between the stard ESS conditions (4a-c) the conditions (17a-c). In Figure la the function i, the fitness of breeding value x when it interacts with breeding value y, is shown as a surface; in Figure 1 b i is drawn using level curves. Obviously, any monomorphic equilibrium must be a point on the line $(x, x), shown crossing the surface. The conditions (4a) (17a) require that a line, labeled A in Figure 1, which goes through the equilibrium, is parallel to the x-y plane, has slope pc in the x-y plane, be tangent to the surface. In this figure PC; is chosen to be Yz. The conditions (4c) (17c) require that the surface increase along both the arrows B which are parallel to A. The third condition for stability under polygenic inheritance is that the surface have negative (downward) curvature along the direction of the arrow A. The equivalent condition for uninvadability by single mutations is that the average of the curvatures of the surface along the arrows C C, weighted by PC 1 - PC, respectively, must be negative. This interpretation helps us underst when the ESS conditions the polygenic stability conditions might differ: just when the surface is saddle shaped so that the curvature of w changes sign between the directions of the arrows C, C. The stability conditions of the two models are always the same when pc = 0 or pc = 1. THE EFFECTS OF ENVIRONMENTAL VARIANCE Often in the study of frequency-dependent evolution, models ignore the fact that environmental variance affects the distribution of phenotypes produced by a particular breeding value. Equations A2 allow us to examine the effect of including environmental variance covariance among the interacting pairs. If there were no environmental variance then the phenotypic fitness function could be substituted for the genotypic fitness function. Similarly, if the logarithm of the fitness function had no curvature, the genotypic phenotypic fitness would,be the same. YOKOYAMA FELSENSTEIN (1 978) have examined the case when the log fitness function is flat, i.e., when w is an exponential surface,, consistent with these results, found that environmental covariance does not change the predictions of kin selection theory. BOYD RICHERSON (1980) have also considered the effect of phenotypic variation on kin selection. They conclude that the heritability of a character will influence the amount of altruism present in the population, the equilibria found under a model of quantitative genetics differ from those found by ESS analysis. It is instructive to examine how they reached these conclusions. BOYD RICHERSON assume that the fitness of an

8 884 A. B. Harper b a A ri 3 Y v) fn a C L ii Breeding value x Breeding value x FIGURE 1.-A graphical interpretation o f the differences between the stability conditions under invasion analysis under polygenic inharit;lnce for a trait that influences interactions among relatives. T h e surface G(x, y) is shown in a perspective view in a, using contour lincs i n b. T h e conditions for a stable equilibrium are discussed in the text. individual is a function of bothits own phenotype of the average phenotype of a group of kin. By just considering groups of2 kin, their results canbe compared directly with mine. They use a model of quantitativeinheritanceintroduced by CAVALLI-SFORZA FELDMAN (1 976), but it seems that the important difference between our assumptions is that they consider the effects of mutational variance, I do not. They assume that at a mutation-selection balance the distribution of breeding values is approximately normal, this allows them to examineequilibria at which there is non-zero heritability. They also assume that there is no correlation in the environmental deviations among interacting kin. For two individuals BOYD RICHERSON S phenotype fitness function is from which they find the following equilibrium mean phenotype: ~ ~ ( +4 (1 7 - PE)G + (1 - PC)E) Here E is the environmental variance, G is the genetic variance at equilibrium. T h e equilibrium can also be found using the methods above: obtain the derivatives v,, etc., of (18), calculate the derivatives of the genetic fitness function 3, using (A2), obtain the equilibrium from (1 4a). T h e equilibrium phenotype obtained in this way is + P E ) ( ~ - pc)e) + 2 P A 1 + P C P + (PC - P E F ) ~ ~ (+ 4 (1 7 x* = 47 + (1 + pc)(e - P E E + 2 s ) (20) Where the assumptions of the model overlap, pe = 0 G = 0, the results agree. BOYD RICHERSON characterize the equilibria of their model as having quite different properties than the ESSs of (18). This is because they compared the equilibriaof thequantitativegeneticmodel in the presence of phenotypic genetic variance ESS to an analysis in which phenotypic variance was assumed to be absent. It is possible to find the ESS when there is environmental genetic variance.assume that the phenotype of an individual is determined by a single locus a rom deviation. This rom deviation has variance G E, being due to both the effects of other loci environmental effects; both the environmental genetic variation are assumed to be uncorrelated with the alleles at the locusused for invasion analysis. Since BOYD RICHERSON assume that the environmental deviations of interacting relatives are uncorrelated, the phenotypic covariance of individuals will be pcg. From these assumptions we can derive the genotypic fitness function 5. (This is easily done by substituting E G for E in (A2b-f) pcg/(g E ) for pn-note that the phenotypic variance is the sum of the genotypic environmental variances). T h e ESS is a t (14a), which evaluates to (19). Thus we see that if genetic environmental variance are taken into account in an ESS analysis, the resulting equilibria for the mean phenotypeare identical to thosefound in apolygenicmodel with the same genetic environmental variance. This equivalence demonstrates the effect of phenotypic variance: since the genotypic fitness function is determined by the phenotypic fitness function the relationship between genotype phenotype, phenotypic variance will change the shape of the genotypic fitness functionthereforetheuninvadablephenotype found by ESS analysis DISCUSSION I have examined the evolution of a trait that influences the outcome of interactions amongindividuals

9 Evolutionary Quantitative Stability 885 when the trait is inherited as a quantitative character. I assume that the (viability) fitness of an individual is a function of that individual s phenotype of the phenotype of another individual with which it interacts. If x y are the phenotypes of two individuals, the fitness of the x individual after an interaction with y is w(x, y). This assumption induces frequency-dependent selection on the phenotypic character. I find the equilibria of the model assuming that log w can be accurately approximated by its first second derivatives. The most important result is that ESS analysis the present model of quantitative inheritance find the same equilibria1 phenotypes, but the stability conditions for the equilibria of the two models differ. A continuously stable ESS (ESHEL 1983) is determined by three conditions; (4a) which defines an equilibrium, (4b) which determines the uninvadability of an equilibrium, (4c) which determines whether the invasion of dominant mutations of small effect will tend to take the population towards an equilibrium. There are three conditions for a stable equilibrium in the present model, two of the conditions, (17a) (17c), are identical to the conditions (4a) (4c), respectively, while the third condition, (17b), is similar to, but not identical to, condition (4b). The conditions (4b) (17b) are identical when unrelated individuals interact, when identical twins interact, or when the the genotypic fitness function (pictured in Figure 1) is not saddle-shaped. The differences between the stability conditions can be seen as due to the fact that invasion analysis assumes that there are only two genotypes segregating in a population, while models of quantitative inheritance allow a continuous distribution of genotypes. This difference results from the assumption of this paper that the distribution of the breeding values of pairs of related individuals can be approximated by a bivariate normal distribution. This approximation will be valid only when a large number of loci of roughly equal effect determine the inheritance of the phenotype. There seem to be no similar conditions for the evolution of a trait influenced by genes of small effect at a few loci (but see MUELLER FELDMAN 1985), but it is to be hoped that the above conditions for one locus many loci will bracket intermediate cases. ESS theory has been used to investigate situations in which interactions might be expected to maintain phenotypic variance in a character, e.g., the war of attrition model in which the ESS is a distribution of waiting times (MAYNARD SMITH 1982). There are at least three reasons why it does not seem possible to naively apply the results obtained here to models of such interactions. I assume here that the fitness after an interaction can be adequately described by an exponential of a quadratic function of the phenotypes of the interactants-this cannot be an adequate ap- proximation when selection is expected to maintain genetic variance as we have shown that no stable genetically polymorphic equilibria exist when we make this assumption. The equilibrium distribution of phenotypes may not be normal as is assumed here- ESS analysis of the war of attrition suggests that interactions may select for an exponential distribution of phenotypes. Thirdly, if the distribution of phenotypes in the population is not normal, the distribution of interactants cannot be bivariate normal as is assumed here. However, the results obtained here could be used to analyze such models if it is possible, biologically reasonable, to assume that the phenotypic distribution of any one individual can be described as a parametric function to assume that the genotype (plus any environmental deviation) determines the value of the parameters of the phenotypic distribution. The phenotypic fitness function, w can then be obtained as a function of these parameters, then the conditions (4) (17) can be used to find the stable equilibria expected for the parameters. The model used here ignores the effects of mutations, it must be asked how robust the conclusions are. Mutational variance will have two effects, it will cause the equilibrium genetic variance to be non-zero, depending on the distribution of mutational effects, it may cause the allelic effects at each locus to be highly non-normal (TURELLI 1984, 1988). The effect of positive genetic variance on the equilibrium mean phenotype can be taken into account by incorporating the genetic variance covariance into the environmental variance covariance in calculating the coefficients of the genotypic fitness function of (12). This was the method used above to find the evolutionarily stable phenotype in the presence of genetic variance, such a method finds the same equilibria as a quantitative genetic model inwhich mutational effects are explicitly considered (BOYD RICHERSON 1980). This method works because as long as each locus has a small contribution towards the total genetic variance, genetic correlations among the loci are negligible, then the effects of other loci on the variation of phenotypes can be accounted for in the calculation of the genotypic fitness function. The effects of each gene on itsown fitness through its effects on relatives is accounted for by the inclusive fitness function (1 6). Incorporating genetic variance in this way can be used to find the equilibrium mean phenotype defined by (17a), but the effect of muta- tions on the stability conditions (17b, c) is more subtle. TURELLI (1984) discusses the available evidence on the distribution of the genes underlying quantitative traits suggests that alleles at any one locus will often not be normally distributed at a mutation-selec- tion balance. BARTON TURELLI (1987) develop models of the evolution of quantitative traits under arbitrary distributions of alleles. They find that while

10 886 A. B. Harper little can be said about the dynamics of evolution without detailed knowledge of the genes affecting the trait, if the number of loci is large enough then under directional selection the mean should tend to evolve to higher fitnesses, under optimizing selection the mean should evolve toward the optimum the variance tend to contract to some mutation-selection balance. The conditions (17a) (17b) are the conditions that the fitness function induced by pairwise interactions should be optimizing with optimal phenotype x* just when the population has mean phenotype x*, (17a) (17c) are the conditions that the induced fitness function will increase in the direction of x* when the population mean is displaced from x*. This gives us confidence that our results are applicable when genetic effects are not normally distributed, as long as the induced fitness function is the same under different distributions of allelic effects. The induced fitness function is the expectation of w over the distribution of interactants that it meets. If the number of loci is large enough, then we might expect that the distribution of breeding values of relatives will be binormally distributed even if the effects at each locus are non-normal. This observation, together with BARTON TURELLI S (1987) analysis, suggest that conditions (17a-c) will describe evolutionarily stable phenotypes even under a non-normal distribution of genes at each locus, as long as there are enough loci, of approximately equal effect, so that breeding values are approximately normally distributed. To apply these results, however, one must fix the genetic variance at a particular value calculate the terms in (A2) as if the genetic variance covariance were incorporated into the environmental variance. If (17b, c) are then satisfied, this genetic variance the mean phenotype predicted by (1 7a) should be a stable equilibrium under an appropriate, but unknown, mutation rate. Even under frequency independent selection, predicting the genetic variance at equilibrium from the mutation rate is difficult: it would seem that adding frequency dependence could only compound the difficulty. Comparing the results of ESS analysis, of the present model, of the exact models of kin selection illuminates how the genetic basis of inheritance affects the evolution of a character that influences interactions among kin. The exact models of kin selection (CAVALLI-SFORZA FELDMAN 1978; UYENOYAMA FELDMAN 198 1, UYENOYAMA, FELDMAN MUELLER 1981) treat the case when the fitness function w is linear, examine the population genetics of alleles of an arbitrary degree of dominance. In these models it is found that unless the alleles determine the phenotype nearly additively, the existence stability of polymorphic equilibria cannot be predicted using inclusive fitness arguments. For instance, it is possible for a partially recessive allele to be able to invade when rare, then rise to a stable intermediate frequency without being fixed. Such unexpected behavior is found only for alleles of large effect on the phenotype; for alleles of small enough effect, any allele that can invade a population will always be fixed unless the allele is overdominant in its effects on the phenotype (TORO et al. 1982). Comparison of the results of ESS analysis, the present model, the exact models of kin selection suggest the following generalizations. When a character is determined by alleles of substantial effect on the phenotype at one or a few loci, especially when the alleles determine the phenotype nonadditively, one cannot predict thexistence or the stability of genetic equilibria by the use of an inclusive fitness arguments. But when differences in the phenotype are caused by alleles ofsmall effect with primarily additive effects, then the analysis of an inclusive fitness function can be substituted for formal analysis of a genetic model. The appropriate inclusive fitness function arising from invasion analysis is (3), giving stability conditions (4a-c); the inclusive fitness function in the present model is (16), giving stability conditions (17a-c). In most cases, (4a-c) (17a-c) will predict the same stable equilibria; but under some parameter values the stability conditions can differ in such cases both sets of conditions should be used to determine the evolutionarily stable phenotype. J. FEUENSTEIN M. SLATKIN contributed in uncountable ways to this project. M. FELDMAN M. TURELLI kindly helped me underst the implications of their work for the model presented here. Three anonymous reviewers the corresponding editor pointed out a number of errors imprecisions in earlier drafts, have markedly improved the paper. Some of the results were obtained using Macsyma beta release running on a Sun computer at the University of California at Berkeley. Parts of this work were supported by a National Science Foundation Predoctoral Fellowship. LITERATURE CITED AOKI, K., 1982 Additive polygenic formulation of Hamilton s model of kin selection. Heredity 49: BARTON, N. H., M. TURELLI, 1987 Adaptive lscapes, genetic distance the evolution of quantitative characters. Genet. Res. 49: BOYD, R., P. J. RICHERSON, 1980 Effect of phenotypic variation on kin selection. Proc. Natl. Acad. Sci. USA 77: BROWN, J. S., T. L. VINCENT, 1987 A theory for the evolutionary game. Theor. Popul. Biol. 31: BULMER, M. G., 1980 The Mathematical Theory of Population Genetics. Clarendon Press, Oxford. CAVALLI-SFORZA, L. L., M. W. FELDMAN, 1976 Evolution of continuous variation: direct approach through joint distribution of genotypes phenotypes. Proc. Natl. Acad. Sci. USA 73: CAVALLI-SFORZA, L. L., M. W. FELDMAN, 1978 Darwinian selection altruism. Theor. Popul. Biol. 14: CROW, J. F., K. AOKI, 1982 Group selection for a polygenic behavioral trait: a differential proliferation model. Proc. Natl. Acad. Sci. USA 79: DICKENS, D. W., R. A. CLARK, 1987 Games theory

11 Quantitative Evolutionary Stability 887 siblicide in the kittiwake gull, Rissa tridactyla. J. Theor. Biol. 125: ENGELS, W. R., 1983 Evolution of altruistic behavior by kin selection: An alternative approach. Proc. Natl. Acad. Sci. USA 80: ESHEL, l., 1982 Evolutionarily stable strategies viability selection in Mendelian populations. Theor. Popul. Biol. 22: ESHEL, I., 1983 Evolutionary continuous stability. J. Theor. Biol. 103: ESHEL, I., M. W. FELDMAN, 1984 Initial increase of new mutants some continuity properties of ESSin two-locus systems. Am. Nat. 124: ESHEL, I., U. MOTRO, 1981 Kin selection strong evolutionary stability of mutual help. Theor. Popul. Biol FALCONER, D. S., 1981 Introduction to Quantitative Genetics, Ed. 2. Longman, London. FEUENSTEIN, J., 1977 Multivariate normal genetic models with a finite number of loci, pp in Proceedings of the International Conference on Quantitative Genetics, edited by E. POL- LAK, O. KEMPTHORNE T. B. BAILEY, JR. Iowa State University Press, Ames. FELSENSTEIN, J., 1979 Excursions along the interface between disruptive stabilizing selection. Genetics 93: GODFRAY, H. C. J., 1987 The evolution of clutch size in parasitic wasps. Am. Nat. 129: GRAFEN, A., 1979 The hawk-dove game played between relatives. Anim. Behav. 27: HARPER, A. B., 1986 The evolution of begging: sibling competition parent-offspring conflict. Am. Nat. 128: HINES, W. G. S., 1987a Can will a sexual population evolve to an ESS: The multi-locus linkage equilibrium case. J. Theor. Biol. 126: 1-5. HINES, W. G. S., 1987b Evolutionarily stable strategies: a review of basic theory. Theor. Popul. Biol. 31: KARLIN, S., C. MATESSI, 1983 Kin selection altruism. Proc. R. SOC. Lond. Ser. B LANDE, R., 1975 The maintenance of genetic variability by mutation in a polygenic character with linked loci. Genet. Res. 26: LESSARD, S Evolutionary dynamics in frequency-dependent two-phenotype models. Theor. Popul. Biol. 25: MAYNARD SMITH, J., 1981 Will a sexual population evolve to an ESS? AIII. Nat. 117: MAYNARD SMITH, J., 1982 Evolution the Theory of Games. Cambridge University Press, Cambridge. MAYNARD SMITH, J., G. R. PRICE, 1973 The logic of animal conflict. Nature MICHOD, R. E., 1982 The theory of kin selection. Annu. Rev. EcoI. Syst. 13: MORRISON, D. F., 1976 Multivariate Statistical Methods. McGraw- Hill, New York. MUELLER, L. D., M. W. FELDMAN, 1985 Population genetic theory of kin selection: a two-locus model. Am. Nat. 125: PARKER, G. A., M. R. MACNAIR, 1979 Models of parentoffspring conflict. IV. Suppression: evolutionary retaliation by the parent. Anim. Behav. 27: SCHUSTER, P., K. SIGMUND, J. HOFBAUER R. WOLFF, 1981 Selfregulation of behaviour in animal societies. I. Symmetric contests. Biol. Cybernet SLATKIN, M., 1978 On the equilibration of fitnesses by natural selection. Am. Nat. 112: SLATKIN, M., 1979 Frequency- density-dependent selection on a quantitative character. Genetics 93: SLATKIN, M., 1980 Ecological character displacement. Ecology 61: THOMAS, B., 1984 Evolutionary stability: states strategies. Theor. Popul. Biol Toro, M., R. Abugov, B. Charlesworth R. E. MICHOD, 1982 Exact versus heuristic models of kin selection. J. Theor. Biol. 97: TURELLI, M., 1984 Hereditable genetic variation via mutationselection balance: Lerch's Zeta meets the abdominal bristle. Theor. Popul. Biol TURELLI, M., 1988 Population genetic models for polygenic variation evolution, pp in Proceedings ofthe Second International Conference on Quantitative Genetics, edited by B. S. WEIR, E. J. EISEN, M. M. GooDMAN G. NAMKOONG. Sinauer Associates. Sunderl, Mass. UYENOYAMA, M. K., 1984 Inbreeding the evolution of altruism under kin selection: effects on relatedness group structure. Evolution 38: UYENOYAMA, M. K., M.W. FELDMAN, 1981 On relatedness adaptive topography in kin selection. Theor. Popul. Biol. 19: UYENOYAMA, M. K., M. W. FELDMAN L. D. MUELLER, 1981 Population genetic theory of kin selection: multiple alleles at one locus. Proc. Natl. Acad. Sci. USA 78: YOKOYAMA, S., J. FEUENSTEIN, 1978 A model of kin selection for an altruistic trait considered as a quantitative character. Proc. Natl. Acad. Sci. USA 75: Communicating editor: M. TURELLI APPENDIX A The fitness function arising from painvise interactions We want to obtain the genotypic fitness function given the phenotypic fitness function w the phenotypic distribution associated with a genotype. We use G(lX, cy) for the expected fitness of an individual with breeding value <x when interacting with an individual with breeding value ly, GI({.) for the expectation of G(cx, cy) over cy. Equation 11 shows G (Q as a function of the phenotypic fitness function the phenotypic distribution; (1 2) gives us the defi- nition ofg(s;, cy). Equation 13 gives us the phenotypic fitness as a second-order approximation of the fitness function using a Taylor series expansion about a specific phenotype, ZO, letting uz, v,, v,, vxy, v, be the partial derivatives of the logarithm of the phenotypic fitness function with respect to the subscript values. It should be noted that unless v,, v, are negative, vxy suitably bounded, some fitnesses will be indefinitely large, giving absurd results when there is large phenotypic variation. From (12) we have: G(cx, cy) is then the value within the braces, can be obtained in closed form: W x, cy) = JJw(lx + tx, cy + s)p(tx, ty)dtdty (A24

12 888 A. B. Harper c = 1 - E(v,, + 2pt:v, + vyy) + E (1 - pe)(v,,v, -?I:,). (A24 In Equations A2 the u, etc. are the partial derivatives of the logarithm of the phenotypic fitness function w(z,, z,). As can be seen from (A2b); the S, etc. are the partial derivatives of the genotypic fitness function zi,(cx, {,) evaluated at = {, = zo. The phenotypic genotypic fitness functions will usually differ due to th effects of environmental variance. We now want to obtain (Al) in closed form. The distribution of the breeding values is known from (9): Let {x, cy, MI be measured as deviations from the phenotype about which we have exped the fitness function: {x = zo + 6 ~, {, = zo + d,, M, = zo + 6u, (A4) substitute into (A3) Substituting (A5) (A2) into (Al), we can find the fitness of an individual whose breeding value differs by 6, from Zo: s G(6,) = W(z0 + a, ZO + 6y)p(6, I Qd6, J exp(s,6, + Cy6, + 5,,62/2 + C,6x6y + ~,,6;/2)P(6Y I 6 W Y (A6) Examining the G = 6 equilibrium first, (Bla) implies that it is stable to perturbations in V C just when da/dv < 0. The stability conditions for M are determined by the sign of am,+,/am,. Since A = 0 at this equilibrium, (Sc) gives us am,+l/am, = 1 + dab/am. (B2) Equations Bla B2 determine the following cqnditions for the existence of a stable equilibrium with G = G: The first two conditions define the existence of the equilibrium, the third is the requirement that the genetic variance be non-negative, the fourth comes from APPENDIX A as a condition for (A6) to converge, the last two are the conditions for the stability. If the condition a, < 0 fails, then the average fitness of the population, IJ {,). p,(s;, {y)d{xd{y, is infinite, suggesting that G is so large that zi, is not adequately approximated byaa quadratic function. Using (46) apd (A7), the value G that gives A = 0 can be found: G = G /( 1 - pz), where or The first equilikrium is an artifact of the order of integration in (AI), as at G = 0 we are not integrating over a welldefined density. At the second, we have where = exp(a6f/2 + B6,), a, = C, - pc( 1 - pc)&,/g a, = 5,- PC( 1 - pc)gm/g axr = G,, - p?;/g G = (1 - p%)g, ax, = Sxy + pc/g a, = C, - l/g (A7) if a, < 0. (When a,, > 0 the integral diverges). Equation A6 is in the form of (7b); (A6) (8) are the recursion equations for this model. We can see that the value of A in (A6) is independent of 6 therefore the recursion Equations 8a 8b are independent of 8c. APPENDIX B Stability criteria There are two types of equilibria for Equations 8, either there is no selection on the character, A = B = 0, G has some value G, or there is no heritable variation G = 0. To find the stability of these classes of equilibria, we must look at the first derivatives of (8). Because (8a, b) are independent of M, we can consider the stability of these two equations together, then consider the stability of (8c). The Jacobian matrix for [V, C] at G = G is I+- i: 2aA/av 6 aa/av 2n (n - 1)6?aA/av (n - ly?aa/av 1 -r+ 2n the Jacobian at G = 0 is 2n, (Bl4 The following argument shows thath equilibrium (B4b) is always unstable when it exists. From (A7), a, = (G G, - l)/g, which is <O by assumption, implying G S, < 1. In order for aa/av < 0 we would need the numerator of (B5) to be positive, or S.Sw - Sf > 0. Now note that using (B4b) Since the denominator of (B6) must be positive, we have an > 0 we find that conditions (B3) cannot all be true. This sho-ws, in effect, that if there were a stable equilibrium at G = G, then the average fitness of the population would be infinite, which suggests that such equilibria could exist if either w were a more complicated function, not approximated by a second-order Taylor series (SLATKIN 1980), or if the distribution of phenotypes were limited to a finite set of values. It is difficult to deal with such cases analytically. We can now examine the stability conditions of the G = 0 equilibrium. We see that the Jacobian matrix (B1 b) implies stability in C if r > 0 but neutral stability in V. To determine the stability in the V direction, we must look at a V,+~/aV~ at V = C = 0, if this second derivative is <0 then the equilibrium is stable to all perturbations in V C. This second derivative is A/n, we see that the G = 0 equilibrium is stable if A < ) Finding the correct stability conditions for M is not as

13 Evclutionary Quantitative Stability 889 straightforward, since (8c) implies that any value of M is a neutrally stable equilibrium for G = 0. As suggested by FELSENSTEIN (1979) we can exp the equations out, not at G = 0, but at some small value for G, G = 6 ~ This. would be appropriate if there were a constant flux of mutations that increase the variance of the trait. If M changes when G is small but non-zero, then it seems unlikely that this value of M would be maintained in a population. It should be noted that in the recursion equations V, C, enter only as G, = V, + C,, so that we do not have to distinguish between the mutational effects at one locus effects between loci. Substituting G = 6C into (8c) we see that the equilibrium value mean phenotype is that dm,+,/dm, < 1 when M = -B/A, A + db/dm < 0. (B8) (B9) In order to compare the above to the ESS conditions, we need to evaluate (B7-9) when the point, zo, around which we have approximated the phenotypic fitness function, w, is the equilibrium, M. This will give us conditions which the partial derivatives of w, evaluated at a certain phenotype, must satisfy in order for that phenotype to be a stable equilibrium of the model. It is also convenient to choose a scale of measurement such that the equilibrium phenotype M is 0. Evaluating the conditions above (in the order B8, B7, B9) at zo = &, = M = 0, using (A6) (A7), gives us the conditions listed in (14a-c). To compare the conditions (14) with the ESS conditions (4), we need to find the ESSs for the model developed above. We do this by retaining the effects of environmental variation, but replacing the assumption of quantitative genetic variation with the ESS assumption that the population is genetically monomorphic except for the invasion of a single rare allele. When there is environmental variation in the expression of the phenotype with variance E covariance between the interactants of p,&, then fitness of individuals with breeding values x y is ;(x, y), where zi, is defined in (A2b). To find the ESS, we just apply the ESS conditions (4) to (A2b) at x = y = ZO, giving: 5, +?+y = 0, (B 1 Oa) S, , +.;YY + r(1 - r)s; < 0, (Blob) Cxx + (1 + r)c, + ri& < 0. (B 1 Oc) In Equations B10, r is the probability of a rare genotype interacting with another rare genotype; this is the same as the genetic correlation between interactants, pc, so that conditions (B10) can be compared to (14) by substituting pc for r.