Edward Pollak and Muhamad Sabran. Manuscript received September 23, Accepted for publication May 2, 1992

Transcription

1 Copyright Q 199 by the Genetics Society of America On the Theory of Partially Inbreeding Finite Populations Fixation Probabilities Under Partial Selfing When Heterozygotes Are Intermediate in Viability Edward Pollak Muhamad Sabran Department of Statistics, Iowa State University, Ames, Iowa Manuscript received September 3, Accepted for publication May, 199 B ABSTRACT In a previous paper by the senior author, an approximation to the probability of survival was given for a mutant, which is originally present in a single heterozygote, in a population that reproduces partly by selfing partly by rom mating. The population was assumed to be very large, but the result obtained is general with regard to the level of dominance in viability. In this paper two errors which were made in that earlier work are corrected. A general approximate expression is then derived for the probability that an allele A is fixed in a partially self fertilizing population of size N, if its initial frequency is p, selection is weak heterozygotes with the allele are exactly intermediate viability compared with genotypes AA a-. A rigorous proof is given for a special case that is a generalization of the classical binomial sampling model. In this case, but not in general, the approximate fixation probability is independent of the probability of reproduction by selfing.someimplications are discussed. Y using the theory of branching processes, POL- LAK (1987) derived an approximation to the probability that an allele A ultimately survives in a large population in which individuals reproduce by selfing with probability /3 by rom mating with probability 1-8. This approximation is general with regard to the level of dominance of A in viability, but is based on the assumption that the population size N is very large. It would be desirable if the theory could be extended to moderately large values of N by the use of the diffusion approximation that has proven to be so effective for dealing with finite rom mating pop ulations. Unfortunately, the theory of diffusion approximations has not yet been generalized in this direction. It is, however, possible to imitate the reasoning of MORAN (I 96 l) derive an expression for the probability of fixation of A if its initial frequency is p AA, AA- a-genotypes have zygote-toadult survival probabilities in the proportions 1 + s: 1 + s:l. In a special case that is a generalization of the classical binomial sampling model, it is possibie to give a rigorous derivation if we assume that s is small. The resulting formula does not depend upon /3 is thus identical to what was obtained by KIMURA (196), who used the Kolmogorov backward differential equation. If we apply this approach to a more general model than the one for which we can give a theoretically sound derivation, a more general formula, which does depend upon 8, can be obtained. This heuristically derived expression does have the satisfying property Genetics 131: (August. 199) of reducing to the approximate formula of POLLAK (1987) as N becomes very large. In our discussion we shall use our expressions for fixation probabilities to develop the theory of expected limits to mass selection in a finite population. As far as we can determine, the results are of the same form as the corresponding results obtained by ROB- ERTSON (1960), in the sense that the expected response to selection in the long run is approximately equal to the response in one generation times twice the effective population size. Before proceeding to the main results that have just been described, some errors made by POLLAK (1987) will be corrected in the next section. SURVIVAL PROBABILITIES IN LARGE POPULATIONS Following the reasoning in a previous paper (POL- LAK 1987) we assume that there is a large population in which a mutant allele A is originally present in one individual of genotype AX, where A represents any allele that differs from A. Then, at least approximately, each individual carrying A at time t gives rise to a line that develops independently of lines descended from other individuals of the same generation. Such lines can contain individuals of types AA AX if the probability of selfing is not negligible. Therefore the process of change in the numbers of AA AX individuals can be modeled as a two-type branching process, in which AA AX are defined to be of types 1, respectively.

2 980 E. Pollak M. Sabran Now let the probabilities of survival to adulthood of zygotes of types AA, A x xx be in the propor- tions of 1 + sl:l + sp:l, where s1 s are assumed to be small. Then, as shown by POLLAK (1987), P (survival of A 1 one Ax ancestor) where G is the number of successful gametes produced by a particular individual when there is no selection which is the inbreeding coefficient in the long run when the population size is infinite. When the population size N is large, N', the reciprocal of the probability that two gametes contributing to rom separate adults come from same the parent was shown to be approximately equal to ( - P)N/ [Var(G) + ( 1 - P)]. Thus the right side of (1) may be rewritten as (N'/N)[Ffssl + (1 - FIS)~~]. It was, however, erroneously stated by POLLAK (1987) that this quantity is equal to (1 + FIs) times the effective population size Ne, rather than Ne itself. A correct derivation will now be given by making use of Equation 5 in that paper, which gives expressions at time t for F,, the inbreeding coefficient of a rom individual B,, the coancestry of two rom separate individuals. Thus, population size derived from inbreeding theory should satisfy the approximate equation 1 -Ftk 1" (1 -Ft-l) ( ) ; when t is large. Because 1 - FIs = (1 - 0) 1 + FIs = /( - P) Ne N - (1 - FIs) + (1 + FIs)Var(G)' As pointed out by CABALLERO HILL (199), G is equal to twice the number of selfed offspring plus the number of offspring from rom mating. Thus (3) is consistent with their expression for Ne. A second error made by POLLAK (1987) was to assume that Var(G) = when there is a Poisson distribution of offspring. Instead, if the number of selfed nonselfed progeny have independent Poisson distributions, &=4@+(1-P)=+/3 in which case (3) reduces to ( + ~FIs)/( 1 + FIS), 4N Ne = (1 - FIs) + + 6FIs as given by LI (1976, p. 56). In general, (1) takes the form P(surviva1 of A I one AX ancestor) N =- 1 + FIs' (3) (4) = (1 - &)Ft-l Ne Nc -[FISSI + (1- FIS)~~], (5) N where (Ne/N) is given by (3). If, in particular, (4) holds s1 = s = s, P(surviva1 of A I one A x ancestor) = s, (6) as found by HALDANE (197). 1 -(1-&)= 1 -(1 -&)(l "- 1 + ( 1" ;)Bt-1 Ne as t This means that when t is large, (NJ-I is approximately equal to (N')-', the probability that two rom copies of a gene among offspring are derived from one successful gamete of the previous generation. Thus Ne = N' not N'/(l + FIS). The error in the earlier derivation was in replacing N by N' in the original recurrence equation for B,, which is legitimate if p = 1/N, as in an idealized monoecious population, but not otherwise. In general the effective FIXATIONPROBABILITIESWHEN HETEROZYGOTES ARE INTERMEDIATE We note that if the survival probabilities of AA, AX xx are in the proportions 1 + s:l + s:l, (5) reduces to (6), which does not depend upon p. We shall show that if s is small, a more general approxi- mate expression for the survival probability is independent of the degree of departure from rom mating. Let us consider apopulation in which the frequencies of AA, AX Ax among zygotes of generation t are PI P10 Poo the frequencies of A x are p q. 1 - P be the probabilities that offspring arise from selfing rom mating. Be-

3 Selfing Partial with Fixation 981 cause there is not complete rom mating there is a nonzero numberf, such that P11 = p +fpq, PlO = pq( 1-5). By deterministic theory, the expected frequencies of AA AX among zygotes in generation t + 1 are where PIl(1 PI1 = (1- P)p: + + s) + %P10(1 + s) P 1 + sp Pll(1 + s) + PlO(1 + s) pl = = Pi Pi0 sp is the frequency of A1 among zygotes in generation t + 1. Thus, if s is small, (P +f,pq)(l + s) + (1 -f,)p4(1 + pl 4 = 1 + sp which is consistent with WRIGHT (1969, pp ). If there is selectionf, it not equal to the inbreeding coefficient, but, if s is small, it should not be very different from FIs. This will be shown to be the case in the APPENDIX. We also know that the mean squared shift from p is approximately pq/(ne) before many generations have passed. Thus, if u(p) is the probability that A is ultimately fixed, given that its initial frequency is p, it is reasonable to expect that it is approximately the solution of with the boundary conditions u(0) = 0, u( 1) = 1. Upon solving this equation we obtain ters N( 1 - P) pl, independently, when there is selfing, the distribution of the numbers of zygotes of the three genotypes is trinomial with parameters NP p = PIl(1 + s) + 1/P10(1 + s) PlO(1 + s) p3 = 1 + sp ' p4= 1 -p-p sp Let p ' be the frequency of A in generation t + 1, given its frequency in generation t is p. The moment generating function of the distribution of N(p' - p) is then M(6) = E[e'NQ"P) 1 = e-'nv[ 1 + pl@e - l)]'n'"8) - [I + p(eze - 1) + p3(ee - 1)IN8. Thus the cumulant generating function is K(e) = ~ep + ~ ( 1 - P)ln[l + pl(es - I)] + NPh[ 1 + p(ez8-1) + ps(es - I)] (IO) = Kld + 1/Kd + '/6K3d3 + '/4K4d It is clear from (10) that the first four cumulants are of the same order of magnitude as N. The first two of these are - PNspq( 1 +f,) = NsPq( 1 +f,) O(s), sp s-0, t If, in particular, Ne = N/(1 + FIs), (9) reduces to the expression obtained by KIMURA (196) for a rom mating population. Thus, in this case u(p) is independent of (3. The foregoing heuristic argument can be made rigorous in a special case, which is based upon the following assumptions. First, if we assume that there are fixed numbers NP N(l - (3) of offspring in each generation that arise from selfing rom mating, respectively. Second, if there is rom mating, the distribution of the number of A alleles among zygotes of generation t + 1 is binomial with parame- - as s 0. Moments can now be derived from the cumulants from stard expressions, stated, for example, by KENDALL STUART (1977, p. 71). Thus, if s is of

4 98 E. Pollak M. Sabran the same order of magnitude as N-, Now it follows from inequalities (A.9) (A. 10) in the APPENDIX that 8 > s[l + 5/0 - %p + NS] + 0(s)) = e* 41 + s[l - 5/0 + + o(s )} = ol. Because we are therefore considering 8 that is of the same order of magnitude as s, i.e., O(N ), (1 5) (1 6) imply that the neglected terms O(8 ) in the curly brackets are negligible compared with the remainder. Thus, if s 8 are small, Therefore ~[~-N@o(P f)] < 1.= E[~-~N@I(P P)]. E[e- N@Op lp] < e- N@oP If we substitute -8 for 8 in the moment generating function we obtain ~(-0) = E[~- N@(P P)I = 1 e + - E[(N(p E[(N(p - 4 where 0 < C#J < 8. Third fourth moments of N(p - p) must vanish with p = 0 or p = 1 hence contain a factor p(l - p). Thus, if 8 is small, it follows from (1 3)-( 16) that M(-8) = 1 - Nf@{s(l + A)/(l + sp) - - I9 * (1 +J) + -(1 - #)( + B + ( + 3PS) + s~pqs (1 + I + O(s) + O(8 )). S (17) Following the reasoning of MORAN (1961) we first ignore the term that is O(8 ). The remainder of the expression in the curved brackets is positive or negative according to whether 8 is less than or greater than where o* = 4 1 +$)A d a? = 1 + (P/)( 1 +A) + (s/)( 1 - px + P + ( + 3PK) + NPqs( 1 +A) when s 19 are small positive numbers. qe-n4p I p ] > e-n*lp. Because A is either fixed or lost in the long run it follows that, if p is the initial frequency of A, + [ 1 - u( p)] < e-neop u(p)e-n81 + [I - u(p)] > e-n@lp. Therefore 1 - e- N@oP 1 - e- N@lP 1 - e-n@o < u(p) 1 - e-n@l * (18) If, however, s is small, both Bo O1 are approximately equal to s. Thus, as s decreases, u(p) lies in between two terms that both approach the right side of (9), in the special case in which N, = N/(1 + Fls). If Ne = N/( 1 + Fls) p is equal to 1/N, then (9) reduces to s 1 - e-4ns which agrees with the result obtained by FISHER (1930) WRIGHT (1931). As N + m, the right side of this equation tends to s; this agrees with (6). DISCUSSION In the special case that has just been discussed the fixation probability u( p) is approximately independent of /3 when s is small. Equation 3 indicates that this is not true in general because 8 ( - P)N 4Ne( 1 + F IS) = (1-0) + a: 16N - (1-0) + a:

5 Selfing Fixation with Partial 983 When ab = (1 this reduces to 4N, but it isby referring to work of GHAI (198) COCKERHAM dependent otherwise. WEIR (1984), who showed that in an equilibrium Another implication of (9) is that if Nd << 1, then population with two alleles the parent-offspring co- 4 x 1,......r. variance is equal to (1 1 + FIs)u?/. The additive Ne( 1 + Frs)sp] genetic variance in the corresponding rom mating ds[' - Ne(1 + F1s)s] (19).. population is u;, which in our case is equal to pqa. " i p + N4l + FIs)p(l - p). This covariance is also equal to the covariance between the phenotypic value of the offspring the Now let us consider mass selection on a quantitative phenotypic mean of the parents, which we denote by character for which there is additive gene action, so cov(0,m). The variance of the mean of the parents is that the genotypic values of AA, AX can be written as a, 0, -a, respectively. Let the stard- Var(M) = Yz(1 + ized selection differential, or the intensity of selection, because in a large population romly mated individ- be denoted by i. Then, if the phenotypic values associated with each genotype have a normal distribution with a stard deviation up that is large in comparison with a, it can be shown that uals have uncorrelated phenotypes, but the conditional covariance of the parental phenotypes is u; if there is selfing. If, therefore, we denote the selection differential by I, we obtain the regression equation approximately. [See, for example, FALCONER (1989, pp. 01, 09, whose derivation is not based upon the population having a Hardy-Weinberg array of geno- 'Y pes. 1 Now the expected change in the long run in the genotypic mean is L = Mp)a - (1-4 P)b) - Np +fop+ - (q + fopq)a) = [u(p) - lla - I9 - q1a = a[u(p) - p]. Hence it is clear from (19) (0) that L f Ne(1 + F,s)["l. pqa UP i Now let us consider R, the expected response to selection in one generation. We note thathe mean of the quantitative character in generation 0 is (p - q)a = (p - 1)a. Hence, by (13) (0), the expected response is approximately a i Pq(l +fo) -. UP Moreover, expression (A.6) in the appendix allows us to make the further approximation a i R 5 ( 1 + FIs)pq -. UP By combining (1) () we obtain () L 5 NeR, (3) which is consistent with the result obtained by ROB- ERTSON (1960). Expression () can be derived in an alternative way which agrees with (). In the particular case in which Ne = N/(1 + F1s), it can be seen from () (3) that the response to selection in one generation is (1 + FIs) times as large > 0 as it is when /3 = 0. But the expected limit L remains the same, so that partial inbreeding results in more rapid progress toward the limit in this case. We have not derived a general expression for the fixation probability of an allele A when heterozygotes are exactly intermediate in viability. If, however, p = 1/(N), Equation 9 reduces to which is consistent with (5), if s, = s = s. We are grateful to A. CABALLERO anonymous referee for helpful suggestions. Journal Paper No of the Iowa Agriculture Home Economics Experiment Station, Ames, Iowa, Project No LITERATURE CITED CABALLERO, A., W. G. HILL, 199 Effective size of nonrom mating populations. Genetics 130: COCKERHAM, C. C., B. S. WEIR, 1984 Covariances of relatives stemming from a population undergoing mixed self rom mating. Biometrics 40: FALCONER, D. S., 1989 Introduction lo Quantitative Genetics, Ed. 3. Wiley, New York. FISHER, R. A., 1930 The Genetical Theory of Natural Solution. Clarendon Press, Oxford. GHAI, G. L., 198 Covariances among relatives in populations under mixed self-fertilization rom mating. Biometrics 38: HALDANE, J. B. S., 197 A mathematical theory of natural

6 I, 984 E. Pollak artificial selection, Part V: Selection mutation. Proc. Cambridge Philos. SOC. 43: KENDALL, M. G., A. STUART, 1977 The Advanced Theory of Statistics, Volume 1, Distribution Theory. Macmillan, New York. KIMURA, M., 196 On the probability of fixation of mutant genes in a population. Genetics 47: LI, C. C., 1976 First Course in Population Genetics. The Boxwood Press, Pacific Grove, Calif. MORAN, P. A. P., 1961 The survival of a mutant under general conditions. Proc. Cambridge Philos. SOC. 57: POLLAK, E., 1987 On the theory of partially inbreeding finite populations. I. Partial selfing. Genetics 117: ROBERTSON, A., 1960 A theory of limits in artificial selection. Proc. R. SOC. B 153: WRIGHT, S., 1931 Evolution in Mendelian populations. Genetics 16: WRIGHT, S., 1969 Evolution the Genetics of Populations. Vol., The Theory of Gene Frequencies. The University of Chicago Press, Chicago. Communicating editor: B. S. WEIR APPENDIX Given that P11, P10 POO are the frequencies of AA, AX Xx in generation t, the expected frequency of heterozygotes in generation t + 1 is P;o = (1 -J+l)plql, where q1 = 1 - Pl. Thus, by (7), M. Sabran it follows that Hence Thus, by (A. 1) (A.) < pq + Ap < pq(1 + s). 1 Pq 1 <-<- 1 + s p1q1 1 - s',b 1 + 7s + 8s <-[ 1 4s + 4s +h] + 3s + 4s =q s + 4s +h] 1 < q 1 +3s+4s+& Hence By the repeated application of inequalities (A.3) (A.4) it can be shown that -P -P < DL -8 + (;)(fo - O h ), 64.5) We also have, from (1 3), that the expected shift in the frequency of A, given that it is p in generation t, is if s > 0. Because where C = 1-3s/(l - s) D = 1 + 3s + 4s. Inequalities (A.5) illustrate the fact that ft differs only by terms of order of magnitude O(s) from the value it would have if there were no selection. We now assume that the population has been subjected to the same mating system for many generations. Thus, if A is absent from the population before time 0,fo = P/( - P), whereas if it is present Therefore, by (A.5), P C- <fo<d P' P C- <J<D--- -P - P'

7 Hence, by (A.6) fi + (3s + 4s)@ 1 + (3s + 4s)P/) -- - ( + 4s)@/ + + (3s + ( + 4s) = I-- + (3s + 4sy Fixation with Partial Selfing <1+sp+-@(-P)s+(s-sp) ( 9 +sp(l-p)~+ns+0(s) 1 +fi 5 3 = 1 +s[l +-p---p+ns]+o(s) 4 1 +?(l - p) l+fi 1 +fi (1 + sp)[p P + ( + 3afi 3P( - P)s = I + [ - (4 + %)SI' It follows from (A.7) (A.8) that +@+(+3@& 1 +fi + NPqs(1 +J)] > 1 - qp( - 3 S)s + s (A. 10)