Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes

Transcription

1 Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes Simona Grusea, Willy Rodríguez, Didier Pinchon, Lounès Chikhi, Simon Boitard, Olivier Mazet To cite this version: Simona Grusea, Willy Rodríguez, Didier Pinchon, Lounès Chikhi, Simon Boitard, et al.. Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes. Journal of Mathematical Biology, Springer Verlag (Germany), 8. <hal-698> HAL Id: hal Submitted on 9 Nov 7 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

2 Coalescence times for three genes provide sufficient information to distinguish population structure from population size changes Simona Grusea,a, Willy Rodríguez a, Didier Pinchon b, Lounès Chikhi c,d,e, Simon Boitard f, and Olivier Mazet a a Institut de Mathématiques de Toulouse, Université de Toulouse, Institut National des Sciences Appliquées, F-77 Toulouse, France b Institut de Mathématiques de Toulouse, Université de Toulouse, F-77 Toulouse, France c CNRS, Université Paul Sabatier, ENFA, UMR 574 EDB (Laboratoire Évolution & Diversité Biologique), Bât. 4R, F-6 Toulouse, France d Université de Toulouse, UPS, EDB, F-6 Toulouse, France e Instituto Gulbenkian de Ciência, Rua da Quinta Grande, No. 6, P Oeiras, Portugal f GenPhySE, Université de Toulouse, INRA, INPT, INP-ENVT, Castanet Tolosan, France November 9, 7 Corresponding author. grusea@insa-toulouse.fr

3 Abstract The increasing amount of genomic data currently available is expanding the horizons of population genetics inference. A wide range of methods have been published allowing to detect and date major changes in population size during the history of species. At the same time, there has been an increasing recognition that population structure can generate genetic data similar to those generated under models of population size change. Recently, Mazet et al. [6] introduced the idea that, for any model of population structure, it is always possible to find a panmictic model with a particular function of population size change having an identical distribution of T (the time of the first coalescence for a sample of size two). This implies that there is an identifiability problem between a panmictic and a structured model when we base our analysis only on T. A natural question that deserves to be explored is whether and when this identifiability problem disappears for larger sample sizes. In this paper, based on an analytical study of the rate matrix (or Q-matrix) of the ancestral lineage process, we obtain new theoretical results about the joint distribution of the coalescence times (T, T ) for a sample of three haploid genes in a n-island model with constant size. In particular, we show that this distribution is always different from the analogous one obtained in a panmictic population, for any scenario of population size-change. Even if, for any k, it is always possible to find a size-change scenario for a panmictic population such that the marginal distribution of T is exactly the same as in a n-island model with constant population size, we show that the joint distribution of the coalescence times (T, T ) for a sample of three genes contains enough information to distinguish between a panmictic population and a n-island model of constant size. Keywords: IICR (inverse instantaneous coalescence rate); population structure; population size change; demographic history; rate matrix; structured coalescent.

4 Introduction Coalescent theory was developed in the 98s by theoretical population geneticists interested in the statistical properties of gene trees [Kingman, 98, Hudson, 98, Tajima, 98]. By focusing on the properties of samples within populations, coalescent theory allowed a significant reduction in the computational cost of simulations [Hudson et al., 99, Hudson, ]. Also, by stressing the importance of backward inference, it allowed new insights in our understanding of the shapes of gene genealogies obtained from real species [Slatkin, 99, Rogers and Harpending, 99]. This change in focus (a backward sample view instead of a forward population view) allowed the development of new methods to infer the demographic history of populations and species [Beaumont, 999, Hudson,, Nielsen and Wakeley, ] and the detection of ancient population size changes in many species [Storz and Beaumont,, Goossens et al., 6, Quéméré et al., ]. However, there has been an increasing recognition that population structure can generate false signatures of population size change. Indeed, coalescent theory predicts that gene trees obtained from bottlenecked populations may be similar in shape to those obtained from structured populations [Wakeley, 999, Storz and Beaumont,, Mazet et al., 5]. Significant work has been done to find a solution to this inference problem, most of which is based on simulations rather than analytical work [Beaumont, 4, Chikhi et al.,, Peter et al.,, Heller et al., ]. One reason for this is that the coalescent theory of structured population is difficult. Indeed, while it is straightforward to simulate coalescent times for nearly any model of population structure and any sampling scheme [Hudson, ], few theoretical results exist regarding the distributions of these times for sample sizes above two. Thus, while there has been significant progress in our understanding of coalescent theory for a wide variety of models, several difficult problems in demographic inference could still be addressed using analytical approaches. For instance, Mazet et al. [5] used the distribution of the coalescence time T for a sample of two haploid genomes (or one diploid individual) to separate two models for which these distributions could be derived. The first model was a structured model, the n-island model of Wright [9], whereas the second model was a simple panmictic model with only one stepwise population size change. In that particular case they showed that the distributions were different and could thus be separated with a reasonably limited number of independent values of T. This provided a promising result, since it suggested that genomic data from a single diploid individual could be enough to separate one model of population structure from a model of population size change. Such a method would contribute to solve problems pointed in several simulation studies based on larger sample sizes [Chikhi et al.,, Peter et al.,, Heller et al., ]. However, in a more recent study, Mazet et al. [6] have shown that, given the distribution of the coalescence time T obtained under any model of population structure, there always exists a function λ( ) of population size-change which perfectly mimics this distribution. This function was derived for a sample of size two for the n-island model and called IICR, which stands for inverse instantaneous coalescence rate. In other words, the T distribution alone does not allow distinguishing between a panmictic population whose size can vary arbitrarily and a structured population, however complex that structure may be, as they will have exactly the same IICR. Mazet et al. [6] noted that this limitation might be overcome by increasing the sample size to more than two haploid genomes and including information from other coalescence times. They also noted that the sampling scheme could potentially also be used to separate two demographic models, because the IICR is a function of the sampling scheme, an issue that has been explored by Chikhi et al. [8]. Altogether, obtaining the distribution of different T k, for several sample sizes k and several sampling schemes, could in principle allow us determining whether these distributions differ significantly for models of population size changes and different models of population structure

5 Chikhi et al. [8]. This would be crucial for demographic inference, in an era where genomic data are becoming increasingly available. In this article we focus on the joint distribution of the coalescence times (T (), T () ) for a sample of three haploid genes and show that it can be used to distinguish between panmixia with population size changes and a n-island model with constant size. Throughout this article, the superscript () identifies the fact that we consider a sample of genes ; in particular, it allows distinguishing the second coalescence time T () for a sample of three genes from the coalescence time T () for a sample of only genes, which we simply denote T. In Section we derive new theoretical results about the joint distribution of (T (), T () ) in a n-island model with constant size, for n. In particular, we explicitly diagonalise the rate matrix (also called Q-matrix) of the ancestral lineage process associated to this model and obtain closed analytic expressions for the transition probabilities. We also study in detail the distribution of T () and the population size change model mimicking this distribution, as previously done by Mazet et al. [6] for a sample of size. In Section we use the results obtained in Section to compare the joint distributions of (T (), T () ) for a panmictic population and a n-island model with n, and demonstrate that these two distributions are always different, for any population size change and any sampling configuration. Appendix A contains proofs of the results presented in Sections and. In Appendix B we treat the special case of a -island model and give analogs for the main results presented in Sections and. Joint distribution of coalescence times for three genes. Panmictic model with population size changes Consider a panmictic population whose population size history is represented by the function λ(.), i.e. whereby the population size at time t in the past is given by N(t) = N λ(t), with N being the present population size. We are interested in the coalescence times for a sample of three (haploid) genes. After rescaling time by units of N generations and taking N, we let T (),λ and T (),λ denote the first and second coalescence times for the sample, under this model. Using known results on the coalescent in populations of variable size (see for example Griffiths and Tavaré [994]), the distribution of the first coalescence time, T (),λ, is given by where Λ(t) = t ds. λ(s) P(T (),λ > t) = exp{ Λ(t)}, () Conditional on T (),λ, the distribution of the second coalescence time T (),λ is given by P(T (),λ > u T (),λ = t) = exp{ (Λ(t + u) Λ(t))}. The marginal distribution of T (),λ is thus obtained by integrating over all possible values of t: P(T (),λ > u) = = P(T (),λ > u T (),λ = t)f (),λ T (t)dt exp{ Λ(t)} exp{ (Λ(t + u) Λ(t))}dt. λ(t) 4

6 . The symmetrical n-island model Let us now consider the symmetrical n-island model, with n islands (also named subpopulations or demes), where N is the haploid size of each subpopulation, supposed constant over time. In the same way as in Wilkinson-Herbots [998], we rescale time by units of N generations and take N, in such a way that in this new continuous time scale, two genes in a single subpopulation have coalescence rate (going backwards in time) and we call M/ the scaled backward migration rate (i.e. the rate at which each gene leaves its subpopulation when we go backwards in time). In other words, if we call m the proportion of immigrant genes in each subpopulation (forward in time), we have M = Nm. For more details on the structured coalescent process see Herbots [994], Wilkinson-Herbots [998] and Notohara [99]. As in the previous section, we are interested in the coalescence times for a sample of three genes. We denote T (),n,m and T (),n,m the first and second coalescence times for the sample, respectively. Every pair of genes in the same subpopulation may coalesce at rate and every gene migrates at rate M/. Due to population structure, we now need to distinguish three different configurations for the sample (for the case where n ) : () the three genes were sampled from the same deme, () two genes were sampled from the same deme and the third gene was sampled from a different deme, or () the three genes were sampled from three different demes. For i =,,, we will denote P i the conditional probability starting from sampling configuration i. The special case of n = islands needs a special treatment, since in this case the situation () cannot exist. We will treat this case in Appendix B... The ancestral lineage process Consider three (haploid) genes sampled in a population described by a symmetrical n-island model with n islands and migration parameter M >. In order to study the distribution of the first coalescence time T (),n,m of these three genes, we introduce a Markovian jump process (called ancestral lineage process) describing the configuration in which the ancestral lineages of the three genes are at any time in the past, with the time going backwards, until their first coalescence event. The possible configurations, denoted by i =,..., 5, are the following :. the three lineages are in the same island,. two lineages are in the same island and the third one is in a different island,. the three lineages are all in different islands, 4. there are only two ancestral lineages left and they are in the same island, 5. there are only two ancestral lineages left and they are in different islands. The transition rate matrix Q of this Markovian jump process (also called Q-matrix, see Norris [998] for definition) can be easily constructed (see Rodriguez [6] and Rodriguez et al. [X] for more details): M M Q = M (n ) n M n, () M(n ) (n ) M(n ) M n These transition rates are easily obtained using the fact that any two lineages among the three coalesce at rate if they are in the same deme, and any of the three lineages migrates at rate 5

7 M ; when a lineage migrates, it migrates to one of the other n islands with equal probability. Given that we stop the process at the first coalescence event, the states 4 and 5 are absorbing. Note that Herbots [994] introduced the structured coalescent and gave an expression for the rate matrix Q in a general model of population structure. She considered an ancestral lineage process describing the number of ancestral lineages in each of the n-islands, backwards in time. In this article we only consider the case of a symmetrical n-island model, and thanks to the symmetries, we only need to consider the five configurations (or states) presented above and the reduced rate matrix Q given in Equation. Let P t := e tq, for t >, be the transition kernel of the above Markovian jump process (see again Norris [998] for details). By definition, P t (i, j) represents the probability of the process being in state j at time t, conditional on starting in state i at time. The cumulative distribution function (cdf ) of the first coalescence time T (),n,m can easily be expressed in terms of the matrix P t, as follows : P i (T (),n,m t) = P t (i, 4) + P t (i, 5). () Moreover, using the fact that d dt P t = P t Q, we have the relations d dt P t(i, 4) = P t (i, ), d dt P t(i, 5) = P t (i, ), from which we deduce the following expression for the density function of T (),n,m : f T (),n,m,i (t) = d dt P i(t (),n,m t) = P t (i, ) + P t (i, ). (4) In order to obtain a closed form expression for P t, and consequently for the distribution of T (),n,m (see equation ()), we study below the diagonalization of the rate matrix Q... A closed form expression for P t The characteristic polynomial of Q is given by χ Q (µ) := det(q µi 5 ) = µ p(µ), with p(µ) = µ + 8 n Mn µ + Mn 4 M + n 4 n + Mn + + M n n (n ) µ + 9 M ( n + Mn ) (n ). Thus, the matrix Q has the double eigenvalue. Concerning the other three eigenvalues, we can obtain the first lemma below: Lemma. The polynomial p(µ) has three distinct strictly negative real roots, µ < µ < µ <, such that µ < nm n < µ < n < µ <. The proof of this lemma is given in Appendix A. Using the Cardan-Viète method to express the roots of a third degree polynomial, such as exposed e.g. in Nickalls [99], classical computations lead to the following explicit expressions for µ, µ and µ : 6

8 Proposition. Let us define the real numbers γ, K, x, r and α by γ = M n, K = 5γn + 8, 6 x = (γn )(γ n + 4γn + 54γ + 4), r = 7γ n + 6γn 7γ + 8, α = arccos(x/r ). Then the three negative real roots µ < µ < µ < of p(µ) are given by µ = r cos(α) + K, µ = r cos(α + 4π ) + K, µ = r cos(α + π ) + K. Using the diagonalization of the rate matrix Q, we finally obtain the following expression for the transition kernel P t : Proposition. The transition kernel P t is given for any t > by P t = e µ it A(µ i ) + B, (5) i= where we denote A(µ) = with δ(m, n, µ) M u (n )Mu (n )(n )M 9M u v µv Mu Mu (n )uv (n )(n )Mv µ M (n )(n )M v 9M (n )Mv u µ (n )Mu µ (n )uv µ (n )Mv µ B = b b b b b b,, δ(m, n, µ) = (n ) p (µ), u = u(m, n, µ) = (n )µ + M, v = v(m, µ) = µ + (M + ), b = b (M, n) = M + (n ) nm + (n ), b = b (M, n) = The proof of the above proposition is deferred to Appendix A. M nm + (n ). 7

9 .. Joint distribution of (T (),n,m, T (),n,m ) The joint distribution function of the two coalescence times for three genes sampled in configuration i =,, can be obtained as follows: P i (T (),n,m u, T (),n,m t) = P i (T (),n,m u, T (),n,m t, C s ) + P i (T (),n,m u, T (),n,m t, C d ), where C s (respectively C d ) denotes the event that, after the first coalescence, the two remaining lineages are in the same deme (respectively in two different demes). Using the Markovian property of the ancestral lineage process and the fact that the states 4 and 5 are absorbing, we have P i (T (),n,m u, T (),n,m t, C s ) = F,s (u)p t (i, 4), P i (T (),n,m u, T (),n,m t, C d ) = F,d (u)p t (i, 5), where F,s (resp. F,d ) denotes (using the notations in Mazet et al. [5]) the distribution function of the coalescence time T,s (resp. T,d ) for a sample of two genes taken in the same deme (resp. in two different demes). We thus deduce P i (T (),n,m u, T (),n,m t) = F,s (u)p t (i, 4) + F,d (u)p t (i, 5). (6) We can specify the above formula by plugging the expressions for F,s and F,d obtained in previous studies (see Herbots [994] and Mazet et al. [5]): where F,s (t) = P(T,s t) = a ( ) e αt + a ( ) e βt, (7) α β F,d (t) = P(T,d t) = c ( ) e αt c ( ) e βt, (8) α β a = γ α β α, c = γ β α and α and β are the roots of the polynomial q(x) = X +(+nγ)x +γ, whose discriminant equals = ( + nγ) 4γ, and therefore α = ( + nγ + ), β = ( + nγ ), () with γ = M = αβ. Note that α < β < are the negative eigenvalues of the analogous n rate matrix of the ancestral lineage process for a sample of size. From Equation (6) we derive the conditional distribution of T (),n,m given the value of T (),n,m, as follows P i (T (),n,m u T (),n,m = t) = F,s(u) d P dt t(i, 4) + F,d (u) d P dt t(i, 5) f (t) (),n,m T,i (9) = F,s(u)P t (i, ) + F,d (u)p t (i, ). () P t (i, ) + P t (i, ) Moreover, by Equation (6), the marginal distribution function of T (),n,m can be expressed as: P i (T (),n,m u) = F,s (u) lim t P t (i, 4) + F,d (u) lim t P t (i, 5). Using the results in Proposition, we finally obtain explicitly the above limits and hence the distribution function of T (),n,m : 8

10 Proposition 4. For every u > we have: P (T (),n,m u) = M + n Mn + n F,s(u) + and for i = and i =, P i (T (),n,m u) = with F,s and F,d given by Equations (7) and (8). ( M + n ) F,d (u), () Mn + n ( ) M Mn + n F M,s(u) + F,d (u), () Mn + n The expected value of T (),n,m can be deduced from the above proposition. Indeed, we have E (T (),n,m ) = M + n ( Mn + n E(T,s) + M + n ) E(T,d ). Mn + n Using the fact that E(T,s ) = n and E(T,d ) = n + n M that For i = or i =, we get and thus E i (T (),n,m ) = E (T (),n,m ) = n + (see [Herbots, 994]), we easily deduce (n ) Mn + n. ( M Mn + n E(T,s) + ) M E(T,d ), Mn + n E i (T (),n,m ) = n + (n ) (M + ), i =,. M(Mn + n ) Note that the same formulae were obtained in Herbots [994] using Laplace transform methods...4 Distribution of T (),n,m and corresponding population size-change model Equation (4) and Proposition provide analytic expressions for the probability density function of T (),n,m for each of the three sampling configurations. In guise of example, in Figure we show the plot of the density of T (),n,m for three genes sampled from the same deme (i.e. in sampling configuration i = ), for n = and M = (in blue), and M =. (in red). Considering a sample of two genes, Mazet et al. [6] introduced a function λ( ) called the inverse instantaneous coalescence rate (IICR), which they defined as λ(t) = P(T > t), t. (4) f T (t) They showed that the distribution of their coalescence time T can always be expressed as a function of λ( ), as follows: P(T > t) = exp{ Λ(t)}, Λ(t) = t λ(s) ds. This expression holds for any model of population structure, but it is also exactly what we expect for a panmictic model whose population size along time is described by N(t) = N()λ(t). This implies that for a structured model of any complexity, including thus the symmetrical 9

11 Probability density function of T (),n,m M = M = Time Figure : Plot of the probability density function of T (),n,m for three genes sampled from the same deme (i.e. in sampling configuration i = ) for a n-island model with n = and M = (blue), and M =. (red). n-island model considered here, there always exists a panmictic model with population size changes that perfectly fits the T distribution of this model. A similar result can be obtained for the first coalescence time (denoted simply T k ) for a sample of size k, for any k. However, the population size history mimicking the distribution of T k is not exactly equal to the IICR, but to ( k ) times the IICR. This comes from the fact that the first coalescence in a sample of size k arrives ( k ) times faster than in a sample of size, because all pairs of lineages can coalesce. In particular, the population size function mimicking the distribution of the first coalescence time T, for a sample of three genes, is times larger than the IICR corresponding to T, which can easily be seen by derivating and re-arranging equation () in order to obtain the corresponding IICR. We study below this population size-change function for the symmetrical n-island model, and compare it asymptotically when t to that obtained by Mazet et al. [6] for a sample of size. We will use the notation λ( ) for this population size-change function, rather than for the IICR. Another way of addressing this issue would be to define the IICR k = ( k ) IICR. This notation issue did not arise in [Mazet et al., 6], since in the particular case of k = the two quantities are equal, namely IICR = IICR. Based on results from the previous section, the population size-change function mimicking the distribution of T (),n,m, for a sample of three genes in configuration i =,,, is given by λ i (t) = P i(t (),n,m > t) f (t) = P t(i, 4) P t (i, 5) (),n,m T P,i t (i, ) + P t (i, ). (5) In Figure we plot this function for the different sampling schemes (i =,, ), for a model with n = islands and migration parameter M = (left panel) and M =. (right panel). We can see that the three population size functions converge to a same asymptotic value when t, which is confirmed by the following Proposition 5. Proposition 5. When t, λ i (t), i =,,, have the following limit lim t λ i(t) = µ, (6) where µ is the largest of the three distinct negative eigenvalues of the rate matrix Q.

12 λ (t) n = M = 4 λ (t) n = M =. /µ λ (t) λ (t) /µ λ (t) λ (t) 4 6 t t Figure : Plot of λ i ( ) i =,, for n = and M = (left), and M =. (right). The dashed red line corresponds to the asymptotic value /µ. The proof is given in Appendix A. This result is strikingly similar to that of Mazet et al. [6] for a sample of size. Indeed, the population size-change function mimicking the distribution of T, for two genes sampled either from the same or from different demes, converges to /β, where β is the largest of the two negative eigenvalues of the corresponding rate matrix (see Mazet et al. [6]); an explicit formula for β is given in Equation ()). As pointed out by Mazet et al. [6], /β can be related to the notion of effective population size in a structured population. In particular, N/β is always strictly larger than the total population size nn, and for large M it is actually equal to the effective population size proposed by Nei and Takahata [99]. For a sample of size, the inequality /n < µ in Lemma implies that the asymptotic value of N λ i ( ) is also strictly larger than nn. However, as demonstrated below (Lemma 6 and Remark 7), this asymptotic value is also strictly larger than N/β. This highlights the ambiguity inherent to the notion of effective population size, as even this equilibrium value actually depends on the sample size used to compute it. See also Chikhi et al. [8] for a discussion on several crucial differences between the IICR and the notion of N e. Lemma 6. We have α < µ < µ < µ < β <, (7) where µ < µ < µ < are the ordered roots of p(µ) and α < β < are defined in Equation (). The proof is given in Appendix A. Remark 7. Together with Proposition 5, the result in Lemma 6 implies that, for i =,,, we have lim t λ i(t) > β. Another similarity with the results of Mazet et al. [6] is that λ i ( ) is increasing when all genes are sampled from the same island (i = ), and decreasing when all genes are samples from different islands (i = ) (see Figure ). We do not provide a formal proof of this statement, only the following intuition: when all genes are sampled from the same island, the corresponding IICR is initially lower than the asymptote, because the coalescence rate is greater in this state than in all other states of the ancestral lineage process. In contrast, when all genes are sampled

13 from different islands, coalescence is initially impossible because a migration has to occur first, so the IICR is infinite. The last sampling configuration i = (two genes sampled from the same island and the third one from a different island) does not exist for a sample of size, and is thus specific to the case k =. As expected intuitively, the curve of the corresponding IICR, λ ( ), lies between the two other curves, and when the migration rate M is low the curve becomes increasingly closer to that of λ ( ). Using the joint distribution of coalescence times (T (), T () ) to distinguish structure from panmixia We will now compare the two models described in Subsections. and. : a panmictic model with population size-change function λ( ), and a symmetrical n-island model with parameters n and M >. As previously, we denote T (),λ and T (),λ the two coalescence times in the panmictic model, and T (),n,m and T (),n,m the two coalescence times in the n-island model. We want to show that the joint distribution of (T (),λ, T (),λ ) is always different from the joint distribution of (T (),n,m, T (),n,m ), for all values of n, M, λ( ) and all initial sampling configurations (i =,, ) for the n-island model. In order to do so, we will fix n, M and a sampling configuration i, and consider the function λ i ( ) defined in Equation (5), for which the distributions of the first coalescence times T (),n,m and T (),λ i are the same. We will show that even in this case, the distribution of T (),λ i is different from that of T (),n,m. This demonstration will be based on the study of the conditional distributions of T () given T () in the two models. For several values of n and M, we will also quantify the difference between the two models by comparing the marginal distributions of the second coalescence times T (),n,m and T (),λ i the case when the distributions of the first coalescence times T (),n,m and T (),λ i, in are the same.. Comparison of the conditional distributions of T () given T () We start by comparing, for a given t > and a given sampling configuration i =,,, the conditional distribution functions P(T (),λ i T (),λ i = t) and P i (T (),n,m T (),n,m = t). Using the results from Subsection., we deduce that P(T (),λ i u T (),λ i = t) = exp{ (Λ i (t + u) Λ i (t))} ( ) / Pt+u (i, 4) P t+u (i, 5) =. (8) P t (i, 4) P t (i, 5) On the other hand, an expression for P i (T (),n,m T (),n,m = t) was given in Equation (). In order to compare these two conditional distributions, let us introduce the functions g i (u, t), i =,,, defined for u, t > by g i (u, t) := P i (T (),n,m u T (),n,m = t) P(T (),λ i u T (),λ i = t). In Figures and 4 we plot respectively the functions g (u, t) and g (u, t) as functions of u and t, in the case of a symmetrical n-island model with n = and M = (left panels), and M =. (right panels). In these particular cases, we clearly see that there exists at least one pair (u, t) for which g i (u, t) is different from zero, i.e. for which the two conditional distributions are different. In order to demonstrate that this is the case for any choice of n, M and i, we further study in detail the behaviour of the functions g i (u, t) in the neighborhood of (u, t) = (, ) (Proposition 8) and their asymptotic behaviour when u (Proposition 9) or t (Proposition ).

14 ..8 z = g (u,t) n = M =..5 z = g (u,t) n = M = z = z = -. u 4 t u 4 5 t 4 5 Figure : Difference of the conditional cdf in Equations () and (8) as a function of u and t, for n = and M = (left), and M =. (right) and three genes sampled from the same deme z = g (u,t) n = M = -.4 z = g (u,t) n = M = u 4 5 t 4 5 u 4 5 t 4 5 Figure 4: Difference of the conditional cdf in Equations () and (8) as a function of u and t, for n = and M = (left), and M =. (right) and three genes, two of which were sampled from the same deme and the third one from a different deme. Note that the z axis is inversed, for the sake of readability.

15 Proposition 8. For (u, t) in the neighborhood of (, ) we have (i) P (T (),n,m u T (),n,m = t) = u M + u M ut + o(u + t ), P (T (),λ u T (),λ = t) = u M + u Mut + o(u + t ). As a consequence, we have g (u, t) = M ut + o(u + t ) as (u, t) (, ). (ii) (iii) P (T (),n,m u T (),n,m = t) = M (n ) u + M (n ) ut + o(u + t ), P (T (),λ u T (),λ = t) = M(n ) + n u u 8(n ) (n )M (n ) ut + o(u + t ). Thus g (u, t) = u + Mn + n u M(n + ) + 8(n ) 6(n ) ut + o(u + t ) as (u, t) (, ). P (T (),n,m u T (),n,m = t) = P (T (),λ u T (),λ = t) = Thus g (u, t) = M 4(n ) ut + o(u + t ) as (u, t) (, ). M (n ) u + M 4(n ) ut + o(u + t ), M (n ) u + M n ut + o(u + t ). Proposition 9. For fixed t >, when u +, where K,i (n, M, t) > is given by g i (u, t) = K,i (n, M, t)e βu + o(e βu ), i =,,, (9) K,i (n, M, t) = P t (i, ) a P t (i, ) + P t (i, ) β with the constants β, a, c defined in Equations (9) and (). P t (i, ) c P t (i, ) + P t (i, ) β, () Proposition. For fixed u >, we have where K (n, M, u) > is given by lim g i(u, t) = K (n, M, u), i =,,, () t + K (n, M, u) = c e βu + c e αu e µ u, () with φ(µ ) = M (n )µ + Mn + 6(n ), c = φ(µ ) α β α, c = β φ(µ ) β α. () 4

16 The proofs of these three propositions are given in Appendix A. Propositions 8, 9 and imply that for every n and M >, we can always find at least one pair (u, t) for which the two conditional distributions in Equations () and (8) are different. Indeed, such pairs can easily be exhibited for small values of both u and t, or for large values of u or t. As a consequence, the joint distribution of (T (),n,m, T (),n,m ) for a sample of three genes is always different from the distribution of (T (),λ, T (),λ ) in a panmictic population, for any possible size-change function λ( ) and any initial sampling configuration. The results of this section also provide interesting insights into the comparison of coalescence times between panmictic and symmetrical n-island models. When the three genes are sampled from two or three different islands (i =, ), g i (u, t) is negative at least in the neighboorhood of (, ) and for sufficiently large values of u or t. This is in line with Figure 4, which suggests that g i (u, t) is always negative; if true, this implies that, for a panmictic and a symmetrical n-island model that have the same distribution of T (), the cdf of T () is always smaller in the n-island model, which actually means that T () is stochastically larger in the n-island model than in the panmictic model. The situation is more complex when the three genes are sampled from the same deme. For fixed t > and sufficiently large values of u, g (u, t) is also negative, meaning that, given that T () has the same value in the panmictic population and in the symmetrical n-island model, T () tends to be asymptotically larger in the n-island model than in the panmictic case. However, for small values of both u and t, g (u, t) is positive, so for a sufficiently small common value T () = t and for small values of the second coalescence time T (), this last coalescence time tends to be smaller in the n-island model than in the panmictic case. Actually, this latter situation is by far the most relevant in practice, because the asymptotics when u is reached very slowly (see Figure ) and the probability density of T () is mostly concentrated on small values of t (see Figure ). Note also that, in the neighbourhood of (, ), the leading term of g i (u, t) is very different depending on i: it is of order ut and depends linearly on M for i = or i =, while it is of order u and independent of M for i =.. Comparison of the marginal distributions of T () The results of the previous section prove that the joint distributions of (T (),λ i, T (),λ i ) and (T (),n,m, T (),n,m ) always differ, even if the size-change function λ i ( ) is such that T (),λ i and T (),n,m have the same distribution. In order to quantify the distance between the coalescence time distributions in the panmictic case versus the symmetrical n-island model, it is easier in practice (even if less informative) to compare only the marginal distributions of T (),λ i and T (),n,m, in the case when the size-change function λ i ( ) is such that the distributions of T (),λ i and T (),n,m are the same. Using the results from the previous section, we have P(T (),λ i u) = = P(T (),λ i > u T (),λ i ( Pt+u (i, 4) P t+u (i, 5) P t (i, 4) P t (i, 5) = t)f (),λ T i (t)dt ) / [P t (i, ) + P t (i, )]dt, which we will compare with the expression for P i (T (),n,m u) provided by Equations () and (). Figure 5 shows these two marginal distribution functions for a symmetrical n-island model with n = and M = (left), and M =. (right), and three genes sampled in the same deme 5

17 Cumulative distribution function of T () n-island with n =,M = the panmictic model having the same T () distribution 4 5 Time Cumulative distribution function of T () n-island with n=, M =. the panmictic model having the same T () distribution 4 5 Time Figure 5: Comparison of the cdf s of T (),λ and T (),n,m for three genes sampled in the same deme, for a n-island model with n = and M = (left), and M =. (right) n = n = n = 6 n = D(n, M) Migration rate M Figure 6: -Wasserstein distance D (n, M) between the distributions of T (),n,m and T (),λ as a function of M, for different values of n. (i.e. in sample configuration i = ). As discussed in the previous section, for this particular sampling configuration, the cdf of T () appears to be larger (hence T () stochastically smaller) in the n-island model than in the panmictic case. We compare the marginal probability distributions of T (),n,m and T (),λ i using the - Wasserstein distance (see e.g. Vallander [97]), which we denote D i (n, M) : = = P i (T (),n,m u) P(T (),λ i u) du g i (u, t)[p t (i, ) + P t (i, )]dt du. In Figure 6 we show several plots of D (n, M) as a function of M, for different values of n, in the case of three genes sampled in the same deme (i = ). We see that, as expected, the difference between the two distributions increases with n, the number of demes, and decreases when M is large or very small. When M increases the n-island model tends towards a large panmictic population of size nn. When M is very small, each deme is nearly isolated from the other demes, and thus increasingly behaves as a simple panmictic population of size N. 6

18 4 Discussion and perspectives In this study we obtain new theoretical results about the joint distribution of the coalescence times (T (), T () ) for a sample of three genes in a symmetrical n-island model with constant size, for all possible sampling configurations. When comparing this distribution with the analogous one in a panmictic population with population size changes, we show that for any size-change function λ( ), the two distributions are different. Indeed, it is always possible to construct a size-change function λ( ) which perfectly mimics the distribution of T () in a symmetrical n-island model; but we show that even in this case, the conditional distributions of T () given T () in this panmictic model and in the n-island model are different. Consequently, our results imply that the joint distribution of coalescence times for a sample of three genes contains enough information to distinguish between a panmictic population and a symmetrical n-island model of constant size. As illustrated by Figures 5 and 6, the difference between the two models can be substantial, even in the case where these models lead to the same distribution of T (). Although this result is extremely important from a theoretical perspective, we note that, currently, it cannot be directly implemented into a model choice procedure based on real genomic data. Indeed, to our knowledge, no statistical method has yet been proposed to estimate neither the joint distribution of (T (), T () ) or the marginal distribution of T () from genomic data. However, some important progresses have been made recently into this direction. For instance, the distribution of the first coalescence time T k in a sample of size k can be estimated quite accurately by the MSMC approach [Schiffels and Durbin, 4], and other approaches might be implemented in the near future to estimate also more ancient coalescent times (see for instance the recent work of Weissman and Hallatschek [7]). Besides, the fact that considering jointly several coalescence times allows discriminating structured and panmictic models, might be exploited by statistical procedures based on the Site Frequency Spectrum (SFS). Indeed, this quantity is directly observable from genomic data, as it records the proportion of genomic positions with,,... n copies of the mutant allele in a sample of n genes, and the limit of this quantity for large numbers of loci can be expressed as a linear function of the successive expected coalescence times E(T n ), E(T n ),..., E(T ) [Griffiths and Tavaré, 998]. Moreover, it should be possible in theory to apply the ideas presented in this article to compare the joint distribution of (T (), T () ) between a panmictic population and a structured population for more general models of population structure. In this article we focused on the symmetrical n-island model, but our results can be straightforwardly generalized to any model of population structure, provided that we can compute the corresponding transition kernel P t. The theoretical results of Herbots (see Herbots [994], Wilkinson-Herbots [998]) allow one to write the rate matrix Q for any given model of population structure under the structured coalescent. One important issue is that complex models are characterized by very large rate matrices Q, on the order of n n for k =. For a model with n demes, it may be difficult to study the transition kernel P t, when the rate matrix Q has on the order of, elements. This is currently explored by Rodriguez et al. [X] who show that the IICR k can be obtained for highly complex models. Altogether, the formal proofs presented here and the work of Rodriguez et al. [X] suggest that demographic inference under complex models of population structure may become easier in the next few years. 7

19 Appendix A. Proofs for Sections and Proof of Lemma We first observe that Then the calculus gives ( p ( p nm n lim µ nm n < n <. 9M(Mn + n ) (n ) >, p() = ) = 9(Mn + (n )(n )) <, n n ) (Mn + n )(M(n ) + n ) = > (n ) p(µ) =. The intermediate value theorem applied to p(µ) thus provides a proof of the result. Proof of Proposition In order to simplify computations, we make the following change of parameters: M = N, n = a + N. a The new parameters a et N verify a > and N > and the matrix Q becomes N N a + a N a + N Q = 6 a 6 a. The matrix Q has the double eigenvalue and it is easy to check that the corresponding eigenspace is generated by the following non colinear vectors: V = [N, N, N, N, N a ] T, V = [,,,, ] T. We then consider the change of basis matrix N N P = N, N whose inverse is P = N a a+ N+a+ N N+a+ a N+a+ N+ N+a+ a N+a+ N+ N+a+ N+a+ N+a+ N a N+a+ 8 N N+a+.

20 We have thus obtained a partial diagonalization (by blocks) of the matrix Q: N N a + a N a + N Q = P QP = 6 a 6 a, (4) and we put R := N N a + a N a + N 6 a 6 a. The characteristic polynomial of R is the polynomial p(µ), which has the following expressions using the new parameters: p(µ) = µ + (5N + 5a + 4)µ + (N + 4aN + N + a + 7a + )µ + 8a(N + a + ), which has three strictly negative real roots. These roots are all distinct by Lemma. If µ is an eigenvalue of R, we then determine a corresponding eigenvector W (µ) by solving the equation RW (µ) = µw (µ). The computations show that we can choose W (µ) = µ + (N + 5a + )µ + 6a(a + ) a(µ + 6a) 6a. We then consider the passage matrix P, whose column vectors are the W (µ i ), i =,,, where the µ i are the three eigenvalues of R: P = [W (µ ), W (µ ), W (µ )]. Some easy computations on the rows show that the determinant of P is a Van der Monde determinant, and hence: det(p ) = (µ µ )(µ µ )(µ µ ). The computations of the inverse P gives P = [Z(µ ), Z(µ ), Z(µ )] T, where Z(µ) = p (µ) µ + N + a µ + (N + a + 4)µ + (N + a + ) aµ, with p (µ) = µ + (5N + 5a + 4)µ + (N + 4aN + N + a + 7a + ). 9

21 We further obtain e tr = P = e µ t e µ t e µ t e µ j W (µ j )Z(µ j ) T. j= P, We then introduce the matrices A(µ) and B defined by A(µ) := P W (µ) Z(µ) T P, where W j (µ) (resp. Z(µ)) is a vector of length 5 obtained by adding two null coordinates to W (µ) (resp. Z(µ)), and B = P P. Emphasizing the rank one property of matrix A(µ), let us define the vectors E = M v M u, E = Mu (n )uv (n )(n )Mv Mu µ (n )uv µ where µ is an eigenvalue of R, and u, v are functions of µ defined by Note that, since M n and (M + ) u = (n )µ + M, v = µ + (M + )., ( p M ) 9 (n )[(n ) + (n )M]M = <, n (n ) ( ) (M + ) p = With δ(m, n, µ) = (n ) p (µ), we have 98 [(n ) + (n )M]M (n ) >, cannot be eigenvalues and thus u et v. A(µ) = which gives the expression of A(µ) in Proposition. The stated result easily follows. δ(m, n, µ) E E T, (5)

22 Proof of Proposition 5 Using Equation (5) and Proposition, we deduce that lim λ i(t) = A(µ )(i, 4) + A(µ )(i, 5), i =,,. t A(µ )(i, ) + A(µ )(i, ) Note that the matrix E E T introduced in the proof of Proposition can also be factorized in a different manner. If we define the matrix C, the 5 matrix D and the 5 matrix D by u M u v C := u uv v, D := M v, u M M µ D := n n µ, (n )(n ) we may check that, for any value of µ, we have E E T = D CD. The vectors V and V, defined by V := µ, V :=, µ verify D V j = for j =,, and hence, using Equation (5), A(µ)V j =, for j =,. Therefore, for i =,,, and the result follows. Proof of Lemma 6 A(µ)(i, ) µa(µ)(i, 4) =, A(µ)(i, ) µa(µ)(i, 5) =, Note that α and β, with α, β defined in Equation (), are the roots of the polynomial q (X) = q(x/), where ( q(x) = X + + nm ) X + M n n, and on the open subset D = {(n, M) : n >, M > } of R the polynomials p(x) and q (X) have a common root if and only their resultant R(n, M) = Res(p(X), q (X), X) with respect to X is null (see e.g. [Lang,, chapter IV-8]). Because R(n, M) = M < on D, and because the roots α and µ 9(n ) are continuous functions of (n, M) on D, the inequality α < µ for one value of (n, M) D implies the inequality everywhere on D. The same is true for the inequality µ < β. This achieves the proof of the lemma.

23 Proof of Proposition 8 We will only give the proof of (i), which corresponds to the case of three genes sampled from the same deme. The proofs ot the analougous results (ii) and (iii), corresponding to the two other sample schemes, are similar. When t, using the well-known relation P t = I 5 +tq+o(t), where I 5 denotes the identity matrix, we have the following Taylor expansions : P t (, ) = + ( M ) t + o(t), In particular, we have P t (, ) = M t + o(t). P t (, ) + P t (, ) = (M + )t + o(t) = + (M + )t + o(t). Using Equations (7), (8), (9), () and (), and a Taylor expansion of order of the exponential in the neighborhood of, we easily obtain F,s (u) = u M + u + o(u ), M F,d (u) = (n ) u + o(u ). When substituting the above expressions into Equation (), straightforward calculations give P (T (),n,m u T (),n,m = t) = u M ut M + u + o(u + t ). Further, using Equation (8), let us denote h(u, t) := P(T (),λ u T (),λ = t) = ( ) / Pt+u (, 4) P t+u (, 5). P t (, 4) P t (, 5) We will write a Taylor expansion of order of h(u, t) in the neighborhood of (, ). We have h(, ) = and direct computations give h u (u, t) = ( P t(, 4) P t (, 5)) / ( P t+u (, 4) P t+u (, 5)) / (P t+u Q(, 4) + P t+u Q(, 5)), h t (u, t) = ( P t+u(, 4) P t+u (, 5)) / ( P t (, 4) P t (, 5)) 4/ {(P t+u Q(, 4) + P t+u Q(, 5)) ( P t (, 4) P t (, 5)) (P t Q(, 4) + P t Q(, 5)) ( P t+u (, 4) P t+u (, 5))}. Using the fact that P = I 5, and Q(, 4) =, Q(, 5) =, we obtain h h (, ) =, (, ) =. u t We further compute the second partial derivatives of h in (, ). With Q being the square matrix of the rate matrix Q, and using the fact that Q (, 4) = 9 ( M + ) and Q (, 5) = M,

24 we easily derive h u t (, ) = { (Q(, 4) + Q(, 5)) + Q (, 4) + Q (, 5) } = M, h u (, ) = { } (Q(, 4) + Q(, 5)) + Q (, 4) + Q (, 5) = (M + ), h (, ) =, t The Taylor expansion of order of h(u, t) near (, ) finally gives which finishes the proof. Proof of Proposition 9 P(T (),λ u T (),λ = t) = u Mut M + u + o(u + t ), Because < β < α, using Equations (7) and (8), we get and F,s (u) = a α e αu a β e βu = a β e βu + o(e βu ) F,d (u) = c α e αu + c β e βu = + c β e βu + o(e βu ). Therefore, using Equation (), we obtain P i (T (),n,m u T (),n,m = t) = K,i (n, M, t)e βu + o(e βu ), where K,i (n, M, t), given by (), is strictly positive because < a < and c <. Using Proposition we get P t+u (i, j) = (A(µ )(i, ) + A(µ )(i, ) + A(µ )(i, )) e µ (t+u) + o(e µ (t+u) ). j= Therefore, using Equation (8) and relation P t (i, 4) P t (i, 5) = j= P t(i, j), we obtain where P(T (),λ i u T (),λ i = t) = K,i (n, M, t) e µ u + o(e µ u ), ( ) A(µ )(i, ) + A(µ )(i, ) + A(µ )(i, ) µ K,i (n, M, t) := e t. P t (i, ) + P t (i, ) + P t (i, ) Because µ < β from Lemma 6, we have e µ u = o(e βu ), which achieves the proof of (9). Proof of Proposition We will first prove a useful lemma. Lemma. Let us define φ(µ) by φ(µ) := where the matrix A(µ) is given in Proposition. Then, A(µ)(, ) A(µ)(, ) + A(µ)(, ),

25 (i) < φ(µ i ) < µ i (ii) Defining h(n) by < α, i =,,. we have the following results: h(n) = (n )(5 9n + 5 9n 8n + 5) n, (a) If M > h(n), then φ(µ ) < β < φ(µ ) < φ(µ ) < α. (b) If M = h(n), then φ(µ ) < β = φ(µ ) < φ(µ ) < α. (c) If M < h(n), then φ(µ ) < φ(µ ) < β < φ(µ ) < α. Proof. From Proposition and using p(µ i ) =, we get φ(µ i ) = (n )µ i + (nm + n )µ i + 6M Mn + (n ) ) = (n ) ( nm + (n ) q µi M(µ in + ) nm + (n ) µ i. From Lemma, it follows that µn + >. Because α < µ i < β from Lemma 6, we get q ( µ i ) <, that proves φ(µi ) < µ i. Now, using again that p(µ i ) =, we obtain the following expression for φ(µ i ) φ(µ i ) = M (n )µ i + nm + 6(n ). (6) M Introducing the new variable ν =, and thus (n )µ + nm + 6(n ) µ = (nm + n ))ν M, we get that φ(µ i ), for i =,, are the three real roots of the (n )ν polynomial r(ν) = 4(n )(nm + n )ν [n M + (n )(n + 4)M + 8(n ) ]ν +M(nM + 5n 5)ν M. The coefficient signs of r(x) show that r(x) has no negative roots, implying that φ(µ i ) >, i =,,, which achieves the proof of (i). The set of (n, M) such that r(x) and q( X) have a common root is obtained by computing their resultant with respect to X, denoted by R(n, M) : R(n, M) = Res(r(X), q( X), X) = M [ 6n M + (n )(9n 5)M 6(n ) ]. In the domain D = {(n, M), n >, M > } the curve 6n M +(n )(9n 5)M 6(n ) = is identical to the graph of the function M = h(n), n >. The set D \ {(n, h(n)) : n > } has two connex open components in which the relative position of β, α and φ(µ i ), i =,, are the same. 4

26 In the component D := {(n, M) : n >, M > h(n)}, one may choose n =, M = 4 for which β = 57 6 α = φ(µ ) = 5 4 φ(µ ) =,.4694,.7585,.998, φ(µ ) = , that proves (ii a). In the component D := {(n, M) : n >, M < h(n)}, one may choose n =, M = for which that proves (ii c). β = , 8 α = , 8 φ(µ ).489, φ(µ ).4659, φ(µ ).8868, The curve arc {(n, h(n)) : n > } may be parametrized by n = u + 8u, M = (u ) 8u, u ], [, (u + ) and we obtain β = u u +, α = (u + ) u +, φ(µ ) = 9u + 5 7u + 44u + 59, 4(u + ) φ(µ ) = u u +, φ(µ ) = 9u u + 44u + 59, 4(u + ) and (ii b) is proved. Lemma is now used to prove Proposition. First note that we have φ(µ) = A(µ)(i, ) A(µ)(i, ) + A(µ)(i, ), 5

27 for every i =,,. When t +, P t (i, j) = A(µ )(i, j)e µ t + o(e µ t ), j =, and thus P i (T (),n,m u T (),n,m = t) A(µ )(i, ) = A(µ )(i, ) + A(µ )(i, ) F A(µ )(i, ),s(u) + A(µ )(i, ) + A(µ )(i, ) F,d(u) + o() = φ(µ )F,s (u) + ( φ(µ ))F,d (u) + o(). Using the definitions of F,s (u) and F,d (u) in Equations (7) and (8), we get P i (T (),n,m u T (),n,m = t) = c e βu c e αu + o(), where c and c are given by (). On another hand, we have P t (i, j) = A(µ )(i, j)e µ t + o(e µ t ), j =,,, we obtain P t+u (i, j) = ((A(µ )(i, ) + A(µ )(i, ) + A(µ )(i, )) e µ (t+u) + o(e µ t ), j= P t (i, j) = ((A(µ )(i, ) + A(µ )(i, ) + A(µ )(i, )) e µ t + o(e µ t ). j= Using the fact that P t (i, 4) P t (i, 5) = j= P t(i, j), this implies and thus () holds. It remains to show that K (n, M, u) >. Using c + c =, we may write K (n, M, u) = β φ(µ ) β α P(T (),λ i u T (),λ i = t) = e µ u + o(), ( ) e αu e µ u + φ(µ ) α ( ) e µ u e βu. β α From Lemma 6, α < µ < β and thus e αu < e µ u < e βu. On the other hand, from Lemma we have φ(µ ) < β < α. Therefore K (n, M, u) >. 6