THE FATE OF A RECESSIVE ALLELE IN

Transcription

1 THE FATE OF A RECESSIVE ALLELE IN A MENDELIAN DIPLOID MODEL Dissertation zur Erlangung des Doktorgrades Dr. rer. nat. der Mathematisch-Naturwissenschatlichen Fakultät der Rheinischen Friedrich-Wilhelms-Universität Bonn vorgelegt von Rebecca Sarah Ströer aus Leverkusen Bonn, Mai 2017

2

3 Angeertigt mit Genehmigung der Mathematisch-Naturwissenschatlichen Fakultät der Rheinischen Friedrich-Wilhelms-Universität Bonn 1. Gutachter: Pro. Dr. Anton Bovier 2. Gutachter: Pro. Dr. Patrik Ferrari Tag der Promotion: Erscheinungsjahr: 2017

4

5 Abstract The omnipresence o sexual reproduction in a highly competitive world is a ascinating phenomenon. Its evolution and maintenance is up to now not completely understood since it is known that sexual reproduction require rom organisms a lot more costs and energy than asexual reproduction would do. In this thesis we show mathematically that sexual reproduction provides populations an evolutionary advantage because they can better adapt to a changing ecological system. To this end, we study a stochastic individual based model which describes the genetic evolution o a diploid hermaphroditic population reproducing sexually according to Mendelian laws. This single locus model describes a population o interacting individuals that incorporate the canonical genetic mechanisms o birth, death, mutation, and competition. In the irst part o this thesis the genetic evolution o the population with two alleles is studied under the assumption that a dominant allele is also the ittest one. It is shown that ater the invasion o a dominant allele in a resident population o homozygous recessive genotypes, the recessive allele survives in heterozygous individuals or a time o order at least K 1/2 α, where K is the carrying capacity and α > 0. This time o survival o the unit allele is much longer than it would be in a population reproducing asexually. Thereore, a suitable rescaling o the mutation rate made the appearance o a new advantageous mutation possible beore the extinction o the recessive allele. In the second part o this thesis, we study the ate o the recessive allele ater the occurrence o a urther mutation to a more dominant allele. It is proven that resulting changes in the composition o the population indeed opens the possibility that individuals o homozygous recessive genotype can reinvade and that coexistence o dierent genotypes is possible. This leads to genetic variability and can be seen as a statement o genetic robustness exhibited by diploid populations perorming sexual reproduction as well as an indicator or the overwhelming biodiversity in nature. i

6

7 Acknowledgement Without the support o many people I would not have been able to write this thesis. I would like to use the opportunity to express my gratitude. First o all I want to thank my supervisor Anton Bovier: Thank you or always taking time or me and the continuous encouragement when I miss the orest or the trees. I beneited a lot rom our discussions, your enormous mathematical knowledge and especially your excellent intuition which impresses me time ater time. You support me in many ways, not only mathematically and on your pannier reck. Thank you or being my "Doktorvater". The second person who took an important role during my PhD studies is Loren Coquille: It was a pleasure or me to work together with you at so many dierent places. Thank you or being such a good riend and or being my witness and or welcoming me in Grenoble. Special thanks go to Nikolaus Schweizer or being a riend and a mentor and to Raphael Zimmer the best room mate I can imagine. It was a pleasure to be a member o the stochastic group and I thank all ormer and current colleagues or creating a riendly atmosphere, especially our kind soul Mei-Ling. I want to express my gratitude to my husband Henrik and my amily. Thank you or your patience, understanding and encouragement. I am grateul or inancial support rom the German Research Foundation DFG through the Hausdor Center or Mathematics, the Cluster o Excellence ImmunoSensation, and the Priority Programme SPP1590 Probabilistic Structures in Evolution. It was a pleasure to participate in those programs and the Bonn International Graduate School in Mathematics BIGS. Finally, I thank the members o my examination committee. iii

8

9 Contents I. Introduction 1 1. Biological background and motivation Genetic background The contribution o Mendel to evolution Sexual reproduction - a biological puzzle Modelling o sexual reproduction The evolutionary process Population genetics Adaptive dynamics Adaptive dynamics and sexual reproduction The stochastic individual based model o adaptive dynamics Mendelian diploid model The Mendelian diploid model on base Outline o Chapter II and III Outline: Survival o a recessive allele in a Mendelian diploid model Outline: The recovery o a recessive allele in a Mendelian diploid model II. Chapter: Survival o a recessive allele in a Mendelian diploid model Introduction Model setup and goals Introduction o the model Goal Assumptions on the model Main theorems Outline o the proos Heuristics leading to the main theorems Organisation o the proos The deterministic system The large population approximation Proos o Theorem II.4 and the main theorems Analysis o the deterministic system Proo o the main theorems in Section Phase 1: Fixation o the mutant population Phase 2: Invasion o the mutant Phase 3: Survival o the recessive a allele v

10 Contents III.Chapter: The recovery o a recessive allele in a Mendelian diploid model Introduction The stochastic model Previous works Goal o the paper Model setup Birth rates Death rates Large population limit Initial condition Results Linear stability analysis Heuristics o the proo Proo Preliminaries Phase 1: Perturbation o the 3-system aa, aa, AA until AB reaches Θ Phase 2: Perturbation o the 3-system AA, AB, BB until n aa = Θn AA Phase 3: Exponential growth o aa until co-equilibrium with BB Phase 4: Convergence to p ab = n a, 0, 0, 0, 0, n B Discussion Bibliography 125 vi

11 List o Figures I.1. Let: Law o independent segregation grey colour is dominant, white colour is recessive. Right: Law o independent assortment grey colour and long tail are dominant, white colour and short tail are recessive I.2. Twoold cost o sex: asexual reproduction let, sexual reproduction right 6 I.3. Levene s two habitat sot-selection model I.4. Birth rate o an aa individual in Case 2 let and Case 3 right I.5. Evolution o the model rom a resident aa population at equilibrium with a small amount o mutant aa, and when the alleles a and A are co-dominant let or when the mutant phenotype A is dominant right I.6. time vs. individuals plot: deterministic model let stochastic model right. 26 I.7. 6-population ixed point: η = I.8. Model without reproduction between a and B: c ab = 0.1 and η = 0 let, c ab = 0.1 and η = 0.6 right I.9. All-with-all model: c ab = 0 and η = 0 top, c ab = 0.1 and η = 0 center, c ab = 0.9 and η = 0.4 bottom II.1. Our Model: A ittest type and dominant II.2. Collet et al. Model: a and A co-dominant II.3. The three phases o the proo II.4. Steps o the proo II.5. The curves Γ 0, Γ and the tube V in the perturbed vector ield X III.1. Evolution o the model rom a resident aa population at equilibrium with a small amount o mutant aa, and when the alleles a and A are co-dominant let or when the mutant phenotype A is dominant right III.2. Simulation o the stochastic system or = 6, D = 0.7, = 0.1, c = 1, η = 0.02, ε = and K = III.3. General qualitative behaviour o {n i t, i G} and projection o the dynamical system on the coordinates aa, AA and BB. The re-invasion o the aa population happens sooner and sooner as η grows η = 0.02 or both pictures III.4. Numerical solution o the deterministic system or η = 0.02, logplot III.5. Log-plots o {n i t, i G} or η = 0 top, η = center and η = bottom III.6. Zoom-in when aa recovers, general qualitative behaviour o {n i t, i G} lhs and log-plot rhs vii

12 List o Figures III.7. Flow o the dynamical system in the center maniold o the ixed point p ab, or η = 0.02 let and η = 0.6 right III.8. Numerical solution o the deterministic process, loglogplot or η = 0 and c ab = III.9. Numerical solution o the deterministic process, loglog-plot let or η = 0.01 and c ab = 0.1, right or η = 0.01 and c ab = III.10.Numerical solution o the deterministic process, loglog-plot let or η = 0.56 and c ab = 0, right or η = 0.17 and c ab = III.11.Numerical solution o the all-with-all deterministic process, loglog-plot let or = 6, η = 0 and c ab = 0, right or = 6, η = 0 and c ab = III.12.Numerical solution o the all-with-all deterministic process, loglog-plot let or η = 0.02 and c ab = 0, right or η = 0, c ab = 0 and = III.13.Numerical solution o the all-with-all deterministic process, loglog-plot or η = 0.17 and c ab = let, rescaled individual plot or η = 0.17 and c ab = let viii

13 I. Introduction In nature, sexual reproduction is the avoured kind o propagation in higher plants and animals. Most organisms belong to diploid populations which reproduce sexually. However, a comparison o the reproduction types shows that asexual reproduction is more eective or the survival o a population and less costly as the energy-sapping and ineicient partnerdependent sexual reproduction. Up to now, many more or less convincing hypotheses exist why sexual reproduction nevertheless evolved and is maintained in nature and evolutionary advantageous. Yet or a universally valid explanation which is able to outweigh the many costs o sex, biologists are still searching. In this context, the mathematical endeavour is to construct models which mirror the real lie processes or the actual acts in demand as well as possible to get some interesting conclusions on how evolution acts. The unpredictability and complexity o the biological world turns this into a big challenge. The task is to reduce the high amount o natural acting mechanisms and phenomena to the essential ones such that it is possible to build up a mathematical tractable model which gives relevant and realistic implications or the biological questions in demand. For this new methods had to and have to be developed. Such models o populations give insight into the evolutionary population process and an understanding o the acting mechanisms and give biologists a tool to ind and to prove new hypothesis. In this thesis, we use mathematical modelling to show that sexual reproduction enables competitive populations to react and adapt better to changes in the ecological system. To be more precisely, we consider a stochastic population model consisting o diploid, hermaphroditic individuals which reproduce sexually according to Mendelian laws and prove that a subpopulation, carrying a recessive allele, which is orced to extinction by the whole population can resurrect when the genetic composition o the population changes. The study o Mendelian diploid models started at the beginning o 1900 by Yule, Fisher, Wright and Haldane, the ounding athers o population genetics. But the models in this context typically deal with inite population sizes and models inheritance at the expense o competitive interaction o populations among themselves and their environment. In our approach, we take these actors into account and consider a competitive population model, described by a stochastic system o interacting individuals, which include the genetic results o reproduction, as well as ecological dynamics. Since individuals are competing or resources, the environment o an individual it lives in and interacts with, is deined by the composition o the population. This kind o model belongs to the theory o adaptive dynamics, developed in the 90 ies, which is a variant o population genetics and models the phenotypic evolution o a population in a varying environment. It allows variable population sizes controlled by the competitive interaction o the population. However, most o 1

14 I. Introduction the results in the context o adaptive dynamics are based on haploid asexual reproducing population since, unortunately, especially the Mendelian reproduction and the interaction o genotypically and phenotypically dierent individuals makes the modelling o sexual reproducing populations quite complicated. Thereore, the consideration o stochastic models on the individual level in a genetic setting just started in In this thesis we pick up this line o research and show that in a Mendelian diploid model, under a dominance-recessivity assumption, there can appear a hitchhiking phenomenon o recessive alleles in heterozygous individuals which leads to their prolonged survival in the population and, moreover, ater changes in the population composition and under airly natural assumptions, to genetic variability as well as biodiversity. The remainder o the introduction is organised as ollows. First a simpliied overview o the genetics needed to understand the biological background is presented ollowed by an introduction o the Mendelian laws. We then discuss some historical acts which lead to the theories o population genetics and adaptive dynamics and review the already existing research o diploid models. The irst section ceases with the many costs o sexual reproduction and theories why it is maintained in nature nonetheless. The second section concerns the mathematical modelling approaches o the evolution o sexually reproducing populations in the context o population genetics and adaptive dynamics. In Section 3 the stochastic individual-based Mendelian diploid model studied in Chapter II and III is introduced. The contributions o this thesis, presented in detail in Chapter II and III, are shortly summarised in the last section o the introduction. In Chapter II we examine the genetic evolution o the three genotypes aa, aa and AA o the Mendelian diploid model, introduced in Section 2 o this introduction, in the large population and rare mutation limit. The main results are, that under a dominant-recessivity assumption on the alleles, the recessive a allele can survive in the population long enough such that under a suitable rescaling o the mutation rate a new advantageous mutation can appear beore its extinction. This chapter is published in the Journal o Mathematical Biology [84]. The content o Chapter III is the preprint [10] available on arxiv. It builds on Chapter II and studies the population in the large population limit, when still enough recessive a alleles are present and a mutation to a more beneicial B allele has appeared. Therein, a six dimensional deterministic system is analytically studied and under rather natural assumptions, the coexistence o the two homozygous subpopulations o genotype aa and BB is proven. Both chapters together show that sexual reproduction gives to populations an advantage because they can better adapt to a changing ecological system. It could be an indication or the overwhelming biodiversity in nature. 1. Biological background and motivation The simplest reason why it is important to study sexual reproduction is certainly its overwhelming occurrence in nature. The vast majority o organisms in nature reproduce this way. In this section we take a closer look at this kind o reproduction rom a biological viewpoint. 2

15 1. Biological background and motivation 1.1. Genetic background We start by introducing the minimum genetics that is needed to understand the basic mechanisms o reproduction and Mendelian inheritance. For urther readings and more details we reer to Lodish [70] and Barton et al. [4]. An organism or individual is built up by one to many cells. Each o these cells has a cell membrane which separate it rom the others and encloses the cytoplasm and other cell organelles. The genetic material, called genome, is contained in the cytoplasm and consist o Deoxyribonucleic acid DNA. In procaryotes cells without nucleus the DNA is located ree in the cytoplasm whereas most o the genome o eukaryotes cells with nucleus is located in its nucleus and is organised linearly in chromosomes whose number is characteristic o each specie. A particular region o the DNA is called a gene. Each chromosome consists o many genes and the position o a gene is called a locus. For example the human genome is composed o 23 chromosomes and genes Human Genome Project, 2001 [65]. The set o genes deines the genotype o the organisms where the variant orms o a given gene are called alleles. Dierent alleles result in dierent phenotypes which are the expressions o the genotype. Consequently, the phenotype o an individual is the sum o all morphological and physiological traits plus behavioural eatures which are determined by the alleles e.g. body size, hair colour, intelligence. Individuals with a single set o chromosomes are called haploid, and with more than two copies polyploid. In this thesis we consider sexually reproducing diploid individuals, meaning organisms whose cells have two homologous copies o each chromosome, usually one o the mother and one o the ather. Thus, or each gene the individual can carry two variant orms. I the alleles o the two genes at a particular locus in a diploid individual are the same it is called homozygous, otherwise heterozygous. An allele is called recessive i it is only expressed in the homozygotes otherwise it is called dominant. As a consequence heterozygotes which consist o one dominant and one recessive alleles always express the dominant character in their phenotypes. I an organism reproduces asexually, like every prokaryote, then the ospring s genome is an exact copy o the individual s one unless mutations alter alleles. This is why it is also termed clonal reproduction. In contrast, during the sexual reproduction the diploid organism irst creates gametes sperm or egg by a mechanism o cell devision, called meiosis, which are haploid cells and contain a single set o the individual chromosomes. During ertilisation two gametes o two dierent individuals use together to a zygote, rom which the ospring develops. The zygote is diploid again and contains one set o chromosomes o each parent. Thereore, the genome is the base or heredity and the basic laws how genes rom the parents are transmitted to the osprings were irst studied by the Austrian monk Gregor Mendel during seven years o hybridisation experiments with peas plants The contribution o Mendel to evolution Gregor Mendel published his work, Versuche über Planzen-Hybride [73], 1866 just seven years ater Charles Darwin s publication o The origin o species [25] where he argued that natural selection would be the driving orce o evolution. Darwin deined natural 3

16 I. Introduction selection as the process o unavoidable selection avouring individuals best adapted to their environment. His work was the starting point o a big, controversially discussion about evolution under biologists see [32]. In contrast, Mendel s results ound less attention and its importance or inheritance was unregarded and orgotten [77]. Only ater his death in January 1884, Mendel s work was rediscovered in 1900 by Hugo de Vries [26], Carl Correns [22] and Erich von Tschermak [94]. The Mendelian laws known today were not originally stated by Mendel but named in his honour: 1. Law o independent segregation During meiosis, the two alleles or each gene segregate rom each other and are distributed to dierent gametes. Thereore, the gametes only contain one o the two alleles or each trait. At ertilisation, two gametes o two dierent individuals use together and deine the genotype o the ospring. Consequently, the ospring is diploid again and receives a pair o alleles or each trait one rom each parent see Figure I.1 let. 2. Law o independent assortment During meiosis, the segregation o the two alleles o one allelic pair is independent o the segregation o the two alleles o another allelic pair. see Figure I.1 right. 3. Law o dominance The phenotypic trait is always deined by the dominant allele in Figure I.1 let the allele "g" and alleles "C" and "T" right. These laws are only presented as hypotheses in Mendel s paper [73]. De Vries was the irst who mentioned the expression "Mendel s law" and it was rigorously deined by Correns [22] in acknowledgment o Mendel s work. But both did not distinguish between dierent laws. Morgan irst explicitly talked about the law o independent segregation and the law o independent assortment [78] and integrated it with the Boveri-Sutton chromosome theory o 1915 in the book "The Mechanism o Mendelian Heredity" [79]. It was Sir Ronald Aylmer Fisher, in 1930, who worked out that Darwin s theory o natural selection does not stand in contradiction with Mendelian inheritance and how the two theories needs to be combined [34, 35]. He considered evolution in a mathematical ramework and is one o the ounding-athers o population genetics Sexual reproduction - a biological puzzle The evolution o sexual reproduction, precisely its origin and its maintenance in a highly competitive world, is still a major problem in biology. Sexual reproduction compared to asexual reproduction seems to be very costly e.g. [23, 69, 93]. For a start, there are the cellular-mechanical costs [68] which correspond to the large amount o cellular mechanisms meiosis, usion o gametes syngamy and o nuclei karyogamy which sexual reproduction requires and which take a long time and energy. Furthermore, during meiosis, genetic recombination can destroy advantageous allele combinations which can result in a decrease o the individual s itness and selection can act again [83]. Also noteworthy 4

17 1. Biological background and motivation Figure I.1.: Let: Law o independent segregation grey colour is dominant, white colour is recessive. Right: Law o independent assortment grey colour and long tail are dominant, white colour and short tail are recessive. are the costs or searching or potential partners, and the possibility o spreading diseases between the partners [86]. Additionally, in sexually reproducing populations there is the amous twoold cost o sex [93], see Figure I.2 or producing males. To make this concrete, in asexually reproducing population there is only one sex, precisely each individual owns the emale unction, i.e. it gives birth to osprings which again own the emale unction. By contrast, in sexually reproducing populations there are both sexes present, emales and males, at the same raction. But only emales give birth to osprings o these 50 percent are males and 50 percent are emales. Thereore, asexual emales invest their ull energy in producing osprings owning the emale unction whereas sexual emales invest 50 percent o their resources in producing males which or their part cannot give birth to osprings. To be more precisely, assuming two osprings per emale by reproduction, an asexual population double the size o individuals owning the emale unction whereas in a sexual population the amount o emales stay constant see Figure I.2. In this thesis, we consider a hermaphrodite sexual population organisms with both sexes, where each individual can undertake the male or the emale unction by reproduction. Nevertheless, also such population grow more slowly compared to asexual ones due to the needed partner or reproduction. To be more precisely, assuming an asexual and a sexual population o the same size and that one reproduction event results in the same amount o osprings in both populations. Then, the asexual one has double the amount o osprings compared to the 5

18 I. Introduction sexual one because therein each individual produces descendants whereas in the sexually reproducing population two individuals are needed or the same amount o descendants. Figure I.2.: Twoold cost o sex: right asexual reproduction let, sexual reproduction Since sexual reproduction is such a costly endeavour one would suggest that it is rare in nature. But quite to the contrary, the vast majority o eukaryotic organisms reproduce sexually and only 0.1% reproduce exclusively asexually [99]. Consequently, there must be advantages or sexual populations which can outweigh the costs. Many models and theories were developed to ensure the evolution and maintenance o sexual reproduction e.g. [6, 13, 87]. We will summarise briely the most popular and respected ones. 1. Fisher-Muller hypothesis [35, 80] Sexual reproduction accelerates the process o ixation o advantageous mutations by combining beneicial mutations which initially appeared in dierent individuals into the same genome. Comparatively, in asexual populations a second beneicial mutation will only ixate i it appears by reproduction o an individual carrying already the irst beneicial mutation. 2. Muller s ratchet [33, 81] This hypothesis states that there is an accumulation o deleterious mutations in small asexual populations. Ignoring backward or compensatory mutations, in each generation there is a probability that the class o individuals with the least number o deleterious mutation will disappear due to mutation at the same time or due to the stop o reproduction. Thus, in uture generations there will be no individuals with ewer deleterious mutations. In this way, the number o mutations in the population increases. Comparatively, in sexual population, due to recombination, these classes o individuals with the least number o deleterious mutations can be recreated. 3. Red-Queen hypothesis [5, 98] This is an environmental hypothesis describing that sexual reproduction enables individuals to adapt and evolve constantly to survive in coevolved interaction with 6

19 2. Modelling o sexual reproduction other organisms in a deterministically changing environment. Thereby, recombination can create genotypes which are better adapted and have a competition advance. In coevolution o established populations, this leads to sustained oscillations between genotype requencies. The best-known application is the host-parasite, coevolution, showing that sexual reproduction implies the ability or hosts to resist parasitic inection. Parasites may thus be the driving orce or the maintenance o sexual reproduction in their host [47, 48, 53]. 4. Mutational deterministic hypothesis [62, 64] Under the assumption that each additional deleterious mutation decreases the itness, sexually reproducing individuals have a short-term advantage because the amount o mutations passed to the osprings has a big variance. Since osprings with a majority number o deleterious mutations have also low itness they will die out and with them a vast number o mutations is eliminated. Under strict assumptions all these theories give reasons or the evolution and maintenance o sexual reproduction but one generally valid theory is not ound so ar. It is reasonable to take a pluralistic approach [50,102] to explain the omnipresence o sexual reproduction in nature. But up to now it remains a big puzzle and urther work and rigorous studies o population models are needed. 2. Modelling o sexual reproduction Our contribution in this direction is to show in a rigorous mathematical ramework advantages o sexual compared to asexual reproduction by incorporating ecological dynamics and the random elements o evolution in a population model with Mendelian reproduction. To be more precise, we show that in such a population, the time until a recessive allele at one locus becomes extinct is prolonged since it survives in the itter heterozygous individuals. I the genetic composition o the population changes, the population o homozygous individuals carrying this recessive allele can recover and can probably live in coexistence with other genetic dierent subpopulations. In this way, sexually reproducing populations can better adapt to changes in the ecological system because recessive alleles, which can be more advantageous in the changed environment, survive in heterozygotes. Under airly natural assumptions, this can also lead to genetic diversity, as we will show The evolutionary process To understand the evolution o a population it is necessary to take a closer look at this process o high complexity. The main acting mechanisms driving the evolutionary process are: heredity - transmits the parental characteristics morphological and physiological properties to the ospring, 7

20 I. Introduction variation - changes the characteristics o an individual e.g. due to mutations, genetic recombinations, selection - acts onto the dierent reproduction and survival ability itness o the individuals, a consequence o competition between individuals. As already said, each individual is characterised by its genotype which is expressed in its phenotype. The dierent phenotypes o a population compete or resources e.g. water, space, ood, or with other species e.g. predator, prey, parasites. This competition process changes the mortality and ertility o each individual and aects the selection process acting onto the viability o the dierent phenotypes. Consequently, the adaptation o a phenotype depends on its environment which in turn is inluenced by the phenotypes o the present population. Thereore, the population dynamics is in interaction with the environment and also inluences the genotypes through mutations in the genome and additionally in sexually reproducing populations through meioses and recombination. Moreover, the environment has an eect on the translation o the genotype into the phenotype e.g during its lietime. This mapping is up to now not completely understood [89]. Consequently, the variable environment o a population plays a crucial role or its evolution and it is unavoidable to include it in a mathematical population model to be as realistic as possible. Unortunately, the whole complexity o the evolutionary process is not mathematically easible and or modelling there are some simpliications needed at the expense o realistic actors. The rest o this section is concerned with approaches in this direction with a ocus on Mendelian diploid populations Population genetics A irst attempt or modelling Mendelian inheritance was done by Fisher, Haldane and Wright who are the ounding athers o the population genetics theory developed in the 1920 s and 1930 s [24, 32]. The theory integrates Mendel s inheritance theory into Darwin s theory o natural selection or modelling evolution. Population genetics does not ocus on individuals but rather deals with the changes o allele requencies in the whole population over time [82]. In this approach, the selection advantage o an individual, called itness, depend on its genotype. It is deined a priori and is equal to the expected number o progeny [82]. Consequently, selection is directly acting on genotypes instead o phenotypes and changes allele requencies in accordance with their dierent itnesses. The interaction o individuals with their environment is mostly omitted. Thereore, in these models the adaptive landscape, irst introduced by Wright in 1932 [104], is ixed and selection drives the population in this itness landscape upwards to a local maximum [103,105,106]. In this way, population genetics reduces the complexity o the real world and gives insight into the complex patterns o genetic variation. It includes observational, experimental and theoretical components [30, 32] and studies the dynamics and statics o evolution: irst arrived by changes in gene requency under selection, latter by indicating actors which maintain the population in an approximate equilibrium [24]. One milestone in this context and the traditional starting point o population genetics is the Hardy-Weinberg-law, published in 1908 by Hardy [49] and Weinberg [101]. It states 8

21 2. Modelling o sexual reproduction that, in the absence o gene requency changing actors mutation, selection, migration, random drit, in an ininite population which has non-overlapping generations and mates randomly, the genetic variation is conserved. To be more precisely, the gene requencies stay constant rom one generation to the next rom the second generation on [24, 32]. With the newly discovered molecular understanding o inheritance and the availability o genomic data in the 60 ies, a new theory based on population genetics to treat evolution, especially on the molecular level, arose. In 1968 Kimura [54,55], independently rom King and Jukes 1969, see [57], developed the neutral theory o molecular evolution which states that most o the variation within and between populations on a molecular level is achieved by genetic drit change in allele requency due to random sampling o neutral mutations which do not aect the itness o the organism such that natural selection is not acting on them. It was extended to nearly neutral mutations which are only slightly deleterious in 1973 by Ohta [85]. The theory is not in contradiction with Darwins theory o natural selection as driving orce or adaption, as Kimura pointed out [54, 82]. It only considers evolution on the molecular level and states that phenotypic evolution is still controlled by natural selection [54 56]. Despite this extensive knowledge about the genome structure and its evolution, achieved by population genetics, the ormation o new species in this context is not well understood [75]. Nevertheless, the genetic aspect o the models in population genetics provides a good tool to give insights or the overwhelming occurrence o sexual reproduction in nature. These models are used, among others, to examine the genetic processes o segregation and recombination, arrived by meiotic crossover, which are only present in sexually reproducing populations and thereore have to be advantageous. For example Kirkpatrick and Jenkins [58] use this theory to show that segregation could be one reason or the maintenance o sex. They argue that in sexually reproducing diploid populations the substitution o an advantageous mutation happen aster since it needs only the beneicial mutation o one allele in individuals whereas in an asexual diploid population it needs this mutation two times in the same lineage the second mutation in a descendant o the particular individual where the irst took place. Further examples like a critical analysis o the contribution o recombination to the maintenance o sex and o Mueller s ratchet as well as the maintenance o sex due to synergistic epistasis the interaction o alleles at dierent loci, can be ound in [24, 42, 63]. For more insights on population dynamics, we reer to [24, 32, 82] Adaptive dynamics One weakness o the models o population genetics is that they mostly neglect the important aspect o ecology or evolution and the coevolution and interaction o the individuals with their environment. Moreover, mostly constant population sizes are assumed but a real biological population should be able to regulate its size dependent on the environment it lives in. At the beginning o the 90ies, the theory o adaptive dynamics was developed e.g. [52, 72, 76]. This theory is a stochastic approach and studies the eects o ecological 9

22 I. Introduction aspects o populations on evolution and mostly omit the genetics. It highlights the coevolution o the environment with the population due to ecological interaction by a densitydependent selection, that models the survival and reproduction ability o one individual in relation to the whole population. Since genetics is omitted the only source o variation are mutations acting directly on the phenotypes. Only phenotypic evolution is considered whereas the genotypic background o the individuals is ignored. The main dierence to earlier approaches o population genetics is that the adaptive landscape is replaced by an invasion itness landscape which is not ixed anymore and measures the selective advantage o an appearing mutant depending on the environment. Starting points or the theory o adaptive dynamics are mostly asexually reproducing populations consisting in the same phenotypic trait. These populations are called resident, monomorphic population. Furthermore, it is assumed that the population size is large and mutations are rare [29, 75] resulting in a separation o the evolutionary timescale rom the ecological one in such a way that the phase o competitive interaction is small compared to the evolutionary one driven by mutations. This allows to deine the invasion itness o an appearing mutant which is deined by the initial growth o the mutant under the environmental conditions set by the resident populations. Invasion itness is related to the probability o whether the mutant gets extinct or can ixate, i.e. grows rom one individual to a notable size, in the resident population [75, 76]. Thus compared to the ixed itness landscape, the invasion itness landscape o adaptive dynamics coevolves with the population since it deines the environment and describes the evolution by successive mutation invasions [74,75,100]. Since the trait space is assumed to be continuous these models have the advantage to represent the whole evolutionary process compared to earlier modelling approaches [3]. Thus each new mutation has never appeared beore and evolution does not have to stop at the asymptotic evolutionary state [28, 39, 59]. The strength o this approach is that the techniques can be used to understand evolutionary phenomena in various ecological system especially the possibility o phenotypic diversiication. This interesting phenomenon o evolutionary branching where a phenotypic trait splits into two coexisting lineages which evolve in dierent directions was irst investigated by Metz et al. in [75] and urther studied in [28,39,40]. For example, in [75] and [39] it is shown that the occurrence o evolutionary branching depends on the derivative o the itness unction at so called evolutionary singularities. However, outside a neighbourhood o such an evolutionary singularity and under the additional assumption that the invasion o a mutant implies the extinction o the resident phenotype the population stays monomorphic over time. This major concept is the trait substitution sequence TSS modelling evolution as a continuous time Markov process jumping rom one monomorphic population to another according to higher itness. It was introduced by Metz et al. [76] see also [28, 75] and mathematically studied in [14,15,17,18]. It can be extended to the polymorphic case, polymorphic evolution sequence PSE, where the mutant invades a polymorphic population at the expense o one or more resident populations [28, 75]. The canonical equation o adaptive dynamics CEAD, introduced by Dieckmann and Law [28], is a deterministic approximation o the monomorphic TSS, under the assumption that the dierence between the mutant and resident phenotypes is small. It models evolution as a gradual process due 10

23 2. Modelling o sexual reproduction to small phenotypic changes. In other words it is a sequence o successully established mutations. Adaptive dynamics and sexual reproduction The theory o adaptive dynamics deals with the phenotypic long-term evolution o an initial haploid and asexual monomorphic population. The techniques can be applied to many interesting evolutionary issues like virulence, seed size and cyclic evolution, parasite coexistence and predator-prey systems [41, 61, 71, 91]. The phenomenon o evolutionary branching is one o the most important results o these techniques since in biological implications it can be interpreted in the asexual case as morphological speciation and as a driving orce or the biodiversity. But as mentioned above most individuals in nature reproduce sexually and, accordingly, the most interesting phenomenon is the speciation in sexual populations, where one population splits into two dierent lineages which are reproductively isolated. Consequently, to get a realistic insight into evolution, including ecological dynamics and genetics, it is necessary to apply the techniques o adaptive dynamics to sexually reproducing populations. I one wants to adapt the concept o evolutionary branching to sexually reproducing populations one is conronted with several problems. The irst is to model the reproduction event. It is now not only a cloning anymore but rather depends on mating success and ecundity o two individuals. Moreover, it is subject to the Mendelian rules and mutations change alleles. Consequently, the genotype o an individual is to take into account and a rule or mapping the genotype to the phenotype has to be assumed since selection acts on phenotypes. Recall that in an asexual model, the genotype is identiied with the phenotype and is thus omitted in the evolutionary process. This way, mutations alter the phenotype instead o the genotype. One solution to model the long-term evolution is to go back and consider evolution in the allele space rather than in the phenotypic trait space [27, 38, 60, 95]. But even i we get the branching phenomenon in the allele space, when translating back to the phenotype level we have to deal with the presence o the heterozygotes. Thus, or speciation there are additional assumptions needed such as assortative mating i.e. like individuals mate more preerentially [27, 38] or spatial segregation, called allopatric speciation [60]. The consideration o Mendelian diploid models just started in 1999 with a paper o Kisdi and Geritz [59] ollowed by a number o works [38, 60, 90, 95 97]. All this papers consider special models with Mendelian reproduction on a heuristic level. Most o them, [38, 60, 95, 97], as also the one o Kisdi and Geritz [59], use a continuous diploid version o Levene s sot selection model [67], see Figure I.3 because o its relative simplicity and its well-known population genetics. A haploid clonal counterpart o this model was already studied in 1998 by Kisdi and Geritz [39] which makes a comparison o this model with the Mendelian diploid model versions possible. In [59], the authors consider one locus with a continuum o possible alleles and an environment which consists o two habitats, 1 and 2, each having an optimal phenotype, m 1 and m 2. They assume that the population size is constant in each generation and that the habitats are o relative size c 1 = c and c 2 = 1 c see Figure I.3. In these ecological settings they model evolution o the population in allele space under requency-dependent 11

24 I. Introduction selection. They assume rare mutations with small phenotypic eects, such that there is enough time or a mutant to invade or extinct and or the population to reach its genetic equilibrium beore a urther mutation occurs. In this way and since a continuum o possible alleles is assumed in contrast to earlier aporaches, long-term evolution is studied as a sequence o mutation invasions. In the model, all individuals have the same ecundity and discrete, non-overlapping generations. At the start o each generation, individuals are distributed randomly in the two habitats. There is irst a phase o viability selection within each habitat, where the survival probability o an individual depends on its phenotype, x, and is given by the Gaussian unction, i x = α i exp x m i 2, I.1 2σ 2 or i {1, 2}, with variance σ 2 and α i the maximal survival probability. Then ollows a second phase o nonselective contest competition during which the available living space is allocated at random among the survivors. In this way, a ixed number o individuals is recruited in each habitat a raction c in the irst habitat und the remaining raction 1 c in the second habitat, sot selection, see Figure I.3. These individuals orm a population where mating occurs randomly and osprings are produced accordingly to Mendelian rules at the end o each generation. It is assumed that alleles, x and y, act additively on the phenotype, precisely, the heterozygote is exactly in between the two homozygotes the phenotype o an xy individual is given by x+y. The invasion itness o a rare mutant allele 2 y in a resident monomorphic population o allele x is given by S x y = c x+y c x+y x 2 x. I.2 Kisdi and Geritz [59] show that in the diploid model evolutionary branching in a monomorphic population occurs under exactly the same ecological circumstances as in the haploid model but the urther evolution is completely dierent. In the diploid sexual model, resp. haploid clonal model, the invasion o a mutant allele, resp. trait, implies substitution o the resident allele, resp. trait, as long as the population is away rom an evolutionary singular point. I the haploid clonal population reaches an evolutionary singular point it converges to an evolutionary stable dimorphism. Thus there is only one evolutionary outcome. On the contrary, in the diploid sexual model there are up to three dierent possible outcomes depending on the dierence between the optimal phenotypes in the habitats and the habitats sizes: i a single evolutionary stable which cannot be invaded by any urther mutant allele genetic dimorphism both homozygotes are habitat specialists, ii two convergence and evolutionary stable genetic dimorphism at each one the heterozygote and one o the homozygote are habitat specialists, 12

25 2. Modelling o sexual reproduction Figure I.3.: Levene s two habitat sot-selection model iii three convergence and evolutionary stable genetic dimorphism the ones o i and ii. In [59], Kisdi and Geritz also discuss the probability or reaching these polymorphisms. However, observe that this branching phenomenon is completely dierent rom the asexual case since we achieve only genetic variability in the allele space which does not leads automatically to speciation. To be more precisely, we end up in a protected genetic polymorphism but do not automatically get phenotypically distinct lineages. In their ollowing work [38], Kisdi and Geritz show that there could be indeed sympatric speciation speciation in the same geographic region. To be more precisely, they consider the polymorphic population ater the branching and show that there would be evolution o assortative mating and hence partial reproductive isolation. For this, they extend the single locus model by a second locus with two possible alleles one dominant over the other which decodes the mating group the individual belongs to and does not aect the individual itness. Individuals mate with probability p within its mating group and with 1 p mating is random. In [38], it is discussed that the irst locus which undergoes branching and the second controlling mating have to be in linkage disequilibrium the nonrandom association o the alleles at the two dierent loci. The development o such a linkage disequilibrium is only possible i the dierence between the two habitats is large enough, such that we get the possible evolutionary outcomes ii and iii o the one locus model, and selection against heterozygotes is strong. Then there is evolution o assortative mating and there always exists only one evolutionary outcome where the two homozygotes are habitat specialists as in the clonal model. Nevertheless, the evolution o assortative mating is not the only possibility or speciation as van Dooren [95] shows. For a similar version o Levene s sot selection model, as studied by Kisdi and Geritz [38,59], he demonstrates that the evolution o dominance can lead to elimination o the phenotypic intermediate heterozygote. In [97], van Doorn and Dieckmann study the long-term evolution in a multi-locus version 13

26 I. Introduction o Levene s sot selection model under the ecological settings in [59]. By simulations they get that requency-dependent selection does not maintained genetic polymorphism at a large number o loci. Only one locus with large phenotypic eect retained genetic variation The stochastic individual based model o adaptive dynamics Starting in the mid-90ies, stochastic individual based models were introduced and investigated that allow or a rigorous derivation o many o the predictions o adaptive dynamics on the basis o convincing models or populations o interacting individuals that incorporate the canonical genetic mechanisms o birth, death, mutation, and competition see, e.g., [14 17,28,36] and [3]. To be more precise, in these models inite populations o nonconstant size are considered consisting o individuals each owning a birth rate, a natural death rate and an additional death rate due to competition with the other present individuals. Moreover, there is a probability or mutation at each birth event. The population dynamics in these approaches are density-dependent which keeps the population rom exploding and allows or variable population size. The aim o these models, irst introduced by [8] and [66], is to study the macroscopic phenotypic long-term evolution o a population by tracing it back to the microscopic individual level [14]. In the context o stochastic individual-based models, so ar, the biological approach o adaptive dynamics has been put on a rigorous mathematical ooting almost exclusively or haploid and asexually reproducing populations. In this ramework an important and interesting eature o these models is that the TSS, PES and the CEAD appear as limiting processes on dierent time-scales as the population size tends to ininity while mutation rates and mutation step-sizes tend to zero. In [14], Champagnat proves convergence to the trait substitution sequence TSS in the simultaneous limit o large population and small mutation and in [15], Champagnat, Ferrière and Ben Arous show that this process converges in the limit o small mutation steps to the canonical equation o adaptive dynamics CEAD. Recently, Baar, Bovier, and Champagnat [2] prove the convergence to the CEAD in the simultaneously combined limits o large population, rare mutations and small mutation steps. The phenomenon o evolutionary branching was rigorously derived by Champagnat and Méléard. In [17], they obtained the convergence to the polymorphic evolution sequence PES, where jumps occur between equilibria that may include populations that have multiple coexisting phenotypes. Mendelian diploid model In the context o individual-based models, the study o diploid sexual reproducing population started recently. The irst work in this direction is the one o Collet, Méléard and Metz [18]. Therein they consider a Mendelian diploid model, a single locus model o a inite, diploid population with sexual reproduction ollowing the Mendelian rules, under the assumption that alleles act additively on the phenotype, they are co-dominant. For this 14

27 2. Modelling o sexual reproduction model they derived, in the limit o large population and rare mutations, the convergence o the suitably time-rescaled process to the TSS model o adaptive dynamics, essentially as shown by Champagnat [14] in the haploid case. This paper was the starting point ollowed by a series o works by Coron [19,20] and Coron, Méléard, Porcher, and Robert [21] which all consider single locus models. In the irst work inite populations are considered. Coron studies the ixation probability o one allele in the absence o mutations in the neutral case all individuals have the same birth, natural death and competition death rates and in the non-neutral case the death rate o one allele slightly deviate rom the neutral case. She derived the TSS on the genotype space or the successive ixations o mutations by adding rare mutations and rescaling properly the time. Moreover, i all mutations are deleterious, in [19] it is shown that, ater each ixation o a deleterious mutation the natural death rate o the individuals increases. In this way, the existence o an extinction vortex is shown where the increasing requency o ixation o deleterious mutations is observed and the extinction o the population is inevitable. This mutational meltdown or a model o a inite population similar to the one o [19] is numerically studied in [21]. Therein, the impact o deleterious mutation accumulations on the population size is analysed. Furthermore, the dependence o the strength o the mutational meltdown on demographic parameters is considered. In the last cited work on a Mendelian diploid model [20], Coron proves the convergence to a slow-ast stochastic diusion dynamic in the large population limit where mutations are accelerated as a consequence o acceleration o the birth and death events and under a ine-tuning o competition. In a second step she shows that this diusion, conditioned on non-extinction, permits a unique quasi-stationary distribution. Finally, the long-time coexistence o two alleles in three cases pure neutral competition, over-dominance and separate niches are numerically studied. The results so ar do not give indications or genetic variation and speciation such that one can suggest an advantage o sexual reproduction in adaptive dynamics except under some special ine-tuning o parameters. As in the haploid case, also in the diploid case [18], normally the time or ixation o mutant traits in a monomorphic resident population is o the same order as the time or extinction o the residual trait. Thus as soon as a new mutation appears the resident population is already monomorphic again. One exception in the ramework o individual based models is the work o Dieckmann and Doebli [27]. They show that in a multi locus model with only two possible alleles per locus + and speciation is possible. They assume that mating probabilities o individuals are expressed in an additional quantitative trait depending on the ecological trait or a marker trait which is ecologically neutral. In the irst case they show that evolutionary branching happens under slightly more restrictive conditions as in the corresponding clonal model. However, in the latter case, the development o a linkage disequilibrium between the ecological and the marker trait is needed which can be obtained by genetic drit due to stochastic demographics eects. But in this case the parameter requirements or evolutionary branching are more restrictive than those in the asexual case. However, all these results based only on simulations. In this thesis, we discuss the dramatic impact on genetic evolution in a Mendelian diploid single locus model i we replace the assumption o co-dominance in [18] by assuming that 15

28 I. Introduction the mutant allele is dominant. We will see that the time or extinction o the residual recessive allele is extremely enlarged such that a urther more advantageous mutation can appear which paves the way or the appearance o a richer limiting process. 3. The Mendelian diploid model on base In this section we present the Mendelian diploid model, which is studied in this thesis. It is an extended version o the single locus model introduced in [18] and includes mating groups. At any time t 0, the population under consideration consists o a inite number o individuals N t on which the three basic mechanisms o evolution, Mendelian heredity, mutation and selection, are acting. The genotype o each individual i is determined by two alleles u i 1u i 2 at a single locus, taken rom some allele space U R. We suppress parental eects, which means that we identiy individuals with genotypes u 1 u 2 and u 2 u 1. Each individual can act as ather or mother, i.e. it is hermaphroditic. As explained above the genotype deines the phenotype. In Chapters II and III we make a complete dominantrecessivity assumption on the phenotype, i.e. the dominant allele deines the phenotype. Heterozygous individuals thus exhibit the phenotype o the dominant allele which is the same as the phenotype o the individuals which are homozygous or this allele. We construct a Markov process modelling the Mendelian reproduction and the death o each individual without any assumption on the genotype-phenotype mapping. To this end, we introduce the ollowing parameters where we omit the dependence on the phenotype to shorten the notation: u1 u 2 R + the per capita birth rate ertility o an individual with genotype u 1 u 2, D u1 u 2 R + the per capita natural death rate o an individual with genotype u 1 u 2, K N the carrying capacity, a parameter which scales the population size, c u1 u 2,v 1 v 2 R K + the competition eect elt by an individual with genotype u 1 u 2 rom an individual o genotype v 1 v 2, R u1 u 2 v 1 v 2 {0, 1} the reproductive compatibility o the genotype v 1 v 2 with genotype u 1 u 2, µ K R + the mutation probability per birth event. It is independent o the genotype, mu, dh the mutation law o a mutant allelic trait u + h U, born rom an individual with allelic trait u. Individuals are living in an environment which provides them an amount o resources, called the carrying capacity o the environment and denoted by the scaling parameter K N. Thus the competition is rescaled by 1 which amounts to scaling the population K 16

29 3. The Mendelian diploid model on base size to order K. This way the population size is controlled by the amount o resources the environment oers and we count individuals weighted with 1. We are interested in K asymptotic results when K is large and mutations are rare. We denote by u 1 1tu 1 2t,..., u Nt 1 tu Nt 2 t the genotypes present at time t and the whole population, νt K, at time t is represented by the rescaled sum o Dirac measures on U 2, ν K t = 1 K N t i=1 δ u i 1 tu i 2 t. I.3 Formally, ν K t takes values in the set o re-scaled point measures { } M K 1 n = δ K u i 1 u i n 0, u 1 2 1u 1 2,..., u n 1u n 2 U 2, I.4 i=1 on U 2, equipped with the vague topology. Deine ν, g as the integral o the measurable unction g : U 2 R with respect to the measure ν M K. Then νt K, 1 = Nt = n t and or any u 1 u 2 U 2 the positive number ν t, 1 u1 u 2 is called the density at time t o the genotype u 1 u 2. The dynamics o the process are as ollows see [36], [18]: We start at time t = 0 with a possibly random distribution ν 0 M K. Each individual with genotype u 1 u 2 has three independent exponential a reproduction without mutation Exp u1 u 2 1 µ-clock: When it rings then the individual chooses at random an individual v 1 v 2 as partner and reproduces with it at rate v1 v 2 R u1 u 2 v 1 v 2 u1 u 1 K νr u1. The ospring s genotype is a pair o two alleles, each one u 2, chosen randomly o each parent. Š@10 a reproduction with mutation Exp u1 u 2 µ-clock: When it rings then the individual chooses at random an individual v 1 v 2 as partner and reproduces with it at the same rate as beore but the ospring gets a genotype where one o the parental alleles changes rom u to u + h with h chosen randomly according to the mutation law mu, dh. Since we assume rare mutations, i.e. µ K 1, only one parental allele a death ExpD u1 u K Nt j=1 c u 1 u 2,u j When it rings then the individual 1 2-clock: dies. Observe that this parameter depends uj on the natural death rate and on an additional death rate due to ecological competition with the other individuals present in the population. Once one o the clocks rings all clocks are reset to zero. Let us now construct the generator o this process ν K t t 0. As in [18] we irst deine, K 17

30 I. Introduction or the genotypes u 1 u 2, v 1 v 2 and a point measure ν, the Mendelian reproduction operator A u1 u 2,v 1 v 2 F ν = 1 [ F ν + δ u 1 v 1 4 K R + F ν + δ u 1 v 2 + F ν + δ u 2 v 1 + F ν + δ ] u 2 v 2 F ν, K K K I.5 and the Mendelian reproduction-cum-mutation operator M u1 u 2,v 1 v 2 F ν = 1 [ F ν + δ u 1 +h,v 1 8 K + F ν + δ u 1 +h,v 2 K mu 1, h + F ν + δ u 2 +h,v 1 + F ν + δ u 2 +h,v 2 mu 2, h K K + F ν + δ u 1,v 1 +h + F ν + δ u 2,v 1 +h mv 1, h K K + F ν + δ u 1,v 2 +h + F ν + δ ] u 2,v 2 +h mv 2, h dh F ν. K K I.6 The process ν K t t 0 is then a M K -valued Markov process with generator L K, given or any bounded measurable unction F : M K R by: L K F ν = D u1 u 2 + c u1 u 2,v 1 v 2 νdv 1 v 2 F ν δ u 1 u 2 F ν Kνdu 1 u 2 U 2 U K 2 v1v2r u1u2v 1 v µ K u1 u 2 A u1 u U 2 U νr 2 u1 u 2, 2,v 1 v 2 F ννdv 1 v 2 Kνdu 1 u 2 v1v2r u1u2v 1 v 2 + µ K u1 u 2 M u1 u U 2 U νr 2 u1 u 2, 2,v 1 v 2 F ννdv 1 v 2 Kνdu 1 u 2. I.7 The irst non-linear term is density-dependent and describes the competition between individuals. It makes selection requency-dependent, i.e. the itness o an individual depends on the requencies o the dierent individuals present in the population. The second and last non-linear terms describe birth with and without mutation. Note that R u1 u 2 v 1 v 2 can be interpreted as a mating ability and models whether an individual with genotype u 1 u 2 can reproduce with an individual with genotype v 1 v 2. Thus νr u1 u 2 is the population restricted to the pool o potential partners o an individual o genotype u 1 u 2. For ixed K and all u 1 u 2, v 1 v 2 U 2, under the ollowing assumptions E ν K 0, <, 18

31 3. The Mendelian diploid model on base The unctions, D and c are measurable and bounded, which means that there exists, D, c < such that 0 u1 u 2, 0 D u1 u 2 D and 0 c u1 u 2,v 1 v 2 c, I.8 There exists a unction, m : R R +, such that mhdh < and mu, h mh or any u U and h R, the existence and uniqueness in law o such a process, in the space DR +, M K o càdlàg unctions rom R + to M K, with ininitesimal generator L K can be derived rom Fournier and Méléard [36]. The construction o the process given in [36] as solution o a stochastic dierential equation driven by Poisson point measures describing each jump can be adapted to our settings. We are interested in the large population limit. In this case, under mild restrictive assumptions, the process converges to a deterministic process, which is the solution to a non-linear integro-dierential equation. The proo o this can be deduced rom Fournier and Méléard [36] as well. We state here only the case where the mutation rate is zero which can be interpreted as the short time evolution o an initial population. Theorem I.1 Theorem 3.1 in [18]. When K tends to ininity in law and i ν0 K converges in law to a deterministic measure ν 0, then, or any measurable, symmetric unction g : U 2 R, the process νt K converges in law to the deterministic continuous measure-valued unction ν t t 0 solving t ν t, g = ν 0, g ν s, gu 1 u 2 D u1 u N s c 0 K u1 u 2,v j ds 1 vj 2 j=1 t + ν s ν s, u 1 u 2 v1 v 2 4 ν s, gu 1v 1 + gu 1 v 2 + gu 2 v 1 + gu 2 v 2 ds. 0 I.9 We take a closer look at the three cases o initial populations which are o greatest interest or this thesis: 1 the one allele case where U = {A} and hence all individuals are homozygotes with genotype AA, 2 the two allele case where U = {a, A} and the population consists o the three genotypes aa, aa and AA, 3 the three allele case where U = {a, A, B} and the population consists o the six genotypes aa, aa, AA, ab, AB and BB. Already on these three simplest cases the problems o modelling a sexual diploid population become clear. An allele space U o n alleles provides a genotype space G o 19

32 I. Introduction n k=1 k = nn+1 2 genotypes. One new appearing allele, due to mutation, in the allele space U blows up the genotype space G by n + 1 genotypes through the inevitable occurrence o heterozygotes, whereas in a haploid clonal population it increases only by one. Moreover, the birth rates o the dierent genotypes are completely dierent and rather more complicated compared to the clonal case. In a sexual population the birth rate o an individual carrying a certain genotype depends on the whole state o the population or o several subpopulations and not only on the population carrying this genotype as in the clonal case. In the ollowing we make this more precise in the three special cases. 1 In the irst case let us assume that the initial population consists o n K 0 δ AA individuals with n K 0 n 0, or K. By Theorem I.1 the process n t t 0 converges in law, when K, to the solution o the classical logistic equation ṅt = nt AA D AA c AA,AA nt and n0 = n 0. I.10 The unique stable ixed point o this equation is the equilibrium size o a monomorphic AA population and is equal to the carrying capacity: n AA = AA D AA c AA,AA. I.11 The birth rates b AA nt, can be derived by computing the reproduction rates with the Mendelian rules as described in III.5. In this case it is simply b AA nt = AA n AA t and comparable to birth rates in the clonal case. 2+3 In the last two cases compare Proposition 3.2 in [18] let us assume that the initial condition n K 0 = n aa 0, n aa 0, n AA 0, resp. n K 0 = n aa 0, n aa 0, n AA 0, n ab 0, n AB 0, n BB 0 converges to a deterministic vector x 0, y 0, z 0, resp. u 0, v 0, w 0, x 0, y 0, z 0, or K. Then the process n t t 0 converges in law, or K, to the solution o with ṅt = bnt dnt, ṅ i t = b i nt n i t D i + i G c i,j n j t I.12, i G. I.13 The calculation o the birth rates b i nt, i G, in these cases becomes quite complicated. We will illustrate it by computing the birth rate o an aa individual. In the thesis we consider the case where the ertility is neutral, that means that i =, or all i G, and that the B allele is the most dominant and the a allele the least dominant one. Consequently, the ascending order o dominance is given by a < A < B. Furthermore, we make the dominant-recessivity assumption, that the most dominant allele deines the phenotype and consider the case where the a phenotype consequently only expressed by individuals o genotype aa is not 20

33 3. The Mendelian diploid model on base capable o reproducing with individuals o B phenotype which are the ones with genotypes ab, AB and BB. In mathematical terms, 1, or i, j {aa, aa, AA}, 1, or i, j {aa, AA, ab, AB, BB}, R i j = 0, or i = aa and j {ab, AB, BB}, 0, or i {ab, AB, BB} and j = aa. I.14 In case 2 possible matings which results in an aa individual are see Figure I.4 let: aa aa, with probability 1, aa aa, with probability 1 2, aa aa, with probability 1 4. Since all individuals can reproduce with each other in this case, the whole population acts as pool o possible partners when an individual chooses a partner. In this way the birth rate o an individual o genotype aa is given by: b aa nt = n aat n aat 2 n aa t + n aa t + n AA t. I.15 In case 3 we have to bear in mind that aa individuals are not capable o reproducing with individuals o phenotype B. Consequently, the possible matings which result in an aa individual in this case are see Figure I.4 right: aa aa, with probability 1, aa aa, with probability 1 2, aa aa, with probability 1 4, aa ab, with probability 1 4, ab ab, with probability 1 4. The pool o possible partners or an aa individual consists, as in the case beore, o all individuals with genotypes aa, aa, and AA. In contrast, the whole population acts as pool o possible partners or phenotypic A individuals and or phenotypic B individuals the pool consist only o aa, AA, ab, AB and BB individuals. Thereore, in this case the birth rate o an individual o genotype aa is given by: b aa nt = n aat n aa t + 1n 2 aat n aa t + n aa t + n AA t n aat n aa t + 1n 2 aat + 1n 2 abt n aa t + n aa t + n AA t + n ab t + n AB t + n BB t n abt 1 n 2 aat + 1n 2 abt n aa t + n AA t + n ab t + n AB t + n BB t. I.16 21

34 I. Introduction aa 1 aa 1 2 AA aa aa aa 1 aa aa ab aa aa AB aa AB aa ab ab AA AA AA BB BB Figure I.4.: Birth rate o an aa individual in Case 2 let and Case 3 right. One important result see [18, 36] is that on inite time intervals the behaviour o a large population can be approximated by the solution o the deterministic system I.12. Consequently, the knowledge about the deterministic system is a good starting point or understanding the behaviour o the stochastic system. However, this is not as easy as it seems since the result holds only on inite time intervals and, as we will see in Chapter II, we have to control the behaviour o a stochastic process over a time horizon that diverges like a power o K. This precludes, in particular, the use o unctional laws o large numbers, or the like which justiies that the stochastic system behaves like the deterministic system [31, 36]. Moreover, analysing the behaviour o high dimensional systems like I.12 is a considerable challenge. For the 3-dimensional system this is done by Collet, Méléard and Metz in [18]. In Chapter III a rigorous analytic study o the 6-dimensional deterministic system is provided. 4. Outline o Chapter II and III Chapters II and III present the main results o this thesis. In Chapter II we study the genetic time evolution o the stochastic individual-based model introduced in Section 3 which describes a diploid hermaphroditic population reproducing according to Mendelian laws. The aim is to show that under a complete dominance-recessivity assumption the recessive allele has a prolonged survival time compared to the previous result in [18] under the codominance assumption. Chapter III picks up the result o Chapter II and considers the ate o the recessive allele in the population ater a urther mutation to a more dominant allele in the large population limit. Although both chapters are related they can be read independently rom each other. The ollowing two subsections outline these chapters. Therein we will call an u 1 u 2 population a population containing only individuals with genotype u 1 u 2 which we name u 1 u 2 individuals or more briely u 1 u 2. 22

35 4. Outline o Chapter II and III 4.1. Outline: Survival o a recessive allele in a Mendelian diploid model In the second chapter the evolution o a recessive allele in a Mendelian diploid model under a complete dominance-recessivity assumption is studied. The results appeared as a joint work with Pro. Dr. Anton Bovier in the Journal o Mathematical Biology [84]: R. Neukirch and A. Bovier, Survival o a recessive allele in a Mendelian diploid model, Journal o Mathematical Biology 2016, pp Chapter II contains this article mostly as it has been published, only a straightorward improvement o the lower bound on the survival time o the recessive allele to K 1/2 α and modiications in ormatting are made. We start with a monomorphic population o aa individuals with one additional mutant with genotype aa. That means our allelic trait space consists o the recessive a allele and the dominant A allele, U = {a, A}, which generates a genotype space G = {aa, aa, AA}. In the model, an individual chooses a partner uniormly at random or reproduction, thus R i j = 1, or all i, j G, and all individuals can reproduce with each other. For the mutant to be able to invade the resident aa population, it needs to have a higher itness, which we obtain by assuming that its natural death rate is slightly reduced, namely or some > 0: D aa = D + and D aa = D. I.17 We assume that all the other parameters, u1 u 2 and c u1 u 2,v 1 v 2, are neutral which means that they are the same or each phenotype. In the article [18], Collet, Méléard and Metz studied this Mendelian diploid model under the assumption that the two alleles are co-dominant and that the allele A is slightly itter than the allele a, namely: D aa = D +, D aa = D and D AA = D. I.18 In these settings they show that, ater the invasion o the mutant AA which takes time o order ln K, the genotypes containing the a allele, aa and aa, die out exponentially ast in time o order ln K, as already derived in [14] or the haploid asexual model. Thereore, the time scales or the ixation o a new trait and the extinction o the resident trait are the same and the population is monomorphic again beore a new mutation occurs. Thus, as known rom the haploid asexual case [14], the suitably time-rescaled process converges to the TSS model o adaptive dynamics in the large population and rare mutation limit. In other words, the population jumps rom one homozygote to another homozygote population according to higher itness and no genetic variability is obtained. Thereore, to get genetic variability it is necessary to ensure, that the recessive allele survives in the population long enough such that a new advantageous mutation can appear beore its extinction. For this reason we make the complete dominant-recessivity assumption, precisely we assume that the a allele is recessive and the A allele is dominant. Consequently, individuals with genotype aa and genotype AA have both the same phenotype A, according 23

36 I. Introduction to the third Mendelian law and since the parameters, introduced in Section 3, depend on the phenotype we get or the natural death rates: D aa = D + and D aa = D AA = D. I.19 Through the dependence o the parameters on the phenotype, this yields that the AA individuals are as it as the aa individuals and both are itter than the aa individuals. This assumption has a dramatic eect on the evolution o the population. The resulting dierent behaviour under the two assumptions can be traced back to the deterministic system that arises in the large population limit. The linear stability analysis o the unique stable ixed point n AA, corresponding to a monomorphic AA population, yields that it is degenerated i.e. has a zero eigenvalue under the dominant-recessivity assumption. This leads to a dierent long-term behaviour towards the stable ixed point n AA compared to the model with the co-dominance assumption. To be more precisely, it implies that in the deterministic system, the aa and aa populations decay in time only polynomially ast to zero, namely like 1 and 1, respectively, in contrast to the exponential decay in the co-dominant scenario, corresponding to only strictly negative eigenvalues see Figure I.5. This type o t 2 t decay o a recessive allele has been observed earlier in the context o population genetic models see, e.g., [82], Chapter 4. The main result o Chapter II is that this behaviour o Figure I.5.: Evolution o the model rom a resident aa population at equilibrium with a small amount o mutant aa, and when the alleles a and A are co-dominant let or when the mutant phenotype A is dominant right. the deterministic system translates in the stochastic model into survival o the genotypes carrying the recessive allele or a time o order almost K 1/2 α, or α > 0. This allows or a reasonable scaling o the mutation rate µ K such that a new mutant will occur with high probability in the AA population beore the a allele is extinct. To be more precisely, we scale the mutation rate by lnk 1 Kµ K K 1/2 α, as K. I.20 This scaling can be motivated as ollows: The mutation probability or an individual with genotype u 1 u 2 is given by µ K. Hence, the time until the next mutation in the whole population is o order 1 Kµ K. Since the time a mutant population needs to invade a resident 24

37 4. Outline o Chapter II and III population is o order lnk which can be adapted rom [14, 18] we set the mutation time 1 Kµ K lnk to ensure that the mutant invades beore a new mutation appears. The right hand side o I.20, K 1/2 α, is the time until which the recessive a allele survives in the population, shown in Chapter II, and corresponds to the decay time o a unction o 1 t. The exponent can be explained as ollows: since the population carrying the recessive a allele can get directly extinct when it decreases to the size o order o the natural luctuation, we can ensure its survival only until a slightly higher size, namely K 1/2+α. Notice, that the extinction o the a allele depends on the aa population which decays although it is as it as the AA population but it is disadvantaged in reproduction. The aa population behaves like the square o the aa population and thus can already die out, due to natural luctuations, but will always be reproduced by the aa population. Under this scaling it is shown in Chapter II that the survival time o the a population, consisting o all aa- and aa individuals, is o order K 1/2 α and that there will be a urther mutation beore the a allele get extinct. The main step is to show that the behaviour o the deterministic system translate into the stochastic model. The diiculty thereby is, compared to earlier works [14,18], to control the behaviour o the stochastic system, since now time diverges like a power o K ater the invasion o the AA population. This precludes to adapt the techniques used in [14] and [18], in particular the use o unctional laws o large numbers, or the like. Instead, our proo relies on the stochastic Euler scheme developed by Baar, Bovier and Champagnat [2]. This scheme combines coupling methods with discrete time Markov chains and standard potential theoretic approaches or the exit rom an attractive domain and uses results rom the theory o branching processes. Since we consider a diploid sexual model instead o a haploid asexual one as in [2] we have to handle three subpopulations rather than only two which all inluences each other. Thus to apply the Euler scheme to the present model a main task is to ind the right order to control the subpopulations over small steps such that we can show that the stochastic system ollows the deterministic system. The scaling o the mutation rate I.20 allows the itter AA population to invade the resident population but ensures that there will be a urther mutation beore the a population is extinct. Consequently, unit alleles can survive in heterozygotes and there could appear a new mutant allele, call it B, which has strong competition with the AA population but weak competition with the aa population and can coexists with the recessive aa population. In this way a resurgence o the aa population at the expense o the AA population and coexistence o the types aa and BB may be observed. This would increase the genetic variability o the population and could be a irst step in the direction o speciation. These suggestions are the motivation or the ollowing work o Chapter III Outline: The recovery o a recessive allele in a Mendelian diploid model In the third chapter we study the urther evolution o the stochastic individual based model o Chapter II in the large population limit. We show that ater environmental changes the aa population can recover and that coexistence o homozygous genotypes is possible. The results are a joint work with Dr. Loren Coquille rom the Institute Fourier University o 25

38 I. Introduction Grenoble Alpes and Pro. Dr. Anton Bovier and are available as research paper on the online-portal arxiv: A. Bovier, L. Coquille and R. Neukirch, The recovery o a recessive allele in a Mendelian diploid model, arxiv e-print , March 2017 Chapter III contains this article mostly as it has been published subject to minor modiications in ormatting. The starting point is the scenario o Chapter II ater the invasion o the mutant AA. To understand the behaviour o the stochastic system, it is important to study irst the deterministic system, corresponding to the large population limit o the stochastic counterpart, which should be a good approximation i the a population is not killed by the typical luctuations see Figure I.6. To be more concrete, we start with the Figure I.6.: time vs. individuals plot: deterministic model let stochastic model right ollowing initial conditions or the deterministic system which correspond to the situation o the stochastic model ater invasion o AA: the AA population is large and close to its equilibrium see I.11, the aa population is o size ɛ and consequently the aa population is o order ɛ 2. Moreover we assume that there occurs a urther mutation to a most dominant B allele such that individuals exhibiting the B phenotype are the ittest in the population and we start with AB individuals o size ε 3. Notice, that ater this next mutation, we have six possible genotypes, aa, aa, AA, ab, AB and BB present in the population, three o phenotype B ab, AB and BB, two o phenotype A aa and AA and only one o phenotype a aa. Thereore, we have to deal with a six-dimensional deterministic system. The high complexity o this system, composed o six interacting subpopulations, makes the analysis intricate. To be able to identiy ixed points o the system, we assume that there is no reproduction and no competition between individuals o phenotype a and phenotype B. Observe that this assumption lets aa individuals and B individuals belong to dierent species since they are reproductively isolated. In this way, we can show analytically the existence o a ixed point, p ab, where the two subpopulations aa and BB can coexist. However, the linear stability analysis ails since the system has at p ab two zero eigenvalues. Nevertheless, using the Center Maniold Theorem asserting that the qualitative behaviour o the dynamical system in a neighbourhood o the non-hyperbolic critical point p ab is determined by its behaviour on the center maniold near p ab, [44,51], we can 26

39 4. Outline o Chapter II and III indeed prove the stability o this ixed point and the convergence o the system towards it. This can be translated in an evolution o biodiversity. Notice, that in the stochastic system the population bearing the a allele can die out due to the natural luctuations i it is too small. Thereore, we prove that also in the deterministic model the a population stays above a certain level which corresponds to the order o natural luctuation in the stochastic model. Another possibility to avoid the extinction o the a allele is to ensure that the aa population only grows ater the mutation. For that reason we introduce an additional parameter η which lowers the competition o BB with aa compared to the one o BB with AA. This way, competition does not depend only on phenotypes anymore. Instead it can be interpreted as a reinement o a phenotypic competition or resources with genotypic inluences: the strength or ability to get resources o an individual not only depends on its phenotype but also on the degree o dominance o its genotype. To be more precise, the AA individuals have one more dominant allele compared to the aa individuals, namely a second A allele instead o the a allele. Consequently, they compete more strongly or resources with the BB individuals than the aa individuals do. This lack in competition accelerates the evolution o the system since η allows aa to decrease more slowly in such a way that the AA and aa population get aster to the same order which is important or the aa population to get exponential growth and thus to recover. In particular, the birth rate o an aa individual see I.16 has a term n aa n aa + 1n 2 aa, n aa + n aa + n AA I.21 describing the reproduction o aa individuals with aa- and aa individuals out o the pool o potential partners, which yields the exponential growth as soon as aa and AA are on the same order. An interesting eature o adding the system controlling parameter η is that there arises a biurcation phenomenon or η larger than some threshold. In particular, or these values o η the coexistence ixed point p ab becomes unstable and the system converges to another ixed point where all six subpopulations coexist. We discussed this by numerical simulations see Figure I.7. The reason or this dierent limiting behaviour lies in the act that i η is chosen large enough the competition elt by aa rom BB is not strong enough to orce its extinction. Thereore, a raction o aa individuals survives in the population and through Mendelian reproduction all other subpopulations are reproduced. To sum up, the main result o Chapter III is: Theorem I.2. Consider the dynamical system I.12 started with the initial conditions mentioned above. Suppose the ollowing assumptions on the parameters hold: C1 suiciently small, C2 suiciently large, C3 0 η < c/2. 27

40 I. Introduction Figure I.7.: 6-population ixed point: η = 0.6 Then the system converges to the ixed point p ab. More precisely, or any ixed δ > 0, as ɛ 0, it reaches a δ-neighbourhood o p ab in a time o order Θɛ 1/1+η n B. Moreover, it holds: 1. or η = 0, the amount o allele a in the population decays to Θɛ 1+ /1+ beore reaching Θ1, 2. or η > 4 n B, the amount o allele a in the population is bounded below by Θɛ or all t > 0, where x = Θy whenever x = Oy and y = Ox as ɛ 0. The proo o this theorem is divided into our phases. In the irst phase we show that the mutant B population grows up to a level ɛ 0 exponentially ast, with a rate corresponding to the invasion itness o an AB individual in a resident AA population, without perturbing the 3-system aa, aa, AA. The second phase ends when the aa population and the AA population are o the same order. By a comparison result we show that in this phase the eective 3-system AA, AB, BB is almost unperturbed and behaves like the 3-system aa, aa, AA analysed in Chapter II. We show that in this time BB approaches its equilibrium n BB. In Phase 3 we ensure the exponential growth o the aa population until an ɛ 0 -neighbourhood o its equilibrium n aa. Notice that the growth rate o aa, which corresponds to the invasion itness, S aa,bb, o an aa individual in a resident BB population is given by S aa,bb = D, I.22 is much larger than the one o the B population in a resident AA population in Phase 1, given by S AB,AA = D + c n AA =. I.23 28

41 4. Outline o Chapter II and III This ollows rom Assumption C2 and that there is no competition between aa and BB individuals. Using the Center Maniold Theorem [44, 51] we prove in the last phase that the stable ixed point is approached with speed 1 t. Finally we relax the assumption on the parameters and discuss the dierent limits o the resulting models by numerical simulations. A rigorous analytic analysis is not possible here since we are already unable to calculate all the ixed points. The main requirement or the recovery o the aa population is that it has a positive invasion itness in a resident BB population. Consequently, we can relax the no competition assumption between aa and the B phenotype and add small competition between them, denoted c ab. Thereore, also under this assumptions, we can end up with two or six coexisting populations depending on the parameters choices see Figure I.8. Figure I.8.: Model without reproduction between a and B: c ab = 0.1 and η = 0 let, c ab = 0.1 and η = 0.6 right I we instead remove the no reproduction assumption, that means we set R i j 1, or all i, j G, such that aa also reproduces with phenotype B, the 2- population ixed point, p ab, cannot be observed anymore. The reason or this is that as soon as the aa population could recover the coexisting BB- and aa population will instantly give birth to ab individuals. Consequently, we observe a 3-population ixed point see Figure I.9. The BB population cannot increase to its monomorphic equilibrium, n BB, anymore due to competition with ab. Adding competition c ab and the actor η yields similar results as beore, this time with three or six coexisting populations see Figure I.9. Based on reproduction among all phenotypes this model is named all-with-all model. Notice, compared to the previous models in this model the ecundity has to be much bigger to observe the recovery o the aa population since all individuals act as possible partners and thus the birth rate o aa individuals is smaller. Finally, let us mention that i we set in both models in the model without reproduction between a and B and the all-with-all model the competition between each phenotype neutral, that means c ab = c, and add a certain value η, we naturally end up in a 6-population ixed point, since as argued above the reduced competition ensures the survival o a raction o aa individuals such that all subpopulations will be produced. 29

42 I. Introduction Figure I.9.: All-with-all model: c ab = 0 and η = 0 top, c ab = 0.1 and η = 0 center, c ab = 0.9 and η = 0.4 bottom 30

43 II. Chapter Survival o a recessive allele in a Mendelian diploid model Anton Bovier and Rebecca Neukirch Abstract In this paper we analyse the genetic evolution o a diploid hermaphroditic population, which is modelled by a three-type nonlinear birth-anddeath process with competition and Mendelian reproduction. In a recent paper, Collet, Méléard and Metz [18] have shown that, on the mutation time-scale, the process converges to the Trait-Substitution Sequence o adaptive dynamics, stepping rom one homozygotic state to another with higher itness. We prove that, under the assumption that a dominant allele is also the ittest one, the recessive allele survives or a time o order at least K 1/2 α, where K is the size o the population and α > 0. We acknowledge inancial support rom the German Research Foundation DFG through the Hausdor Center or Mathematics, the Cluster o Excellence ImmunoSensation, and the Priority Programme SPP1590 Probabilistic Structures in Evolution. We thank Loren Coquille or help with the numerical simulations and or ruitul discussions. 1. Introduction Mendelian diploid models have been studied or over a century in the context o population genetics see, e.g., [107], [34], [103], [45], and [46]. Text book expositions o population genetics are given in, e.g., [24], [82], [32], and [12]. While population genetics typically deals with models o ixed population size, adaptive dynamics, a variant that has been developed in the 90ies e.g., [52], [72], and [76], allows or variable population sizes that are controlled by competition kernels that rule the competitive interaction o populations with dierent phenotypes or geographic locations. Diploid models have been considered in adaptive dynamics already by Kisdi and Geritz [59]. Starting in the mid-90ies, stochastic individual based models were introduced and investigated that allow or a rigorous derivation o many o the predictions o adaptive dynamics on the basis o convincing models or populations o interacting individuals that incorporate the canonical genetic mechanisms o birth, death, mutation, and competition see, e.g., [28], [15], [14], [36], [16], and [17]. An important and interesting eature o these models is that various scaling limits when the carrying capacity tends to ininity while mutation rates and mutation step-size tend to zero 31

44 II. Chapter: Survival o a recessive allele in a Mendelian diploid model yield dierent limit processes on dierent time-scales. In this way, Champagnat [14] proves convergence to the Trait Substitution Sequence TSS see, e.g., [28], and [75] and to the Canonical Equation o Adaptive Dynamics CEAD. Champagnat and Méléard [17] also rigorously derive the phenomenon o evolutionary branching under the assumption o coexistence. In a recent paper [2] the convergence to the CEAD is shown in the simultaneously combined limits o large population, rare mutations and small mutation steps. In the context o individual based models, so ar almost exclusively haploid populations with asexual reproduction were studied. Exceptions are the paper by Collet, Méléard and Metz [18] where the TSS is derived in a Mendelian diploid model under certain assumptions that we will discuss below and, more recently, some papers by Coron, Méléard, Porcher, and Robert [21] and Coron [19, 20]. In the present paper we pick up this line o research and study a diploid population with Mendelian reproduction similar to the one o Collet, Méléard and Metz [18], but with one notable dierence in the assumptions. Each individual is characterised by a reproduction and death rates which depend on a phenotypic trait e.g., body size, hair colour, rate o ood intake, age o maturity determined by its genotype, or which there exist two alleles A and a on one single locus. We examine the evolution o the trait distribution o the three genotypes aa, aa and AA under the three basic mechanisms: heredity, mutation and selection. Heredity transmits traits to new osprings and thus ensures the continued existence o the trait distribution. Mutation produces variation in the trait values in the population on which selection is acting. Selection is a consequence o competition or resources or area between individual. Collet, Méléard and Metz [18] have shown that in the limit o large population and rare mutations, and under a codominance assumption o alleles, the suitably time-rescaled process, converges to the TSS model o adaptive dynamics, essentially as shown by Champagnat [14] in the haploid case. We now reverse the assumption made by Collet, Méléard and Metz [18] that the alleles a and A are codominant and assume instead that A is the ittest and dominant allele, i.e., the genotypes aa and AA have the same phenotype. We show that this has a dramatic eect on the evolution o the population and, in particular, leads to a much prolonged survival o the "unit" phenotype aa in the population. More precisely, we prove that the mixed type aa decays like 1/t, in contrast to the exponential decay observed by Collet, Méléard and Metz [18]. This type o behaviour has been observed earlier in the context o population genetic models see, e.g., [82], Chapter 4. The main result o the present paper is to show that this act translated in the stochastic model into survival o the less it a allele or a time o order almost K 1/2, when K is the carrying capacity i.e. the order o the total population size. Let us emphasise that the main diiculty in our analysis is to control the behaviour o the stochastic system over a time horizon that diverges like a power o K. This precludes in particular the use o unctional laws o large numbers, or the like. Instead, our proo relies on the stochastic Euler scheme developed by Baar, Bovier and Champagnat [2]. One could probably give a heuristic derivation o this act in the context o the diusion approximation in the one locus two alleles model o population genetics see, e.g., [32], but we are not aware o a reerence where this has actually been carried out. Sexual reproduction in a diploid population amounts or each parent to transmit one o its two alleles to the genotype o the newborn. Hence, unit alleles can survive in individuals with mixed genotype and individuals with a pure genotype are potentially able to reinvade in the population under certain circumstances, i.e. a new mutant allele B that appears beore the extinction o the a allele that has strong competition with the AA population but weak competition with the aa population may lead to a resurgence o the aa population at the expense o the AA population and coexistence o the types aa and BB. This would increase the genetic variability o the population. 32

45 2. Model setup and goals In other words, i we choose the mutation time scale in such a way that there remain enough a alleles in the population when a new mutation occurs and i the new mutant can coexist with the unit aa individuals, then the aa population can recover. Numerical simulations show that this can happen but requires subtle tuning o parameters. This eect will be analysed in a orthcoming publication. Related questions have recently been addressed in haploid models by Billiard and Smadi [7]. 2. Model setup and goals 2.1. Introduction o the model We consider a Mendelian diploid model introduced by Collet, Méléard and Metz [18]. It models a population o a inite number o individuals with sexual reproduction, where each individual i is characterised by two alleles, u i 1 ui 2, rom some allele space U R. These two alleles deine the genotype o individual i, which in turn deines its phenotype, φu i 1 ui 2, through a unction φ : U 2 R. We suppress parental eects, thus φu i 1 ui 2 = φui 2 ui 1. The individual-based microscopic Mendelian diploid model is a non-linear stochastic birth-and-death process. Each individual has a Mendelian reproduction rate with mutation and a natural death rate. Moreover, there is an additional death rate due to ecological competition with the other individuals in the population. The ollowing demographic parameters depend all on the phenotype, but we suppress this rom the notation. Let us deine u1 u 2 R + the rate o birth ertility o an individual with genotype u 1 u 2, D u1 u 2 R + the rate o natural death o an individual with genotype u 1 u 2, K N the parameter which scales the population size, c u1 u 2,v 1 v 2 K R + the competition eect elt by an individual with genotype u 1 u 2 rom an individual with genotype v 1 v 2, µ K R + the mutation probability per birth event. Here it is independent o the genotype, σ > 0 the parameter scaling the mutation amplitude, mu, dh mutation law o a mutant allelic trait u + h U, born rom an individual with allelic trait u. Scaling the competition unction c down by a actor 1/K amounts to scaling the population size to order K. K is called the carrying capacity. We are interested in asymptotic results when K is very large. We assume rare mutations, i.e. µ K 1. Hence, i a mutation occurs at a birth event, only one allele changes rom u to u + σh where h is a random variable with law mu, dh and σ [0, 1]. At any time t, there is a inite number, N t, o individuals, each with genotype in U 2. We denote by u 1 1 u1 2,..., unt 1 unt 2 the genotypes o the population at time t. The population, ν t, at time t is represented by the rescaled sum o Dirac measures on U 2, ν t = 1 K N t i=1 δ u i 1 u i 2. II.1 33

46 II. Chapter: Survival o a recessive allele in a Mendelian diploid model ν t takes values in { M K 1 = K n i=1 δ u i 1 u i 2 } n 0, u 1 1u 1 2,..., u n 1 u n 2 U 2, II.2 where M denotes the set o inite, nonnegative measures on U 2 equipped with the vague topology. Deine ν, g as the integral o the measurable unction g : U 2 R with respect to the measure ν M K. Then ν t, 1 = Nt K and or any u 1u 2 U 2, the positive number ν t, 1 u1 u 2 is called the density at time t o the genotype u 1 u 2. The generator o the process is deined as by Collet, Méléard and Metz [18]: First we deine, or the genotypes u 1 u 2, v 1 v 2 and a point measure ν, the Mendelian reproduction operator: A u1 u 2,v 1 v 2 F ν [ F = 1 4 ν + δ u 1 v 1 K + F ν + δ u 1 v 2 + F ν + δ u 2 v 1 + F ν + δ ] u 2 v 2 F ν, K K K and the Mendelian reproduction-cum-mutation operator: M u1 u 2,v 1 v 2 F ν = 1 [ F ν + δ u 1 +hv 1 8 K R + F ν + δ u 1 +hv 2 K m σ u 1, h + F ν + δ u 2 +hv 1 + F ν + δ u 2 +hv 2 m σ u 2, h K K + F ν + δ u 1 v 1 +h + F ν + δ u 2 v 1 +h m σ v 1, h K K + F ν + δ u 1 v 2 +h + F ν + δ ] u 2 v 2 +h m σ v 2, h K K II.3 dh F ν. II.4 The process ν t t 0 is then a M K -valued Markov process with generator L K, given or any bounded measurable unction F : M K R and ν M K by: L K F ν = D u1 u 2 + c u1 u 2,v 1 v 2 νdv 1 v 2 F ν δ u 1 u 2 F ν Kνdu 1 u 2 U 2 U 2 K v1 v + 1 µ K 2 u1 u 2 U 2 U 2 ν, A u 1 u 2,v 1 v 2 F ννdv 1 v 2 Kνdu 1 u 2 v1 v + µ K 2 u1 u 2 U 2 U 2 ν, M u 1 u 2,v 1 v 2 F ννdv 1 v 2 Kνdu 1 u 2. II.5 The irst non-linear term describes the competition between individuals. The second and last nonlinear terms describe the birth without and with mutation. There, v1 v 2 u1 u 2 K ν, is the reproduction rate o an individual with genotype u 1 u 2 with an individual with genotype v 1 v 2. Note that we assume random mating with multiplicative ertility i.e. that birth rate is proportional to the product o the ertilities o the mates. For all u 1 u 2, v 1 v 2 U 2, we make the ollowing Assumptions A: 34

47 2. Model setup and goals A1 The unctions, D and c are measurable and bounded, which means that there exist, D, c < such that 0 u1 u 2, 0 D u1 u 2 D and 0 c u1 u 2,v 1 v 2 c. II.6 A2 u1 u 2 D u1 u 2 > 0 and there exists c > 0 such that c c u1 u 2,v 1 v 2. A3 For any σ > 0, there exists a unction, m σ : R R +, such that m σ hdh < and m σ u, h m σ h or any u U and h R. For ixed K, under the Assumptions A1 + A3 and assuming that E ν 0, 1 <, Fournier and Méléard [36] have shown existence and uniqueness in law o a process with ininitesimal generator L K. For K, under more restrictive assumptions, and assuming the convergence o the initial condition, they prove the convergence in DR +, M K o the process ν K to a deterministic process, which is the solution to a non-linear integro-dierential equation. Assumption A2 ensures that the population does not explode or becomes extinct too ast Goal We start the process with a monomorphic aa population, where one mutation to an A allele has already occurred. That means, the initial population consists only o individuals with genotype aa except one individual with genotype aa. The mutation probability or an individual with genotype u 1 u 2 is given by µ K. Hence, the time until the next mutation in the whole population is o order 1 Kµ K. Since the time a mutant population needs to invade a resident population is o order ln K see, e.g., [14], we set the mutation rate 1 Kµ K lnk in order to be able to consider the ate o the mutant and the resident population without the occurrence o a new mutation. In this setting, the allele space U = {a, A} consists only o two alleles. Our results will imply that i the mutation rate is bigger than 1 KK 1/2 α, α > 0, then a mutation will occur while the resident phenotypic a population is small, but still alive, in contrast to the setting o Collet, Méléard and Metz [18], where the a allele dies out by time ln K. This dierent behaviour can be traced to the deterministic system that arises in the large K limit. Figure II.1 A allele ittest and dominant and Figure II.2 a and A alleles co-dominant show simulations o the deterministic systems o the two dierent models. We see that in the settings o Collet, Méléard and Metz [18] the mixed type aa dies out exponentially ast, whereas in the model where A is the dominant allele, the mixed type decays much slowly. We will show below that this is due to the act that, under our hypothesis, the stable ixed point o the deterministic system is degenerate, leading to an algebraic rather than exponential approach to the ixed point. The main task is to prove that this translates into a survival o the unit allele in the stochastic model or a time o order K β. We show that this is indeed the case, with β = 1/2 α. This implies that or mutation rates o order 1/K ln K, a urther mutant will occur in the AA population beore the aa allele is extinct Assumptions on the model Let N uv t be the number o individuals with genotype uv {aa, aa, AA} in the population at time t and set n uv t 1 K N uvt. 35

48 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Deinition II.1. The equilibrium size o a monomorphic uu population, u {a, A}, is the ixed point o a 1-dim Lotka-Volterra equation and is given by Deinition II.2. For any u, v {a, A}, n u = uu D uu c uu,uu. II.7 S uv,uu = uv D uv c uv,uu n u, II.8 is called the invasion itness o a mutant uv in a resident uu population, where n u is given by III.11. We assume that the dominant A allele deines the phenotype o an individual, i.e. AA and Aa individuals have the same phenotype. In particular, the ertility and the natural death rates are the same or aa and AA individuals. For simplicity, we assume that the competition rates are the same or all the three dierent genotypes. To sum up, we make the ollowing Assumptions B on the rates: B1 aa = aa AA =:, B2 D AA D aa =: D but D aa = D +, B3 c u1 u 2,v 1 v 2 =: c, u 1 u 2, v 1 v 2 {aa, aa, AA}. Remark II.1. We choose constant ertilities and constant competition rates or simplicity. What is really needed, is that the itness o the aa and AA types are equal and higher than that o the aa type. Observe that, under Assumptions B, S aa,aa = S AA,aa = D c n aa = D c D =, c S aa,aa = S aa,aa = D c n AA = D c D c =. II.9 Thereore, the aa individuals are as it as the AA individuals and both are itter than the aa individuals. In our model, an individual chooses a partner uniormly at random or reproduction, and, according the Mendelian laws, each individual transmits one allele, chosen uniormly at random rom its genotype, to the ospring s genotype. For example, i we want to produce an individual with genotype aa, there are our possible combinations or the parents: aa aa, aa aa, aa aa and aa aa. The irst combination results in an aa individual with probability 1, the second and third one with probability 1 2 and the last one with probability 1 4. Thereore we have the ollowing birth rates: b aa N aa t, N aa t, N AA t = N aa t N aat 2 N aa t + N aa t + N AA t, b aa N aa t, N aa t, N AA t = 2 N aa t N aat N AA t N aat, N aa t + N aa t + N AA t b AA N aa t, N aa t, N AA t = N AA t N aat 2 N aa t + N aa t + N AA t. II.10 36

49 3. Main theorems The death rates are the sum o the natural death and the competition: d aa N aa t, N aa t, N AA t = N aa t D + + cn aa t + n aa t + n AA t, d aa N aa t, N aa t, N AA t = N aa t D + cn aa t + n aa t + n AA t, d AA N aa t, N aa t, N AA t = N AA t D + cn aa t + n aa t + n AA t. II.11 In the sequel, the sum process, Σt, deined by Σt = n aa t + n aa t + n AA t, II.12 plays an important role. A simple calculation shows that the sum process jumps up to 1/K resp. down with rate b Σ resp. d Σ given by b Σ N aa t, N aa t, N AA t = KΣt, d Σ N aa t, N aa t, N AA t = D + cσtkσt + N aa t. II.13 II Main theorems In the sequel we denote by τ n, n 1, the ordered sequence o times when a mutation occurs in the population. We assume τ 0 = 0 and make the ollowing Assumption C on the mutation rate µ K : C lnk 1 Kµ K K 1/2 α. II.15 We recall one result rom Collet, Méléard and Metz [18] Proposition D.2 which carries over to our setting: it is shown that i the resident population n aa t is in a δ-neighbourhood o its equilibrium n a, then n aa t stays in this neighbourhood or an exponentially long time, as long as the mutant population is smaller than δ. The proo o this result is based on large deviation estimates see [37]. Proposition II.1 Proposition D.2 in [18]. Let suppν0 K = {aa} and let τ 1 denote the irst mutation time. For any suiciently small δ > 0, i ν0 K, 1 aa belongs to the δ/2- neighbourhood o n a then the time o exit o νt K, 1 aa rom the δ-neighbourhood o n a is bigger than e V K τ 1, or V > 0, with probability converging to 1. Moreover, there exists a constant M, such that, or any suiciently small δ > 0, this remains true, i the death rate o an individual with genotype aa, D + ck ν K t, 1 aa, II.16 is perturbed by an additional random process that is uniormly bounded by Mδ. We start the population process when n aa is in a δ/2-neighbourhood o its equilibrium, n a, and there is one individual with genotype aa. The irst theorem says that there is a positive probability that the mutant population ixates in the resident aa population and the second theorem gives the time or the invasion o the mutant population and a lower bound on the survival time o the recessive a allele. Deine τδ mut in{t 0 : 2n AA t + n aa t δ}, II.17 τ0 mut in{t 0 : 2n AA t + n aa t = 0}. II.18 37

50 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Theorem II.1 Proposition D.4 in [18]. Let z K be a sequence o integers such that z K K converges to n a, or K. Then lim lim P z KK δ 0 K δ aa+ 1 K δ mut τ aa δ < τ mut 0 =, II.19 where we recall that is the invasion itness o a mutant aa in a resident aa population c. III.17. We now state the main results o this paper: Theorem II.2. Consider the model veriying Assumptions A and B. Let δ > ε, α > 0 and τ hit η in{t τ mut δ : n aa t η}. II.20 Deine τ sur τ hit τ hit K 1/2+α ε. Then, conditional on survival o the mutant, i.e., on the event {τδ mut < τ0 mut }, with probability converging to one as K, the ollowing statements hold: i τ hit ε = Oln K, and ii τ sur = OK 1/2 α. Remark II.2. As long as there are aa individuals in the population, the smaller aa population does not die out, since the aa population always gives birth to aa individuals. For smaller values o the power 1 2 α in ii, the natural luctuations o the big AA population are too high: the death rate o n aa t can be too large due to the competition elt by n AA t and could induce the death o the aa population and hence also o the aa population. The next theorem states that i the mutation rate satisies Assumption C, then a new mutation to a possibly itter allele, B, occurs while some a alleles are still alive. This mutation will happen ater the invasion o the mutant population and when the mixed type aa population already decreased to a small level again. More precisely, Theorem II.3. Assume that Assumption C is satisied. Then, with probability converging to one, τ hit ε τ 1 = τ hit ε and τ 1 τ hit 0 = τ 1. II.21 The interest in this result lies in the act that a new mutant allele B that appears beore the extinction o the a allele that has strong competition with the AA population but weak competition with the aa population may lead to a resurgence o the aa population at the expense o the AA population and coexistence o the types aa and BB. Numerical simulations, which are objects o a ollowing publication, show that this can happen but requires subtle tuning o parameters. Since the proos o the main theorems Theorem II.1, II.2 and II.3 have several parts and are quite technical, we irst give an outline o them beore we turn to the details Section

51 3. Main theorems 3.1. Outline o the proos Heuristics leading to the main theorems The basis o the main theorems is the observation o the dierent behaviour o the limiting deterministic system, when A is the ittest and dominant one and when the alleles are co-dominant [18]. More precisely, they have dissimilar long term behaviour c. Figure II.1 and II.2. By analysing the systems one gets that both have the same ixed points n aa n a, 0, 0 and n AA 0, 0, n A. The computation o the eigenvalues o the Jacobian matrix at the ixed point n aa yields in both models two negatives and one positive eigenvalues. Hence in both systems the ixed point n aa is unstable. In contrast, the eigenvalues at the ixed point n AA are all negative in the system o Collet, Méléard and Metz [18] whereas in this model there is one zero eigenvalue. This leads to the dierent long term behaviour towards the stable ixed point n AA. In Collet, Méléard and Metz s [18] model the aa population dies out exponentially ast whereas in this model the degenerated eigenvalue corresponds to a decay o n aa t like a unction t = 1 t. The goal is to show that the stochastic system behaves like the deterministic system. Figure II.1.: Our Model: A ittest type and dominant Figure II.2.: Collet et al. Model: a and A co-dominant Organisation o the proos The main theorems describe the invasion o a mutant in the resident population, and the survival o the recessive allele. This invasion can be divided into three phases, in a similar way as in [14], [18], or [2] c. Figure II.3 the general idea that an invasion can be divided in these phases is much older, see, e.g., [92]: Phase 1: Fixation o the mutant population, Phase 2: Invasion o the mutant population, Phase 3: Survival o the recessive allele. The irst two phases are similar to the ones in Collet, Méléard and Metz [18], whereas the last phase will be analysed in eight steps. Technically, the analysis uses tools developed by Baar, Bovier and 39

52 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Figure II.3.: The three phases o the proo Champagnat [2] and classical potential methods see, e.g., [11]. Settings or the steps Step 1: Upper bound on Σt, Step 2: Upper bound on n aa t, Step 3: Lower bound on Σt, Step 4: Upper and lower bound on n AA t, Step 5: Decay o n aa t, Step 6: Decay time o n aa t, Step 7: Decay and decay time o n aa t, Step 8: Growth and growth time o Σt, Total decay time o n aa t. Phase 1: Fixation o the mutant population. In the irst phase we show that there is a positive probability that the itter mutant population At n aa t + 2n AA t ixates in the resident population. More precisely, as long as the mutant population size is smaller than a ixed δ, the resident aa population stays close to its equilibrium n a Proposition II.1 and its dynamics are nearly the same as beore since the inluence o the mutant population is negligible. We can approximate the dynamics o the mutant population At by a rescaled birth and death process and can show that the probability that this branching process increases to a δ-level is close to its survival probability and hence also the probability that the mutant population At n aa t + 2n AA t grows up to a size δ. This is the content o Theorem II.1. 40

53 3. Main theorems Phase 2: Invasion o the mutant population. The ixation Phase 1 ends with a macroscopic mutant population o size δ. In the second phase the mutant population invades the resident population and suppresses it. By the Large Population Approximation Theorem II.11, [36] the behaviour o the process is now close to the solution o the deterministic system II.43 with the same initial condition on any inite time interval, when K tends to ininity. Thus, we get rom the analysis o the dynamical system in Section 4 that any solution starting in a δ-neighbourhood o n a, 0, 0 converges to an ε-neighbourhood o 0, 0, n A in inite time t 2 in Figure II.3. Since we see in the dynamical system that as soon as the AA population is close to its equilibrium, the aa population decays like 1 t, we only proceed until n aa decreases to an ε-level to ensure that the duration o this phase is still inite. In [14], it is shown that the duration o the irst phase is o order Oln K and that the time or the second phase is bounded. Thus the time needed by the aa population to reach again the ε-level ater the ixation is o order Oln K c. Theorem II.2 i. From Proposition II.1 we get that the resident aa population stays in a δ-neighbourhood o its equilibrium n a an exponentially long time expv K as long as the mutant population is smaller than δ. Thus we can approximate the rate o mutation until this exit time by µ K K n a. Hence the waiting time or mutation to occur is o 1 order Kµ K. Champagnat [14] proved that there is also no accumulation o mutations in the second phase. More precisely, he shows that, or any initial condition, the probability o a mutation on any bounded time interval is very small: Lemma II.1 Lemma 2 a in [14]. Assume that the initial condition o ν t satisies sup K E ν 0, 1 <. Then, or any η > 0, there exists an ε > 0 such that, or any t > 0, lim sup P K t ν 0 n 0 : τ n t + ε < η, II.22 K Kµ K Kµ K where τ n are the ordered sequence o times when a mutation occurs, deined in the beginning o Section 3. Using Lemma II.1, we get that, or ixed η > 0, there exists a constant, η > ρ > 0, such that, or suiciently large K, P z KK δ aa+ 1 K δ τ aa 1 < ρ < δ, II.23 Kµ K where τ 1 is the time o the next mutation. Thus, the next mutation occurs with high probability not ρ beore a time Kµ K. Hence, under the assumption that lnk 1 Kµ K, II.24 c. let inequality o II.15 there appears no mutation beore the irst and second phase are completed. Phase 3: Survival o the recessive allele. The last phase starts as soon as n aa t has decreased to an ε-level. This phase is dierent rom the one in [14] and [18], since the analysis o the deterministic system in Section 4 reveals that n aa t decreases only like a unction t = 1 t, in contrast to the exponential decay in [18]. Thus, we may expect that the time to extinction will not be Oln K anymore, and the recessive allele a will survive in the population or a much longer 41

54 II. Chapter: Survival o a recessive allele in a Mendelian diploid model time. This is a situation similar to the one encountered by Baar, Bovier and Champagnat [2], where it was necessary to show that the stochastic system remains close to a deterministic one over times o order K α due to the act that the evolutionary advantage o the mutant population vanishes like a negative power o K. Just as in that case, we cannot use the Law o Large Numbers, but we adopt the stochastic Euler scheme rom [2] to show that the behaviour o the deterministic and the stochastic systems remain close or a time o order K 1/2 α. Let us put this scheme on a mathematically ooting: Figure II.4.: Steps o the proo Settings or the steps. Let γ 4 n A +, γ / ϑ 4 n A + and γ n A +. II.25 We deine stopping times depending on n aa t in such a way that we can control the other processes n aa t, n AA t, and Σt on the resulting time intervals. Fix ε > 0 and ϑ > 0 such that ε < 2 < ϑ <. We set 1 + ϑ 2 x =, II

55 3. Main theorems and, or 0 i lnεk 1/2 α lnx 1 =: σ 1, with α > 0, we deine the stopping times on n aa t by { τ hit τ i aa ε {, } or i = 0, in t τ i 1 aa : n aa t x i II.27 ɛ, else, { τ i+ aa in t τ i aa : n aat x i ɛ + K 1/2+3/4α}. II.28 [ ] During the time intervals t τ i aa, τ i+ aa τ i+1 aa, n aa t [ x i+1 ε, x i ε + K 1/2+3/4α]. The upper bound on i is chosen in such a way that x i+1 ε K 1/2+α. The ollowing eight steps will be iterated rom i = 0 to i = σ 1 = lnεk 1/2 α lnx 1. II.29 Remark II.3. Since in Phase 3 the biggest contribution to the birth rate o n aa t is given by the combination o two aa individuals, n aa behaves like n 2 aa. We let n aa decrease only until K 1/2+α. Aterwards it would be o smaller order than K 1/2 and the natural luctuations o the big AA population, o order K 1/2, would induce its death. Step 1: Upper bound on Σt. We show that, on the time interval t [ τ i aa, τ i+ e V ] Kα/2, there exists a constant, MΣ > 0, such that the sum process Σt does not τ i+1 aa exceed the level n A + M Σ K 1/2+α/2, with high probability: Proposition II.2. For all M > 0 and 0 i τ α Σ,M in lnεk 1/2 α lnx 1, let { t > τ i aa : Σt n A MK 1/2+α/2}. aa II.30 Then there exists a constant M Σ > 0 such that [ P τσ,m α Σ < τ i+ aa τ i+1 aa e V Kα/2] = ok 1. II.31 To prove Proposition II.2, we deine the dierence process between ΣtK and n A K and couple it with a birth-death process. We show that this process jumps up with probability less than 1 2 and show that the probability that the process reaches the level M Σ K 1/2+α/2 beore going to zero, is very small. Then we show that the process returns many times to zero until it reaches the level M Σ K 1/2+α/2 and calculate the time or one such return. Remark II.4. This is only a coarse bound on the sum process Σt but with our initial conditions we are not able to get a iner one. Ater Step 7 we have enough inormation to reine it but or the iteration this upper bound suices. Step 2: Upper bound on n aa t. An [ upper bound on n aa t is obtained similarly as in Step 1. We show that, on the time interval t τ i aa, τ i+ aa τ i+1 aa e V Kα/2], there exists a constant, M aa > 0, such that the aa population does not exceed the level γ x 2i ε 2 + M aa x 2i ε 2 1+α, with high probability: 43

56 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Proposition II.3. For all M > 0 and 0 i lnεk 1/2 α lnx 1, let τ α aa,m in { t > τ i aa : n aat γ x 2i ε 2 Mx 2i ε 2 1+α}. II.32 Then there exists a constant, M aa > 0, such that [ P τaa,m α aa < τ i+ aa τ i+1 aa e V Kα/2] = ok 1. II.33 The proo is similar to the one o Proposition II.2. Step 3: Lower bound on Σt. With [ Proposition II.2 and II.3, we can bound Σt rom below. We show that on the time interval t τ i aa, τ i+ aa τ i+1 aa e V Kα/2] the sum process does not drop below n A +ϑ c n A γx 2i ε 2 M Σ K 1/2+α/2, with high probability: Proposition II.4. For all M > 0 and 0 i 1, let τ α Σ,M in lnεk 1/2 α lnx { t > τ i aa : Σt n A + ϑ } γx 2i ε 2 MK 1/2+α/2. II.34 c n A Then there exists a constant, M Σ > 0, such that [ P τσ,m α Σ < τ i+ aa τ i+1 aa e V Kα/2] = ok 1. II.35 The proo is similar to those o Proposition II.2 and II.3. Step 4: Lower and upper bound on n AA t. Since we now have bounded the processes n aa t, n aa t and Σt rom above and below or n aa t it suices to set the lower bound [ to zero, it is easy to get a lower and an upper bound on n AA t on the time interval t τ i aa, τ i+ aa τ i+1 aa e V Kα/2]. Precisely, there exists a constant, M AA > 0, such that with high probability n AA t does not drop below n A x i ε M AA x 2i ε 2 + K 1/2+3/4α Proposition II.5, and does not exceed the level n A x i+1 ε + M AA K 1/2+α/2 Proposition II.6: Proposition II.5. For all M > 0 and 0 i 1, let lnεk 1/2 α lnx { τaa,m 2i in t > τ i aa : n AAt n A x i ε } Mx 2i ε 2 + K 1/2+3/4α. II.36 Then there exists a constant M AA > 0 such that [ P τaa,m 2i AA < τ i+ aa τ i+1 aa e V Kα/2] = ok 1. II.37 Proposition II.6. For all M > 0 and 0 i τ α AA,M in lnεk 1/2 α lnx 1, let { t > τ i aa : n AAt n A x i+1 ε MK 1/2+α/2}. II.38 Then there exists a constant, M AA > 0, such that [ P τaa,m α AA < τ i+ aa τ i+1 aa e V Kα/2] = ok 1. II.39 44

57 3. Main theorems Step 5: [ Decay o n aa t. We now have upper and lower bounds or all the single processes, or t τ i aa, τ i+ aa τ i+1 aa e V Kα/2]. Using these bounds, we prove that n aa t has the tendency to decrease on a given time interval. We show that, with high probability, n aa t, restarted at x i ε i.e. we set τ i aa = 0, hits the level xi+1 ε beore it reaches the level x i ε + K 1/2+3/4α. Proposition II.7. There exists a constant C > 0 such that, or all 0 i 1 [ P τ i+ aa < τ i+1 ] aa naa 0 = x i ε K 1/2 3/4α exp CK 7/4α. lnεk 1/2 α lnx II.40 For the proo we couple n aa t with majorising and minorising birth-death processes and show that these processes jump up with probability less than 1 2. This way we prove that with high probability n aa t reaches x i+1 ε beore going back to x i ε + K 1/2+3/4α. Step 6: Decay time o n aa t. This is the step where we see that n aa t decays like a unction t = 1 t. Precisely, it is shown that the time which the aa population needs to decrease rom x i ε to x i+1 1 ε is o order x i ε : Proposition II.8. Let θ i aa in { t 0 : n aa t x i+1 ε naa 0 = x i ε }, II.41 [ be the decay time o n aa t on the time interval t τ i aa, τ i+ aa τ i+1 aa e V Kα/2]. Then there exist positive constants, C l, C u, and a constant M > 0, such that or all 0 i lnεk 1/2 α lnx 1 [ Cu P x i ε θ iaa C ] l x i 1 exp MK 2α. ε II.42 To prove this proposition we calculate an upper and a lower bound on the decay time o the majorising resp. minorising processes obtained in Step 5 which are o the same order. Precisely, we estimate the number o jumps the processes make until they reach x i+1 ε, and the time o one jump. Step 7: Decay and decay time o n aa t. To carry out the iteration, we have to ensure that, on a given time interval, the aa population decreases below the upper bound needed or the next iteration step. Precisely, we show that n aa t decreases rom γ x 2i ε 2 + M aa x 2i ε 2 1+α to γ /2 x 2i+2 ε 2, and stays smaller than γ x 2i+2 ε 2 when n aa t reaches x i+1 ε: [ Proposition II.9. For t τ i aa, τ i+ aa τ i+1 aa e V Kα/2], the process n aa decreases rom x i ε to x i+1 ε and n aa t decreases rom γ x 2i ε 2 + M aa x 2i ε 2 1+α below γ x 2i+2 ε 2. The proo o this proposition has three parts: First, as in Step 5, we show that n aa t has the tendency to decrease and that it reaches γ /2 x 2i+2 ε 2 beore going back to γ x 2i ε 2 +M aa x 2i ε 2 1+α. The second part is similar to Step 6, where we estimate the number o jumps and the duration o one jump o the process. In the last part we show, as in Step 2, that the process stays below γ x 2i+2 ε 2 until n aa t hits the level x i+1 ε and the next iteration step starts. 45

58 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Step 8: Growth and growth time o Σt. Similarly to Step 7, we also have to ensure that the sum process increases rom the level n A +ϑ c n A γx 2i ε 2 M Σ K 1/2+α/2 to n A +ϑ/2 c n A γx 2i+2 ε 2 on a given time interval and is greater than n A +ϑ c n A γx 2i+2 ε 2 when the aa population reaches the level x i+1 ε. Observe that this step is only needed or i, lnεk 1/4 α/4 lnx lnεk 1/4 α/4 lnx aterwards x 2i ε 2 K 1/2+α/2 and the bound n A M Σ K 1/2+α/2 suices or the iteration. Proposition II.10. For 0 i, while n aa decreases rom x i ε to x i+1 ε, the sum process Σt increases rom n A +ϑ c n A x 2i ε 2 M Σ K 1/2+α/2 to n A +ϑ/2 c n A γx 2i+2 ε 2 and stays above n A +ϑ c n A γx 2i+2 ε 2 until the aa population hits the x i+1 ε-level. The proo uses the same three parts as described in the proo o Proposition II.9. Final Step: Total decay time o n aa t. We iterate Step 1 to 8 until i= lnεk 1/2 α lnx the value or which n aa t is o order K 1/2+α. Finally, we sum up the decay time o the aa population in each iteration step and get the desired result Theorem II.2 ii. Moreover, we ensure that the upper bound on the mutation probability µ K in Assumption C II.15 is satisied. 4. The deterministic system 4.1. The large population approximation The main ingredient or the second phase is the deterministic system, since we know rom [36] or [18] that, or large populations, the behaviour o the stochastic process is close to the solution o a deterministic equation. Thus we analyse it here. Proposition II.11 Proposition 3.2 in [18]. Let T > 0 and C R 3 + be compact. Assume that the initial condition 1 K N aa, 0 NaA 0, N AA 0 converges almost surely to a deterministic vector x 0, y 0, z 0 C when K goes to ininity. Let xt, yt, zt = φt; x 0, y 0, z 0 denote the solution to baa xt, yt, zt d aa xt, yt, zt φt; x 0, y 0, z 0 = baa xt, yt, zt d aa xt, yt, zt =:Xxt, yt, zt, baa xt, yt, zt d AA xt, yt, zt II.43 where aa xt baa xt, yt, zt = aayt 2 aa xt + aa yt + AA zt, d aa xt, yt, zt = xtd aa + c aa,aa xt + c aa,aa yt + c aa,aa zt, and similar expression or the aa and AA types. Then, or all T > 0, lim sup K t [0,T ] or all uv {aa, aa, AA}. n uv t φ uv t; x 0, y 0, z 0 = 0, a.s., II.44, 46

59 4. The deterministic system Thus, to understand the behaviour o the process we have to analyse the deterministic system II.43 above. The vector ield II.43 o the model we consider is given by Xx, y, z = X x, y, z = which has some particular properties: x+ 1 2 y2 x+y+z 2 x+ 1 2 yz+ 1 2 y x+y+z z+ 1 2 y2 x+y+z Theorem II.4. Assume A+B and let ε > 0, then D + + cx + y + zx D + cx + y + zy, D + cx + y + zz II.45 i the vector ield II.45 has the unstable ixed point n aa n a, 0, 0 and the stable ixed point n AA 0, 0, n A, ii the Jacobian matrix at the unstable ixed point n aa, DX n aa, has two negative and one positive eigenvalues, iii the Jacobian matrix at the stable ixed point n AA, DX n AA, has two negative and one zero eigenvalues, iv or 0 < ϱ <, and as soon as the aa population decreased to an ε-level, then 2 n A + + ϱt + 2 n A+ ε n aa t 2 n A + ϱt + 2 n A+ ε. II.46 There is also a biological explanation or the behaviour o n aa t described in Theorem II.4 iv. Since the A allele is the ittest and dominant one and because o the phenotypic viewpoint the aa population is as it as the AA population and both die with the same rate. The aa population only decreases because o the disadvantage in reproduction due to the less it, decreasing aa population. Observe that Theorem II.4 i+ii also holds in the model o Collet, Méléard and Metz [18] c. Proposition 3.3 therein but the Jacobian matrix o their ixed point n AA has three negative eigenvalues and thus they get the exponential decay o n aa t. The behaviour o solutions o the deterministic system can be analysed using the ollowing result o Collet, Méléard and Metz [18]: Theorem II.5 Theorem C.2 in [18]. Let ζ = u A u a be the variation o the allelic trait. Suppose it is non zero and o small enough modulus. I ζ ds aa,aa dζ 0 > 0 then the ixed point n aa is unstable and we have ixation or the macroscopic dynamics. More precisely, there exists an invariant stable curve Γ ζ which joins n aa to n AA. Moreover there exists an invariant tubular neighbourhood V o Γ ζ such that the orbit o any initial condition in V converges to n AA. I ζ ds aa,aa dζ 0 < 0 the ixed point n aa is stable and the mutant disappears in the macroscopic dynamics. Their proo works as ollows. First they consider the unperturbed version X 0 o the vector ield II.43 in the case o neutrality between the alleles A and a. That is u1 u 2 =, D u1 u 2 = D and C u1 u 2,v 1 v 2 = c, or u 1 u 2, v 1 v 2 {aa, aa, AA}. They get that this system has a line o ixed points Γ 0 which is transversally hyperbolic. Aterwards they consider the system X ζ with small 47

60 II. Chapter: Survival o a recessive allele in a Mendelian diploid model perturbations ζ. From Theorem 4.1 in Hirsch, Pugh and Shub [51] they deduce that there exists an attractive and invariant curve Γ ζ, converging to Γ 0, as ζ 0. Hence, there is a small enough tubular neighbourhood V o Γ 0 such that Γ ζ is contained in V and attracts all orbits with initial conditions in V c. Figure II.5. To show that the orbit o any initial condition on Γ ζ converges to one o the two ixed points n aa or n AA, one have to ensure that the vector ield does not vanish on Γ ζ, except or these two ixed points. Since the curve Γ ζ is attractive, it is equivalent to look or the ixed points in the tubular V. For inding the zero points in V, Collet, Méléard and Metz [18] use linear combinations o the let eigenvectors o DX 0 Γ 0 v t, with the perturbed vector ield X ζ. First they quote to zero the two linear combinations with the eigenvectors which span the stable aine subspace. By the implicit unction Theorem, they get a curve which contains all possible zeros in V c. Proposition B.2 in [18]. Then they consider the last linear combination and look or the points on the received curve where it vanish. Under the conditions that the derivative o the third linear combination at the point n A is non zero and does not vanishes between the ixed points ± n A they get that X ζ has only two zeros in a tabular neighbourhood o Γ 0 c. Theorem B.4 in [18]. We have to do some extra work to get the same result or the model with dominant A allele since the derivative o the third linear combination in our model, described above, is zero at the point n A. But by an easy calculation see II.77 and II.78 we can indeed prove that this point is an isolated zero and we can deduce rom Theorem II.5 the ollowing corollary, which is the main result we need about the dynamical system. Corollary II.1. Let 0 small enough. i The attracting and invariant curve Γ o the perturbed vector ield X II.43 contained in the positive quadrant, is the piece o unstable maniold between the equilibrium points n aa and n AA. ii There exists an invariant tubular neighbourhood V o Γ such that the orbit o any initial condition in V converges to the equilibrium point n AA. Hence, i we start the process in a neighbourhood o the unstable ixed point n aa, it will leave this neighbourhood in inite time and converge to a neighbourhood o the stable ixed point n AA. 5. Proos o Theorem II.4 and the main theorems 5.1. Analysis o the deterministic system Because o Proposition II.11 we have to analyse the deterministic system II.43 a simulation is shown in Figure II.1. Proo o Theorem II.4. In the ollowing we consider the dierential equations o n aa t, 48

61 5. Proos o Theorem II.4 and the main theorems Figure II.5.: The curves Γ 0, Γ and the tube V in the perturbed vector ield X n aa t and n AA t, given by II.45: naa t ṅ aa t = n aat 2 n aa td + + cσt, Σt naa t ṅ aa t = 2 n aat n AA t n aat n aa td + cσt, Σt naa t ṅ AA t = n aat 2 n AA td + cσt. Σt II.47 II.48 II.49 The Fixed Points: By summing II.47 to II.49 irst, and two times II.49 and II.48, we see that the vector ield II.45 vanishes or the points n aa and n AA. The Jacobian matrix at the ixed point n aa is given by + D + + D D + DX n a, 0, 0 = 0 2. II The matrix has the three eigenvalues λ 1 = D, λ 2 = and λ 3 =. For small enough and rom Assumption A2 we know that λ 1, λ 3 < 0, whereas λ 2 > 0. Thus the ixed point n aa is unstable. The Jacobian matrix at the ixed point n AA is given by 0 0 DX 0, 0, n A = 2 0 0, II D + D + D it has the three eigenvalues λ 1 = < 0, λ 2 = 0 and λ 3 = D < 0, rom Assumption A2. The act that one o the eigenvalues is zero is the main novel eature o 49

62 II. Chapter: Survival o a recessive allele in a Mendelian diploid model this system compared to that count in [18]. Because o the zero eigenvalue, n AA is a nonhyperbolic equilibrium point o the system and linearization ails to determine its stability properties. Instead, we use the result o the center maniold theory [51, 88] that asserts that the qualitative behaviour o the dynamical system in a neighbourhood o the non-hyperbolic critical point n AA is determined by its behaviour on the center maniold near n AA. Theorem II.6 The Local Center Maniold Theorem in [88]. Let C r E, where E is an open subset o R n containing the origin and r 1. Suppose that 0 = 0 and D0 has c eigenvalues with zero real parts and s eigenvalues with negative real parts, where c + s = n. Then the system ż = z can be written in diagonal orm ẋ = Cx + F x, y ẏ = P y + Gx, y, II.52 where z = x, y R c R s, C is a c c-matrix with c eigenvalues having zero real parts, P is a s s-matrix with s eigenvalues with negative real parts, and F 0 = G0 = 0, DF 0 = DG0 = 0. Furthermore, there exists δ > 0 and a unction, h C r N δ 0, where N δ 0 is the δ-neighbourhood o 0, that deines the local center maniold and satisies: Dhx[Cx + F x, hx] P hx Gx, hx = 0, II.53 or x < δ. The low on the center maniold W c 0 is deined by the system o dierential equations ẋ = Cx + F x, hx, II.54 or all x R c with x < δ. The act that the center maniold o our system near n AA has dimension one, simpliies the problem o determining the stability and the qualitative behaviour o the low on it near the non-hyperbolic critical point. The Local Center Maniold Theorem shows that the non-hyperbolic critical point n AA is indeed a stable ixed point and that the low on the center maniold near the critical point behaves like a unction 1 t. This can be seen as ollows: Assume that n aa t has decreased to a level ε. Let n AA t = zt, n aa t = yt and n aa t = xt. By the aine transormation n AA n AA n A we get a translated system Y z, y, x = z+y+x+ n A D + cz + y + x + n A z + n A D + cz + y + x + n A y, z+ n A+ 1 2 y2 2 z+ n A+ 1 2 yx+ 1 2 y z+y+x+ n A x+ 1 2 y2 z+y+x+ n A D + + cz + y + x + n A x II.55 which has a critical point at the origin. The Jacobian matrix o Y at the ixed point 0, 0, 0 is given by D D 2 D DY 0, 0, 0 = 0 0 2, II

63 5. Proos o Theorem II.4 and the main theorems which has the eigenvalues λ 1 = D, λ 2 = 0 and λ 3 = + with corresponding eigenvectors D+ 2 D D+ EV 1 = 0, EV 2 = 1, and EV 3 = 2. II We perorm a new change o variable to work in the basis o eigenvectors o Y z, y, x D 1 1 D+ 2 T = 0 1 +, II The change o variables casts the system into the orm z = ż + ẏ + z z ỹ = T y, x x II.59 D D + ẋ, ỹ = ẏ ẋ, x = ẋ. II.60 Let hỹ be the local center maniold. We approximate hỹ = h ỹ2 1 h 2 +Oỹ 3 and substitute the series expansions into the center maniold equation III.304 2h1 ỹ 2h 2 ỹ ỹh 1 ỹ 2, ỹ, h 2 ỹ 2 = zh 1 ỹ 2, ỹ, h 2 ỹ 2 xh 1 ỹ 2, ỹ, h 2 ỹ 2. II.61 To determine h 1 and h 2 we compare the coeicients o the same powers o ỹ. We irst consider ỹh 1 ỹ 2, ỹ, h 2 ỹ 2 and get that the coeicient o ỹ is zero. The irst coeicients which are not zero at the right hand side are the ones o ỹ 2. Thus we have to compare these with the coeicient o ỹ 2 on the let hand side which is zero. Hence, the coeicient o xh 1 ỹ 2, ỹ, h 2 ỹ 2 o ỹ 2, which is h n A, equals zero and we get h 2 = 4 n A +. From the coeicient o zh 1 ỹ 2, ỹ, h 2 ỹ 2 o ỹ 2, which is Dh 1 2D+ 4 n A + D+, we get h 2D+ 1 = 4 n A D+ D+. Thus the local center maniold is given by hỹ = 2 4 n A D+ D+ 4 n A + ỹ 2 + Oỹ 3. Substitution o this result into ỹ yields the low on the local center maniold ỹ = ỹ2 + Oỹ 3. 2 n A + We can bound the solution o this equation with initial condition y0 = ε by 2 n A + + ϱt + 2 n A+ ε ỹt 2 n A + ϱt + 2 n A+ ε II.62 II.63, II.64 or 0 < ϱ <. Thus we see that n AA is a stable ixed point and n aa t approaches this ixed point like a unction 1 t. 51

64 II. Chapter: Survival o a recessive allele in a Mendelian diploid model In contrast to the model o Collet, Méléard and Metz [18], we see in the proo that n aa t does not dies out exponentially ast. Instead it decays like a unction t = 1 t, as soon as n aa + n AA n A and thus survives a longer time in the population c. Figure II.1 and II.2. Up to now, we know how the deterministic system evolves near its ixed points. Namely: i we start the process in a neighbourhood o the unstable ixed point n aa, it will leave this neighbourhood in inite time. Whereas, i the process is in a neighbourhood o the stable ixed point n AA, it converges to this point n AA, but the convergence is slower than in the model o Collet, Méléard and Metz [18]. We now turn to the analysis o the behaviour between these points. Behaviour between the Fixed Points: We show that the deterministic system II.43 moves rom a neighbourhood o the unstable ixed point n aa to a neighbourhood o the stable ixed point n AA Corollary II.1. Proo o Corollary II.1. The proo is similar to the proo o Theorem C.2 in [18]. It only diers in the last step. The general unperturbed vector ield X 0 in the case o neutrality between the a and A alleles is given by X 0 = x+ 1 2 y2 x+y+z 2 x+ 1 2 yz+ 1 2 y x+y+z z+ 1 2 y2 x+y+z D + cx + y + zx D + cx + y + zy. D + cx + y + zz II.65 The content o Theorem B.1 in [18] is that X 0 II.65 has a line o ixed points given by, Γ 0 v = v n A 2 4 n A v2 n 2 A 2 n A v+ n A 2 4 n A, v [ n A, n A ], II.66 and that the dierential o the vector ield X 0 at each point o the curve Γ 0, DX 0 Γ 0 v, has the three eigenvectors v n A 2 v n A 4 n A e 1 v = v2 n 2 2 n A 2 n A, e A 2v = v n A, e 3 v = 1 1 2, II.67 2 n A 1 v+ n A 2 4 n A with respective eigenvalues D < 0, 0 and < 0. DX 0 Γ 0 v t has the three eigenvalues, + D, 0, and, with corresponding eigenvectors β 1 v = 1 1 v+ n v+ n A A 2 n 1 A, β 2 v = v 2 n A n n A, β 3 v = v 2 n 2 A A 2 n A, II.68 1 which satisy, or any i, j {1, 2, 3} and any v, v+ n A 2 n A v n A n A v n A 2 2 n A β i v, e j v = δ i,j. II.69 52

65 5. Proos o Theorem II.4 and the main theorems Next we analyse the asymptotics o the low associated to the perturbed vector ield, X II.45, as t. From Theorem B.1 in [18] we know that the curve o ixed points, Γ 0, is transversally hyperbolic and invariant or the vector ield X 0. Thus, Theorem 4.1 in [51] implies that, or small enough, there is an attractive curve, Γ that is invariant under X. Moreover, Γ is regular and converges to Γ 0, as 0. Hence, there is a small tubular neighbourhood, V, o Γ 0 such that Γ is contained in V and attracts all orbits with initial conditions in V c. Figure II.5. We want to study the low associated to the vector ield X. From the remarks above we know that the curve Γ is attractive or this low. Thus, it suices to analyse the low on the curve. Precisely, we want to show that the vector ield does not vanish on Γ except or the two ixed points, n aa and n AA. Thus the orbit o any initial condition on Γ converges to one o the two ixed points. It is easier to look or the ixed points in the tube V, which is equivalent to looking or ixed points on Γ because o the attractiveness o this curve. Since Hirsch, Pugh and Shub [51] gives us only a local statement, we consider V in local rames. A point x, y, z V is represented by the parametrisation Mv, r, s = Γ 0 v + re 1 v + se 3 v = 1 + rγ 0 v + s d2 Γ 0 v dv 2, II.70 with v [ n A δ, n A + δ] and r, s [ δ, δ], with δ > 0 chosen small enough. The determinant o the Jacobian matrix o the transormation v, r, s x, y, z = Mv, r, s is r+1 2 and thus invertible and does not vanish i 0 < δ < 1. Moreover, it is a dieomorphism which maps [ n A δ, n A + δ] [ δ, δ] 2 to a closed neighbourhood o V. For inding the zero points in V, we use linear combinations o the let eigenvectors β i, i {1, 2, 3}, with the perturbed vector ield X. First we look or zeros o the two linear combinations o X with the eigenvectors β 1 and β 3 which spans the stable aine subspace. By the implicit unction Theorem, we obtain a curve which contains all possible zeros in V. Then we consider the last linear combination o X with β 2 and zeros o this linear combination on the curve above. Proposition II.12 Proposition B.2 in [18]. For any δ > 0 small enough, there is a number 0 = 0 δ such that, or any [ 0, 0 ], there is a smooth curve Z = r v, s v R 2, depending smoothly on and converging to 0 when tends to zero such that, or any v [ n A δ, n A + δ], we have β 1 v, X Mv, r v, s v = β 3 v, X Mv, r v, s v = 0. II.71 Moreover, i a point v, r, s with v [ n A δ, n A + δ], r and s small enough is such that then r, s = r v, s v. β 1 v, X Mv, r, s = β 3 v, X Mv, r, s = 0, II.72 Next, we look or the points o the resulting curve obtained rom Proposition II.12 where the third linear combination o the components vanishes. Since β 2 v, X0, Mv, r 0 v, s 0 v = β 2 v, X0, Mv, 0, 0 = β 2 v, X0, Γ 0 v =0, II.73 53

66 II. Chapter: Survival o a recessive allele in a Mendelian diploid model this unction vanishes or = 0 in v and we apply the Malgrange preparation Theorem [43], which provides a representation or the linear combination near = 0: β 2 v, X, Mv, r v, s v = 2 h, v + gv, II.74 where h, g are two smooth unctions. To show that the third linear combination indeed vanishes only in small neighbourhoods o the points ± n A, Collet, Méléard and Metz [18] use a representation or the unction g which is independent o the perturbation. Lemma II.2 Lemma B.3 in [18]. The unction g in II.74 is given by gv = β 2 v, X 0 Γ 0 v. II.75 Then they ensure that this unction has only two zeros, which implies, because o the representation II.74, or 0 small enough, that the perturbed vector ield X has only the two known ixed points n aa and n AA. Theorem II.7 Theorem B.4 in [18]. Assume the unction gv = β 2 v, X 0 Γ 0 v, II.76 satisies dg dv ± n A 0 and does not vanish in n A, n A. Then, or 0 small enough, the vector ield X has only two zeros in a tubular neighbourhood o Γ 0. These zeros are n aa, 0, 0 and 0, 0, n AA, with n aa and n AA regular near = 0 and n aa 0 = n AA 0 = n A. While the hypothesis dg dv ± n A 0 does not hold here, the conclusion o Theorem II.7 remains true. Namely, we have rom II.45 that x X x, y, z = 0, II.77 0 and thus gv = β 2 v, X 0 Γ 0 v = v+ n A n A v n A v n A n A, v n A 2 4 n A 0 0 = v + n Av n A 2 4 n 2. A II.78 Obviously, g± n A = 0 and g has no other zeros, in particular, it does not vanish in n A, n A. Hence, it ollows rom the representation II.74 that there is a δ > 0 such that, or small enough, β 2 v, X Mv, r v, s v has only two zeros in v [ δ n A, n A + δ]. From Theorem II.4 we get the existence o these two ixed points, which have to be the points n aa and n AA. Finally, we need the ollowing lemma, which is analogous to Theorem C.1 in Collet, Méléard and Metz [18]. Lemma II.3. a The local stable maniold o the unstable ixed point n aa = n a, 0, 0 intersects the closed positive quadrant only along the line y = z = 0. 54

67 5. Proos o Theorem II.4 and the main theorems b The local unstable maniold is contained in the curve Γ. Proo. We start by proving a. From Theorem II.4 we get the hyperbolicity, thus we can apply Theorem 4.1 in Hirsch, Pugh and Shub [51]. The Jacobian matrix DX n a, 0, 0 c. II.50 has the three eigenvalues λ 1 = D, λ 2 = and λ 3 = with corresponding eigenvectors 1 EV λ 1 = 0 = e 1 n A II EV λ 2 = = e 2 n A + O II.80 D 0 0 EV λ 3 = = e 3 n A + O. II.81 2 n A 2 n 1 A D 0 Let E s aa the two dimensional aine stable subspace spanned by the eigenvectors EV λ 1 and EV λ 3 with origin in n aa : E s aa = {x R 3 x = n a, 0, 0 t + sev λ 1 + tev λ 3, s, t R}. II.82 Again, by the lamination and permanence condition in Theorem 4.1 in [51], we get the existence o a stable maniold Waa s,loc and an unstable maniold Waa u,loc o the ixed point n aa. The local stable maniold Waa s,loc is a piece o regular maniold tangent in n aa to the subspace Eaa. s We see that the x-axis is invariant or X and is contained in Waa s,loc. From II.82, we get that Eaa s intersects the closed positive quadrant only along the line y = z = 0. Hence, the same is true or Waa s,loc since it is a piece o the subspace Eaa. s This shows a. To show b, we show that the local unstable maniold Waa u,loc is contained in the closed positive quadrant. This ollows because Waa u,loc is tangent to the linear unstable direction in EV λ 2 in n aa, which points into the positive quadrant. From Theorem 4.1 in [51] we get that the invariant curve, Γ, is unique, thus Waa u,loc Γ and b ollows by the invariance o the positive quadrant under the low. The preceding steps conclude the proo o Corollary II Proo o the main theorems in Section 3 We carry out the proos o the main theorems Section 3 in ull detail. The mutant process At = 2n AA t + n aa t jumps up resp. down by rate b A resp. d A given by: b A = 2K naa t + 1 Σt 2 n aat 2 + naa t n aat n aa t n aat = K2n AA t + n aa t = AtK, II.83 d A = 2n AA tkd + cσt + n aa tkd + cσt = AtKD + cσt. II.84 55

68 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Phase 1: Fixation o the mutant population Recall the stopping times II.28 and II.27, when the mutant population At increased to a δ-level and its stopping time o extinction. We show Theorem II.1: Proo o Theorem II.1. We start the population process with a monomorphic aa population which stays in a δ/2k-neighbourhood o its equilibrium n a K and one mutant with genotype aa, i.e. τ 0 = 0. Because o II.15, there will be no urther mutation in the process as we will show. Proposition II.1 states that i the resident population n aa is in a δ/2-neighbourhood o its equilibrium n a then n aa will stay in a δ-neighbourhood or an exponentially long time as long as the mutant population is less than δ. Hence we get that, as long as the mutant population is smaller than δ, the time the process n aa t needs to exit rom its domain n a is bigger than expv K with probability converging to 1, or some V > 0 c. [14, 18] and the dynamics o the mutant population are negligible or n aa t. With this knowledge we analyse the ate o the mutants or t < τδ mut use the comparison results o birth and death processes Theorem 2 in [14] to bound the mutant process rom below and above. We denote by the ollowing stochastic dominant relation: i P 1 and P 2 are the laws o R-valued processes, we will write P 1 P 2 i we can construct on the same probability space Ω, F, P two processes X 1 and X 2 such that the law o the processes is LX i = P i, i {1, 2} and or all t > 0, ω Ω : Xt 1 ω Xt 2 ω. First we construct a process A l t At which is the minorising process o the mutant process. This process has the birth and death rates: τ mut 0 e V K. We b l t = A l tk, d l t = A l tk[d + c n a + 2δ]. II.85 At A u t is the majorising process with rates: b u t = A u tk, d u t = A u tk[d + c n a δ]. II.86 We deine the stopping times { Tn/K l in t 0 : A l t = n } {, Tn/K u K in t 0 : A u t = n } K Θ a in {t 0 : n aa t n a > δ}, II.87 II.88 which are the irst times that the processes A l, resp. A u reach the level n K and the exit time o n aa t rom the domain [ n a δ, n a + δ]. Note that both processes A l t and A u t are super-critical. In the ollowing we use the results or super-critical branching processes proven by Champagnat [14]: Lemma II.4 Theorem 4 b in [14]. Let b, d > 0. For any K 1 and any z N/K, let P K z the law o the N/K-valued Markov linear birth and death process ω t, t 0 with birth and death rates b and d and initial state z. Deine, or any ρ R, on DR +, R, the stopping time T ρ = in{t 0 : ω t = ρ}. II.89 Let t K K 1 be a sequence o positive numbers such that ln K t K. I b > d, or any ε > 0, a b d lim K PK 1 T0 t K T εk /K = K b, lim K PK 1 K T εk /K t K = 1 d b. II.90 II.91 56

69 5. Proos o Theorem II.4 and the main theorems With respect to Theorem II.3, we prove the theorem or arbitrary µ K. From II.23 we ρ know that the next mutation occurs with high probability not beore a time Kµ K. Then [ ] [ ] P1 T δk /K l < T 0 l ρ K Kµ K Θ a P1 τδ mut < τ0 mut ρ K Kµ K Θ a [ ] T δk /K u < T 0 u ρ Kµ K Θ a. II.92 P1 K Using Proposition II.1, there exists V > 0 such that, or suiciently large K, P 1 K [ τ1 e V K ] < Θ a 1 δ. II.93 With II.93 and or K large enough we can estimate, [ ] [ P1 T δk /K l < T 0 l ρ K Kµ K Θ a P1 T δk /K l < T 0 l ρ K Kµ K e V K] δ [ ] = P1 T δk /K l < T 0 l ρ K Kµ K δ [{ } { P1 T δk /K l < T 0 l ρ K Kµ K T0 l [{ } { = P1 T δk /K l < ρ K Kµ K T0 l [ [ T δk /K l < T0 l < P1 K ρ Kµ K ] P 1 K ρ Kµ K }] δ ρ Kµ K }] δ ρ Kµ K ] δ. II.94 The extinction time or a binary branching process when b d is given by c. page 109 in [1] n d1 e b dt P n T 0 t = b de b dt, II.95 or any t 0 and n N. Under our condition on µ K II.15 in Theorem II.3, this implies that [ ] T0 l ρ Kµ K = 0. II.96 Together with Lemma II.4, we get lim K P1 K [ ] lim K P1 τδ mut < τ0 mut ρ K Kµ K Θ a 1 d l δ = b l I we next consider the upper bound o II.92, we see that [ T δk /K u < T 0 u P 1 K Similar as in II.97 ρ Kµ K Θ a ] P1 K cm + 1 [ T u δk /K < T u δ. II.97 ] ρ Kµ K. II.98 [ ] lim P K 1 τδ mut < τ0 mut ρ K Kµ K Θ a = + cmδ. II.99 Now, we let δ 0 and get the desired result. 57

70 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Phase 2: Invasion o the mutant I the mutant invades in the resident aa population then the irst phase ends with a macroscopic mutant population. Especially, we know that the aa population is o order δ due to its advantage in recombination in contrast to the AA population. Applying the Large Population Approximation Theorem II.11, [36] with this initial condition, we get that the behaviour o the process is close to the solution o the deterministic system II.43, when K tends to ininity. We approximate the population process by the solution o the dynamical system II.43. Result III.36 in Proposition II.43 is known see [17] or [18]. We use this result only until the aa population decreases to an ε-level. Phase 3: Survival o the recessive a allele This phase starts as soon as the aa population hits the ε value. At this time we restart the population-process, that means we set the time to zero. The analysis o the deterministic dynamical system up to this point shows that we get the ollowing initial conditions: n aa 0 = ε, n aa 0 4 n A + ε2 + M aa ε 2+α, II.100 II.101 n AA 0 n A ε M AA ε 2, II.102 Σ0 n A c n A γε 2 MΣ ε 2+α, II.103 where M i, i {aa, AA, Σ} are constants. In the ollowing stopping times denoted by τ are always stopping times on rescaled processes, whereas stopping times denoted by T are the stopping times o the corresponding non-rescaled processes. We transorm the birth and death rates o the processes N aa t, N aa t, N AA t and the sum process ΣtK in such a way that we can consider them as the birth and death rates o linear birth-death processes: b Σ t = ΣtK, d Σ t = DΣtK + cσ 2 tk + N aa t, b aa t = N aa t 1 n AAt + K Σt 4Σt n2 aat, d aa t = N aa td + + cσt, b aa t = N aa t 1 n aat + 2N aa t n AAt 2Σt Σt, d aa t = N aa td + cσt, b AA t = N AA t 1 n aat + K Σt 4Σt n2 aat, d AA t = N AA td + cσt. II.104 II.105 II.106 II.107 We proceed as described in the outline. Recall the settings or the steps II.26, II.27, II.28 and II

71 5. Proos o Theorem II.4 and the main theorems Step 1: We prove the upper bound o the sum process Proposition II.2. For this we construct a process, the dierence process, which records the drit rom the sum process away rom the upper bound n A K. between ΣtK and n A K is a branch- Proo o Proposition II.2. The dierence process Xt uσ ing process with the same rates as ΣtK. Set Xt uσ = ΣtK n A K, T X,uΣ 0 in{t 0 : Xt uσ = 0}, T X,uΣ α,m Σ in{t 0 : Xt uσ M Σ K 1/2+α/2 }. II.108 This is a process in continuous time. For the ollowing we need the discrete process Yn uσ associated to Xt uσ. To obtain this process we introduce a sequence o stopping times ϑ n which records the times, when Xt uσ makes a jump. Formally, ϑ n is the smallest time t such that Xt uσ Xs uσ, or all ϑ n 1 s < t. We set Xϑ uσ n = Yn uσ. This discretisation has probability less than 1 2 to make an upward jump: [ Lemma II.5. For n N such that ϑ n τ i aa, τ i+ aa τ i+1 aa e V Kα/2] and 1 k M Σ K 1/2+α/2 there exists a constant C 0 > 0 such that P[Y uσ n+1 = k + 1 Y uσ n = k] 1 2 C 0kK 1 p Σ k. II.109 Proo. With the rates II.104, we get by some straightorward computations b Σ P[Y n+1 = k + 1 Y n = k] = b Σ + d Σ n A K + k = n A K + k[ + D + c n A + kk 1 ] + N aa t D 1 2 c n A 1 2 ckk 1 1 N aat 2 n A K+k + D + c n A + kk 1 + Naat n A K+k n aat n A +kk ckk ckk 1 + naat n A + Ox i ε 4+α ckk ckk 1 + naat n A + Ox i ε 4+α = 1 2 C 0kK 1. II

72 II. Chapter: Survival o a recessive allele in a Mendelian diploid model To obtain a Markov chain we couple the process Y uσ n with a process Z uσ n via: 1 Z uσ 0 = Y uσ 0 0, II P[Zn+1 uσ = k + 1 Yn uσ < Zn uσ = k] = p Σ k, II P[Zn+1 uσ = k 1 Yn uσ < Zn uσ = k] = 1 p Σ k, II P[Zn+1 uσ = k + 1 Yn+1 uσ = k + 1, Yn uσ = Zn uσ = k] = 1, II P[Zn+1 uσ = k + 1 Yn+1 uσ = k 1, Yn uσ = Zn uσ 6 P[Zn+1 uσ = k 1 Yn+1 uσ = k 1, Yn uσ = Zn uσ Observe that by construction Z uσ n marginal distribution o Z uσ n P [ Z uσ n+1 = k + 1 Z uσ n = k ] = p Σ kp [ Y uσ n < Z uσ n Z uσ n + p Σk P[Yn+1 uσ =k+1 Y uσ n 1 P[Yn+1 uσ =k+1 Y uσ n =k] = p Σ k P [ Y uσ n =k+1 Y uσ n =k] 1 P[Yn+1 uσ =k+1 Y uσ = k] = p Σk P[Y uσ n+1 = k] = 1 p Σk P[Y uσ n+1 1 P[Yn+1 uσ n =k], II.115 =k+1 Y uσ n =k+1 Y uσ n =k] =k] Yn uσ, a.s. and that P [ Zn+1 uσ = 1 ZuΣ n is the desired Markov chain with transition probabilities =k] = k ] + P [ Yn+1 uσ = k + 1 Yn uσ P [ Y uσ n < Zn uσ Zn uσ = k ] + P [ Yn uσ = k ] P [ Y uσ n. II.116 = 0 ] = 1. The = Zn uσ Zn uσ = k ] = Zn uσ Zn uσ = k ] 1 P [ Yn+1 uσ = k + 1 Yn uσ = k ] = Zn uσ Zn uσ = k ] = p Σ k, II.117 P [ Z uσ n+1 = k 1 Z uσ n = k ] = 1 p Σ k, II.118 and invariant measure µn = n 1 k=1 1 2 C 0kK 1 n k= C 0kK 1, II with µ0 = 1 and µ1 = 1 2 +C 1. We want to calculate the probability that the Markov 0K chain Zn uσ, starting at a point 0 < zk < 1 2 M ΣK 1/2+α/2, reaches irst M Σ K 1/2+α/2 beore going to zero, which is the equilibrium potential o a one dimensional chain see [9]. Using Equation in the book by Bovier and den Hollander [11], Chapter 7.1.4, we get or K large enough P zk [Tα,M Z Σ < T0 Z ] = = zk 1 1 n=1 1 p Σ n µn 3MΣ x 2i ε 2 1+α K 1 1 n=1 1 p Σ n µn zk n 1 n=1 k=1 1+2C 0kK 1 1 2C 0 kk 1 MΣ K 1/2+α/2 n 1 n=1 k=1 1+2C 0kK 1 1 2C 0 kk 1 zk n=1 exp n 1 k=1 ln 1+2C0 kk 1 1 2C 0 kk 1 MΣ K 1/2+α/2 n=1 exp zk MΣ K 1/2+α/2 n=1 exp n 1 k=1 ln 1+2C0 kk 1 1 2C 0 kk 1 n=1 exp n 1 k=1 4C 0kK 1 n 1 k=1 4C 0kK 1 O kk

73 5. Proos o Theorem II.4 and the main theorems zk n=1 exp 2C 0 n 2 K 1 2C 0 nk 1 MΣ K 1/2+α/2 n=1 exp 2C 0 n 2 K 1 2C 0 nk 1 O nnk 1 2 zk exp 2C 0 z 2 K MΣ K 1/2+α/2 exp 2C n=1/2m Σ K 1/2+α/2 0 n 2 K 1 2C 0 nk 1 O nnk 1 2 2zK 1/2 α/2 exp 2C 0 z 2 K M Σ exp 1 2 C 0MΣ 2Kα 2C 0 M Σ K 1/2+α/2 O K 1/2+3/2α exp 2Ĉ0K α 1 4 M Σ 2 z 2 K 1 α. II.120 We denote by R the number o times that the process Zn uσ returns to zero beore it reaches M Σ K 1/2+α/2. Let Rz k = P zk [R = k] be the probability that this number is k when starting in zk. We deine the times o the n-th returns to zero: We then have T 1 0 = in{t > 0 : Z uσ t = 0}, T n 0 = in{t > T n 1 0 : Z uσ t = 0}. II.121 [ ] [ ] k 1 [ ] Rz k = P zk T 0 < Tα,M ZuΣ Σ 1 P 0 Tα,M ZuΣ Σ < T 0 P0 Tα,M ZuΣ Σ < T 0 1 A k 1 A, II.122 where [ ] A P 0 Tα,M ZuΣ Σ < T 0 = [ ] [ ] p0, ip i Tα,M ZuΣ Σ < T 0 = p0, 1P 1 Tα,M ZuΣ Σ < T 0 i 1 exp C 0 MΣK 2 α, II.123 or some inite positive constant C 0 and p0, i = P [ Z uσ n = 0 Z uσ n+1 = i]. Then P[R N] N Rz i i=1 N 1 A i 1 A = 1 1 A N. II.124 i=1 We choose, e.g., N 1 K 2 A, so that P [R N] = o K 1. Let I 0 T0 1 and I n T0 n T 0 n 1 the time the process needs or return to zero. The I j j N are i.i.d. random variables and it holds: R R+1 I n Tα,M ZuΣ Σ I n. II.125 n=1 n=1 The underlying process, the sum process Σt II.12, o Z uσ n jumps with a rate λ Σ = ΣtK + DΣtK + cσt 2 K + N aa t C λ K. II.126 Since the Markov chain Z uσ n has to jump at least one time, it holds that, or all j N, I j > W, a.s., where W expc λ K. Thus P[I j < y] P[W < y] = 1 exp C λ Ky. II

74 II. Chapter: Survival o a recessive allele in a Mendelian diploid model We have [ P Tα,M ZuΣ Σ < τ i+ aa τ i+1 aa e V Kα/2] [ P Tα,M ZuΣ Σ < e V Kα/2] [ ] [ ] = P Tα,M ZuΣ Σ < e V Kα/2 {R > N} + P Tα,M ZuΣ Σ < e V Kα/2 {R N} [ ] P Tα,M ZuΣ Σ < e V Kα/2 {R > N} + P[R N]. II.128 [ ] First we estimate P Tα,M ZuΣ Σ < e V Kα/2 {R > N}. Since Tα,M ZuΣ Σ R n=1 I n, it holds that i n 2 o the I j are greater than 2 n ev Kα/2, then T ZuΣ α,m Σ e V Kα/2. [ ] P T X,uΣ α,m Σ < e V Kα/2 {R > N} n=n n=n [ ] P T X,uΣ α,m Σ < e V Kα/2 {R = n} n P I j < e V Kα/2 n=n j=1 n P 1 { } > n. I j < 2 n ev Kα/2 2 j=1 II.129 [ We have p n P I j < 2 n ev Kα/2] [ P W < 2 n ev Kα/2] = 1 exp 2C λk n ev Kα/2. The number o random variables I j that are greater than 2 Kα/2 nev is parameters p n, n. n P 1 { } > n = I j < 2 n ev Kα/2 2 n=n j=1 n n=n m=n/2 n=n where we used that, in the range o summation, p n p N. 1/2, and 4p 1/2 N 16p N N/ exp = 16 binomial distributed with n 1 p n n m p m n m 4 n p n 2 n 16p N N/2 1 4p 1/2, II.130 N Then, or K large enough, 2C λ N KeV Kα/2 N/2 16 2Cλ N KeV Kα/2 N/2 2C λ AK 3 e V Kα/2 N/2. II.131 Recalling that A = e OKα, one sees that II.130 is bounded by o e Kα. Hence we get [ P T X,uΣ α,m Σ < τ i+ aa τ i+1 aa e V Kα/2] [ P = o This concludes the proo o Proposition II.2. ] α,m Σ < e V Kα/2 {R > N} + P[R N]. II.132 T X,uΣ 1 K 62

75 5. Proos o Theorem II.4 and the main theorems Step 2: We derive a rough upper bound on the process n aa Proposition II.3. Recall that γ n A +. II.133 Proo o Proposition II.3. The proo is similar to the one o Proposition II.2. Again, we deine the dierence process Xt aa between N aa and γ x 2i ε 2 K. This is a branching process with the same rates as n aa. Set Xt aa = N aa t γ x 2i ε 2 K, II.134 T X,aa 0 in{t 0 : Xt aa = 0}, II.135 T X,aa α,m aa in{t 0 : Xt aa M aa x 2i ε 2 1+α K}. II.136 Let Yn aa be the discretisation o Xt aa, obtained as described in Step 1. [ Lemma II.6. For n N such that ϑ n τ i aa, τ i+ aa τ i+1 aa e V Kα/2] and 1 k M aa x 2i ε 2 1+α K, there exists a constant C 0 > 0 such that P [ Y aa n+1 = k + 1 Y aa n = k ] 1 2 C 0 p aa. II.137 Proo. In the ollowing we use Proposition II.1 or the irst iteration step, since the mutant population n AA increased to an ε-neighbourhood o its equilibrium n A and the other two populations decreased to an ε order. Thus the inluence o the small aa- and aa populations is negligible or the dynamics o n AA and the AA population will stay in the n A -neighbourhood an exponentially long time. Now or the i th iteration-step we use the iner bounds o n AA t Proposition II.5 and II.6 and Σt Proposition II.4 estimated in the i 1 th iteration-step beore. By II.105, we have P[Y aa n+1 = k + 1 Y aa n = k] γ x 2i ε 2 K + k 1 n AAt Σt = [ γ x 2i ε 2 K + k 1 n AAt Σt + K 4Σt n2 aa t + D + + cσt n 2 aa t ] + K 4Σt n2 aa t = Σt n AAt + K 8Σt γ x 2i ε 2 K+k 1 2 D cσt. II n AAt Σt + D + + cσt + K n 2 aa t 4Σt γ x 2i ε 2 K+k Using Propositions II.6, II.4, and II.2 and II.25, one sees that the numerator in the second summand o II.138 is bounded rom above by 2 n A x i 1 ε 2 n A +M Σ K 1/2+α/2 + x i ε+k 1/2+3/4α 2 D+ +c n 8γ x 2i ε 2 n A +ϑ A γx c n 2i 2 ε 2 +M Σ K 1/2+α/2 A Ox i ε + + Ox i + ε = 8 n A γ Oxi ε. II.139 For ε small enough, there exists a constant C 0 > 0 such that P[Y aa n+1 = k + 1 Y aa n = k] 1 2 C 0 p aa. II

76 II. Chapter: Survival o a recessive allele in a Mendelian diploid model As in Step 1 we couple Y aa n with a process Z aa n via: 1 Z0 aa = Y0 aa 0, II P[Zn+1 aa = k + 1 Yn aa 3 P[Zn+1 aa = k 1 Yn aa 4 P[Zn+1 aa = k + 1 Yn+1 aa = k + 1, Yn aa 5 P[Zn+1 aa = k + 1 Yn+1 aa = k 1, Yn aa = Zn aa 6 P[Zn+1 aa = k 1 Yn+1 aa = k 1, Yn aa = Zn aa < Zn aa = k] = p aa, II.142 < Zn aa = k] = 1 p aa, II.143 = Zn aa = k] = 1, II.144 =k+1 Y aa n =k] 1 P[Yn+1 aa =k+1 Y aa =k+1 Y aa n =k] 1 P[Yn+1 aa =k+1 Y aa n = k] = p 0 P[Y aa n+1 = k] = 1 p 0 P[Y aa n+1 n =k], II.145 =k]. II.146 Observe that by construction Zn aa Yn aa, a.s.. The marginal distribution o Zn aa desired Markov chain with transition probabilities is the P[Zn+1 aa = k + 1 Zn aa = k] = p aa = 1 P[Zn+1 aa = k 1 Zn aa = k], II.147 and invariant measure µn = n 1 k=1 n k=1 1 2 C C 0 = 1 2 C 0 n C 0 n. II.148 The remainder o the proo is a complete re-run o the proo o Proposition II.2 and we skip the details. Step 3: We estimate the lower bound on Σt, or t [ τ i aa, τ i+ aa τ i+1 aa e V Kα/2]. Proo o Proposition II.4. The proo is similar to those o Proposition II.2 and II.3. We only perorm the crucial steps. This time the dierence process is given by the dierence o Σt and n A +ϑ c n A γ x 2i ε 2. Let X lσ t = ΣtK n A +ϑ c n A γx 2i ε 2 K, II.149 T X,lΣ 0 in{t 0 : Xt lσ = 0}, II.150 T X,lΣ α,m Σ in{t 0 : Xt lσ M Σ K 1/2+α/2 }. II.151 As described in Step 1 we construct the discrete process Yn lσ that Yn lσ jumps down with a probability less than 1 2 : Lemma II.7. For n N such that ϑ n M Σ K 1/2+α/2 there exists a constant C 0 > 0 such that associated to Xt lσ. We show [ τ i aa, τ i+ aa τ i+1 aa e V Kα/2] and 1 k P[Y n+1 = k 1 Y n = k] 1 2 C 0x 2i ε 2 C 1 kk 1 p Σ. II.152 Proo. Using the rates o the sum process II.104 and the upper bound on n aa Proposition II.3, this is a simple computation and we skip the details. 64

77 5. Proos o Theorem II.4 and the main theorems As in Step 1, to obtain a Markov chain we couple the process Y lσ n 1 Z 0 = Y 0 0, with a process Z lσ n via: II P[Zn+1 lσ = k + 1 Yn lσ > Zn lσ = k] = 1 p Σ, II P[Zn+1 lσ = k 1 Yn lσ > Zn lσ = k] = p Σ, II P[Zn+1 lσ = k 1 Yn+1 lσ = k 1, Yn lσ = Zn lσ = k] = 1, II P[Zn+1 lσ = k 1 Yn+1 lσ = k + 1, Yn lσ = Zn lσ 6 P[Zn+1 lσ = k + 1 Yn+1 lσ = k + 1, Yn lσ = Zn lσ = k] = p Σ P[Y lσ n+1 = k 1 Yn lσ= k] 1 P[Yn+1 lσ = k 1 Y lσ n = k], = k]=1 p Σ P[Y lσ II.157 n+1 = k 1 Yn lσ = k] 1 P[Yn+1 lσ = k 1 Y lσ n = k]. II.158 Observe that by construction Zn lσ Yn lσ, a.s.. The marginal distribution o Zn lσ is the desired Markov chain with transition probabilities [ ] P Zn+1 lσ = k + 1 Zn lσ = k = p Σ, II.159 [ ] P Zn+1 lσ = k 1 Zn lσ = k = 1 p Σ, II.160 and invariant measure µn = 1 2 C 0x 2i ε 2 C 1 kk 1 n 1 k=1 2 + C 0x 2i ε 2 + C 1 kk 1. II.161 n 1 k=1 The remainder o the proo ollows along the lines o the proo given in Step 1. We prove that the process returns to zero many times beore it hits M Σ K 1/2+α/2 and calculate the duration o one zero-return to get the desired result. Step 4: With Propositions II.2-II.4 and the settings [ we are able to calculate a lower Step 4.1 and an upper bound Step 4.2 or n AA t, or t τ i aa, τ i+ aa τ i+1 aa e V Kα/2]. Step 4.1: We now prove Proposition II.5, the lower bound on n AA. Proo o Proposition II.5. From Proposition II.4 we know that Σt n A +ϑ c n A γx 2i ε 2 M Σ K 1/2+α/2. With the upper bound in Proposition II.3 or n aa and the settings or the steps used or the aa population, we get n AA t = Σt n aa t n aa t n A x i ε x 2i ε 2 +ϑ c n A γ + γ M aa x 2i ε 2 1+α O n A x i ε O x 2i ε 2 + K 1/2+3/4α. K 1/2+3/4α II

78 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Step 4.2: We prove Proposition II.6, the upper bound on n AA. Proo o Proposition II.6. From Proposition II.2 we know that Σt n A + M Σ K 1/2+α/2. With the lower bound on n aa t deined in the settings we get n AA t = Σt n aa t n aa t n A + M Σ K 1/2+α/2 x i+1 ε n A x i+1 ε + OK 1/2+α/2. II.163 Step 5: Up to now we have estimated [ upper and lower bounds or all single processes: Σt, n aa t, n aa t and n AA t, or t τ i aa, τ i+ aa τ i+1 aa e V Kα/2]. Now, we prove that n aa t has the tendency to decrease on the time intervals deined in the settings. For this we restart n aa when the process hits x i ε and show that with high probability the aa- population decreases to x i+1 ε beore it exceeds again the x i ε + K 1/2+3/4α -value Proposition II.7. For this we couple n aa t with a process which minorises it and on one which majorises it and show that these processes decrease to x i+1 ε beore going back to x i ε + K 1/2+3/4α. Proo o Proposition II.7. As beore let Y aa n to N aa t. We start by coupling Y aa n c. Step 1 be the associated discrete process with a Markov chain Z n such that Zn u Yn aa, a.s.. [ τ i aa, τ i+ aa τ i+1 aa e V Kα/2] there exists a con- Lemma II.8. For n N such that ϑ n stant C 0 > 0 such that P [ Y aa n+1 = k + 1 Y aa n = k ] 1 2 C 0x i+1 ε p u aa. II.164 Proo. For t < τ i+ aa τ i+1 aa e V Kα/2, we have P [ Y aa n+1 = k + 1 Y aa n = k ] = k k 1 kk 1 2Σt kk 1 2Σt + 2N aa t n AAt Σt 1 kk 1 2Σt + 2N aa t n AAt Σt + k[d + cσt] N aa t 1 2 D 1 2 cσt. + 2 Naat n AA t k Σt + [D + cσt] 4Σt kk 1 + n AAt kσt II.165 As in the previous steps, we bound the nominator o the second summand in II.165 using Propositions II.2, II.4, and II.6, rom above by n A kk 1 + n A x i+1 ε+ok 1/2+α/2 kk 1 n A +ϑ c n A γx 2i ε 2 M Σ K 1/2+α/2 γ x 2i ε D 1 2 n c A +ϑ c n A γx 2i ε 2 M Σ K 1/2+α/2 x + O i ε 1+2α ϑ 2 x 4 n A + ϑ xi+1 ε + O i ε 1+2α + K 1/2+α/2. II

79 5. Proos o Theorem II.4 and the main theorems This term is negative since 2 < ϑ. Hence, we get P [ Y aa n+1 = k + 1 Y aa n = k ] 1 2 C 0x i+1 ε. II.167 To obtain a Markov chain we couple the process Y aa n with a process Z u n via: 1 Z u 0 = Y aa 0, II P[Zn+1 u = k + 1 Yn aa < Zn u = k] = p u aa, II P[Zn+1 u = k 1 Yn aa < Zn u = k] = 1 p u aa, II P[Z u n+1 = k + 1 Y aa n+1 = k + 1, Y aa n = Z u n = k] = 1, II P[Zn+1 u = k + 1 Yn+1 aa = k 1, Yn aa 6 P[Zn+1 u = k 1 Yn+1 aa = k 1, Yn aa = Zn u = k] = pu aa P[Y aa n+1 =k+1 Y aa n =k] 1 P[Yn+1 aa =k+1 Y aa = Zn u = k] = 1 pu aa P[Y aa n+1 n =k], II.172 =k+1 Y aa n =k] 1 P[Yn+1 aa =k+1 Y aa n =k]. II.173 Observe that by construction Zn u Yn aa, a.s.. The marginal distribution o Zn u is the desired Markov chain with transition probabilities P[Z u n+1 = k + 1 Z u n = k] = p u aa, P[Z u n+1 = k 1 Z u n = k] = 1 p u aa, II.174 II.175 and invariant measure µn = n 1 k=1 n k=1 1 2 C 0x i+1 ε C 0x i+1 ε = 2 C 0x i+1 ε n C 0x i+1 ε n. II.176 We deine the stopping times T Z i+ in { ϑ n 0 : Z n x i εk + K 1/2+3/4α}, T Z i+1 in { ϑ n 0 : Z n x i+1 εk }. II.177 II.178 For x i+1 ε z < x i ε, we get as beore the ollowing bound on the harmonic unction [ ] P zk Ti+ Zu < Ti+1 Zu = zk n=x i+1 εk+1 x i εk+k max{x 2i ε 2,K 1/2 } n=x i+1 εk+1 K 1/2 3/4α exp n 1 1+2C0 x i+1 ε 1 2C 0 x i+1 ε 1+2C0 x i+1 ε 1 2C 0 x i+1 ε CK 7/4α. n 1 II.179 Now we couple Yn aa with a Markov chain Zn l such that Zn l Yn aa, a.s.. [ Lemma II.9. For n N such that ϑ n τ i aa, τ i+ aa τ i+1 aa e V Kα/2] there exists a constant C 1 > 0 such that P [ Y n+1 = k + 1 Y aa n = k ] 1 2 C 1x i ε p l aa. II

80 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Proo. The proo is completely analogous to the proo o Lemma II.8 and we skip the details. To obtain a Markov chain we couple the process Y aa n with a process Z l n via: 1 Z l 0 = Y 0, II P[Zn+1 l = k + 1 Yn aa > Zn l = k] = p l aa, II P[Zn+1 l = k 1 Yn aa > Zn l = k] = 1 p l aa, II P[Z l n+1 = k 1 Y n+1 = k 1, Y aa n = Z l n = k] = 1, II P[Z l n+1 = k 1 Y n+1 = k + 1, Y aa n 6 P[Z l n+1 = k + 1 Y n+1 = k + 1, Y aa n = Zn l = k] = P[Y n+1=k+1 Yn aa =k] p l aa P[Y n+1 =k+1 Yn aa=k], II.185 = Zn l = k] = 1 P[Y n+1=k+1 Yn aa =k] p l aa P[Y n+1 =k+1 Yn aa=k]. II.186 Observe that by construction Zn l Yn aa, a.s.. The marginal distribution o Zn l is the desired Markov chain with transition probabilities P[Z l n+1 = k + 1 Z l n = k] = p l aa, P[Z l n+1 = k 1 Z l n = k] = 1 p l aa. II.187 II.188 Similar to the upper process, we can show that the lower process reaches x i+1 εk beore returning to x i εk + K 1/2+3/4α, with high probability. This concludes the proo o the proposition. Step 6: In this step we calculate the time which n aa t needs to decrease rom x i ε to x i+1 ε Proposition II.8. Proo o Proposition II.8. Let Zn l Yn aa Zn u be deined as in the step beore and Y 0 = Z0 l = Zu 0 = xi εk. Recalling II.106, we get λ aa t = N aa t 1 n aat + 2N aa t n AAt + N aa t[d + cσt] 2Σt Σt 2x i+1 εk + Ox 2i ε 2 K C λ x i+1 εk λ l aa, II.189 λ aa t = N aa t 1 n aat + 2N aa t n AAt + N aa t[d + cσt] Σt Σt 2x i εk + Ox 2i ε 2 K C λ x i εk λ u aa. II.190 Let n := in { n 0 : Yn aa Y0 aa 1 xx i εk } be the random variable which counts the number o jumps Yn aa Y0 aa makes until it is smaller than 1 xx i εk. The time between two jumps o n aa t is given by τ m τ m 1. It holds that Jm u τ m τ m 1 Jm, l where Jm l resp. Jm u are i.i.d. exponential distributed random variables with parameter λ l aa resp. λu aa. We want to estimate bounds or the times that the processes Zu n, resp. 68

81 5. Proos o Theorem II.4 and the main theorems Zn, l need to decrease rom x i εk to x i+1 εk. Thus we show, or constants C u, C l > 0, that [ n ] i P Jm u > 2C u C m=1 λ x i+1 exp MK 2α, II.191 ε [ n ] ii P Jm l < C l 2C λ x i exp MK 2α. II.192 ε m=1 We start by showing II.191. We need to ind N such that, with high probability, n N. To do this, we use the majorising process Z u. Let Wk u be i.i.d. random variables with P[W u k = 1] = 1 2 C 0x i+1 ε, P[W u k = 1] = C 0x i+1 ε and E[W u 1 ] = 2C 0 x i+1 ε. II.193 W u k records a birth or a death event o the process Zu n. From Lemma II.8 we get [ n P[n N] P n N : W k 1 xx i εk ] k=1 [ N P W k 1 xx i εk ] k=1 [ N 1 P W k EW k 2NC 0 x i+1 ε 1 xx i εk ]. II.194 k=1 By Hoeding s inequality and choosing N = 1 x C 0 x K =: C uk, we get P[n C u K] 1 exp C 0 x i ε 2 Kx1 x 1 exp K 2α C 0 1 xx/2, 2 II.195 where we used that x i ε K 1/2+α. Thus [ n ] [ CuK ] P m=1 J u m > 2C u C λ x i+1 ε P m=1 J u m > 2C u C λ x i+1 ε + exp K 2α C 0 x1 x/2. II.196 By applying the exponential Chebyshev inequality we get [ CuK ] P Jm u > 2C u C λ x i+1 exp C u K/2. ε m=1 II.197 Next we show II.192. For this we need to ind N such that P[n N] is very small. For this we use the process Z l. Let Wk l be i.i.d. random variables which record a birth or a death event o the process Zn. l They satisy P[Wk l = 1] = 1 2 C 1x i ε, P[Wk l = 1] = C 1x i ε and E[W1] l = 2C 1 x i ε. II

82 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Note that rom Lemma II.9 [ { P[n N] P in n 0 : Zn l Z0 l 1 xx i εk } ] N [ n = P n N : W k EW k 2nC 1 x i ε 1 xx i εk ] k=1 II.199 [ N n P W k EW k 2nC 1 x i ε 1 xx i εk ]. II.200 n=0 k=1 I we choose N = 1 x 4C 1 K =: C l K, using Hoeding s inequality, we get, or all n N, [ n P W k EW k 1 xx i εk ] xxi εk exp K 2α C 1 1 x/2. k=1 Thus [ n P Jm l < m=1 It holds that [ Cl K P Jm l > m=1 C l 2C λ x i ε ] C l 2C λ x i ε ] P [ Cl K m=1 = 1 P J l m < [ Cl K m=1 C l 2C λ x i ε J l m < ] C l 2C λ x i ε II C l K exp K 2α C 1 1 x/2. II.202 ] = 1 P [ C l K m=1 C l Jm l > 2C λ x i ε ]. II.203 A simple use o the exponential Chebyshev inequality shows that [ Cl ] K P Jm l < C l 2C m=1 λ x i exp C l K/2. II.204 ε [ n ] Thus we have that P m=1 J m l < C l exp MK 2α, or some constant M > 0. 2C λ x i ε Step 7: In this step it is shown that n aa t decreases under the upper bound γ x 2i+2 ε 2, which we need to proceed the next iteration step, in at least the time n aa t needs to decrease rom x i ε to x i+1 ε Proposition II.9. We set the time to zero when n aa t hits x i ε. Hence n aa 0 γ x 2i ε 2 + M aa x 2i ε 2 1+α. Remember, γ /2 = ϑ 4 n A + and let θ + i aa in { t 0 : n aa t γ x 2i ε 2 + 3M aa x 2i ε 2 1+α}, θ i aa in { t 0 : n aa t γ /2 x 2i+2 ε 2}. II.205 II.206 For the proo we proceed in three parts. First we show that n aa t has the tendency to decrease. For this we construct a majorising process or n aa t and show that this process decreases on the given time interval. This process is used in the second part to estimate an upper bound on the time which the aa population needs or the decay rom γ x 2i ε 2 + M aa x 2i ε 2 1+α to γ /2 x 2i+2 ε 2. As result we will get that n aa t reaches the next upper bound beore the aa population decreases to x i+1 ε. Thus in the third part we ensure that n aa t stays below the bound γ x 2i+2 ε 2 until n aa t reaches x i+1 ε. 70

83 5. Proos o Theorem II.4 and the main theorems Part 1: We show Proposition II.13. For t such that [ ] τ i aa, τ i+ aa τ i+1 aa e V Kα /2 there are constants C, C > 0 P [ θ + i aa < θ i aa n aa0 γ x 2i ε 2 + M aa x 2i ε 2 1+α] exp CK α. II.207 In this part the same strategy as in Step 5 is used. We couple n aa t with a process which majorises it and show that this process decreases rom γ x 2i ε 2 + M aa x 2i ε 2 1+α to γ /2 x 2i+2 ε 2 with high probability. Proo. As beore, let Yn aa the upper process. Lemma II.10. For n N such that ϑ n constant C u > 0 such that Proo. We know that, or t K 1/2+3/4α ]. Again with II.105 we have P [ Y aa n+1 = k + 1 Y n = k ] = be the discretisation o N aa t. We start with the construction o [ ] τ i aa, τ i+ aa τ i+1 aa e V Kα /2 there exists a P [ Y aa n+1 = k + 1 Y aa n = k ] 1 2 C u p u. II.208 [ τ i aa, τ i+ aa τ i+1 aa e V Kα/2] [, n aa t x i+1 ε, x i ε+ k k 1 n AAt Σt Σt n AAt + K 1 n AAt Σt + K 4Σt n2 aa t 1 n AAt Σt + K 4Σt n2 aa t + k[d + + cσt] n 2 aa t k 8Σt 1 2 D 1 2 cσt K n 2 aa t 4Σt k + [D + + cσt] II.209 Using Proposition II.2, II.4 and II.6, we bound the numerator in the second summand rom above by n A γ /2 x 2 + O x i ε 1 + K 1/4α + ϑ 2 1 ϑ ϑ + ϑ + O + ϑ 2 ϑ 2 Since 2 < ϑ there exists a constant C u > 0 such that x i ε + K 1/4α. II.210 P[Y n+1 = k + 1 Y n = k] 1 2 C u p u. II

84 II. Chapter: Survival o a recessive allele in a Mendelian diploid model By replacing p aa by p u, we couple Yn aa in the same way with a process Zn u as it was done in Step 2. Observe that by construction Zn u Yn aa, a.s.. The marginal distribution o Zn u is the desired Markov chain with transition probabilities and invariant measure P[Z u n+1 = k + 1 Z u n = k] = p u, P[Z u n+1 = k 1 Z u n = k] = 1 p u, µn = We deine the stopping times T + n 1 k=1 n k=1 1 2 C u C u = II.212 II C u n C u n. II.214 i aa in { t 0 : Zn u γ x 2i ε 2 K + 3M aa x 2i ε 2 1+α K }, Ti aa in { t 0 : Zn u γ /2 x 2i+2 ε 2 K }. P zk [ T + i aa < T i aa] = II.215 II.216 Again with the ormula o the equilibrium potential we estimate or γ /2 x 2i+2 ε 2 K zk < γ x 2i ε 2 K + M aa x 2i ε 2 1+α K zk 1+2Cu n=γ /2 x 2i+2 ε 2 K+1 γ x 2i ε 2 K+3M aax 2i ε 2 1+α K n=γ /2 x 2i+2 ε 2 K+1 1 2C u n 1 1+2Cu 1 2C u n 1 exp CK α. II.217 Part 2: Similary to Step 6, we calculate an upper bound on the time which n aa t needs at most to decrease rom γ x 2i ε 2 + M aa x 2i ε 2 1+α to γ /2 x 2i+2 ε 2. Proposition II.14. Let θ i aa in { t 0 : n aa t γ /2 x 2i+2 ε 2 n aa 0 = γ x 2i ε 2 + M aa x 2i ε 2 1+α}, II.218 [ be the decay time o n aa t, or t τ i aa, τ i+ aa τ i+1 aa e V Kα/2]. Then there exist inite, positive constants C aa u and M, such that or all 0 i lnεk1/2 α lnx 1 P [θ i aa > C aa u ] exp MK 2α. II.219 Proo. The proo works like the one o Proposition II.8. We calculate an upper bound on the decay time o the majorising process Zn. u Let Yn aa and Zn u be deined as in the step beore and let W u be i.i.d. random variables with P[W u = 1] = 1 2 C u, P[W u = 1] = C u and E[W u 1 ] = 2C u. II.220 The W u s record a birth or a death event o Zn. u Similar as in Step 6, we choose N = γ x 2 γ /2 +M aax i ε 2α C u x 2i ε 2 C K =: C u x 2i ε 2 K and show that P[n C C u x 2i ε 2 K] 1 exp K 2α CCu /2. It holds λ aa t = N aa t 1 n AAt Σt + K 4Σt n2 aat + N aa t[d + + cσt] C λ x 2i+2 ε 2 K λ l aa. II

85 5. Proos o Theorem II.4 and the main theorems Again, let τ m τ m 1 be the time between two jumps o n aa t and let Jm aa be i.i.d. exponential random variables with parameter λ l aa. As in Step 6, using the exponential Chebyshev inequality, we get that P C/C ux 2i ε 2 K m=1 J aa m > 2 C C u C λ x 2 exp C 2C u K 2α, II.222 and hence P [ n m=1 Jm aa > 2 C ] C u C λ x 2 exp MK 2a. II.223 Part 3: We see that n aa t reaches γ /2 x 2i+2 ε 2 beore n aa t decreases to x i+1 ε. Similar to Step 2 we can show that once n aa t hits γ /2 x 2i+2 ε 2 it will stay close to it an exponentially long time and will not exceed γ x 2i+2 ε 2 again. Thus we can ensure that n aa t is below γ x 2i+2 ε 2 when n aa t reaches x i+1 ε. Step 8: For the iteration we have to ensure that the sum process increases on the given time interval rom the value n A +ϑ c n A γx 2i ε 2 M Σ K 1/2+α/2 to n A +ϑ c n A γx 2i+2 ε 2 and stays there until the next iteration-step Proposition II.10. We set the time to zero when the aa population hits x i ε. Hence Σ0 n A +ϑ c n A γx 2i ε 2 M Σ K 1/2+α/2. Observe that we have to carry out this step only until i lnεk1/4 α/4 lnx, since aterwards x 2i ε 2 K 1/2+α/2 and thus Σt n A M Σ K 1/2+α/2 which is enough or the urther iteration. As in the proo o Proposition II.9 we divide the proo into three parts. Part 1: Similarly to Part 1 in Step 7, we show that with high probability Σt increases to n A +ϑ c n A γx 2i+2 ε 2 beore going back to n A +ϑ/2 c n A γx 2i ε 2 3M Σ K 1/2+α/2. We deine stopping times on Σt: { τ i Σ in t 0:Σt n A +ϑ c n A γx 2i ε 2 3M Σ K 1/2+α/2 vσ 3M Σ K 1/2+α/2}, II.224 { } τ i+ Σ in t 0:Σt n A +ϑ/2 c n A γx 2i+2 ε 2 vσ +, II.225 where M Σ > 0. Proposition II.15. There are constants C, C > 0 such that or all 0 i lnεk1/4 α/4 lnx P[τ i Σ < τ i+ Σ Σ0 n A +ϑ c n A γx 2i ε 2 + M Σ x 2i ε 2 1+α ] exp CK α. II.226 Proo. From the step beore we know that n aa t decreases under the value γ x 2i+2 ε 2 in a time o order 1 and does not exceed[ this bound once it hits it. ] Thus, with the knowledge o Step 3, we show that, or t τ i aa + θ iaa, τ i+ aa τ i+1 aa, the sum process has the tendency to increase and exceed the lower bound n A +ϑ c n A γx 2i+2 ε 2 beore n aa t hits x i+1 ε. As beore, let Yn Σ be the associated discrete process to Σt. 73

86 II. Chapter: Survival o a recessive allele in a Mendelian diploid model [ Lemma II.11. For n N such that ϑ n τ i aa + θ iaa, τ i+ aa τ i+1 aa e V Kα/2] there exists a constant C 0 > 0 such that or all 0 i lnεk1/4 α/4 lnx P [ Y Σ n+1 = k 1 Y Σ n = k ] 1 2 C 0x 2i+2 ε 2 p 0 Σ. II.227 Proo. To show the lemma we use the results o the steps beore. It holds: P [ Y Σ n+1 = k 1 Y Σ n = k ] D ckk 1 + 2k N aat + D + ckk 1 + k N. II.228 aat We estimate the nominator D ckk γ x 2i+2 ε 2 k 1 K. II.229 This term assumes its maximum at k =vσ +. I we insert this bound, n A +ϑ/2 c n A γx 2i+2 ε 2, we can estimate D + c 2 2 n A + ϑ/2 γx 2i+2 ε 2 + γ x 2i+2 ε 2 + Ox 4i+4 ε 4 2 n A 2 n A ϑ 2 16 n 2 A + ε 2 + Ox 4i+4 ε 4. x2i+2 II.230 Thus, there exists a constant C 0 > 0 such that P [ Y Σ n+1 = k 1 Y Σ n = k ] 1 2 C 0x 2i+2 ε 2. II.231 Again we couple Y Σ n with Z l n via: 1 Z l 0 = Y Σ 0, II P[Z l n+1 = k + 1 Y Σ n > Z l n = k] = 1 p 0 Σ, II P[Z l n+1 = k 1 Y Σ n > Z l n = k] = p 0 Σ, II P[Z l n+1 = k 1 Y Σ n+1 = k 1, Y Σ n = Z l n = k] = 1, II P[Z l n+1 = k 1 Y Σ n+1 = k + 1, Y Σ n = Z l n = k] = p 0Σ P[Y Σ n+1 =k 1 Y Σ n =k] 1 P[Y Σ n+1 =k 1 Y Σ n =k], II P[Zn+1 l = k + 1 Yn+1 Σ = k + 1, Yn Σ = Zn l = k] = 1 p 0Σ P[Yn+1 Σ =k 1 Y n Σ=k] 1 P[Yn+1 Σ =k 1 Y. II.237 n Σ =k] Observe that by construction Zn l Yn Σ, a.s.. The marginal distribution o Zn l is the desired Markov chain with transition probabilities [ ] P Zn+1 l = k + 1 Zn l = k = 1 p 0 Σ, II.238 [ ] P Zn+1 l = k 1 Zn l = k = p 0 Σ, II

87 5. Proos o Theorem II.4 and the main theorems and invariant measure µn = n 1 k=1 n k= C 0x 2i+2 ε C 0x 2i+2 ε 2 = 2 + C 0x 2i+2 ε 2 n C 0x 2i+2 ε 2 n. II.240 Again we get a bound on the harmonic unction, or n A +ϑ c n A γx 2i ε 2 M Σ K 1/2+α/2 z < n A +ϑ/2 c n A γx 2i+2 ε 2, P zk [ T i Σ < T i+ Σ ] = KvΣ+ n=zk+1 KvΣ+ n=kvσ 3M Σ K 1/2+α/2 +1 n 1 1 2C0 x 2i+2 ε 2 1+2C 0 x 2i+2 ε 2 1 2C0 x 2i+2 ε 2 1+2C 0 x 2i+2 ε 2 n 1 exp CK α. II.241 Part 2: As in Step 6, to calculate an upper bound on the time which Σt need to increase rom n A +ϑ c n A γx 2i ε 2 M Σ K 1/2+α/2 to n A +ϑ/2 c n A γx 2i+2 ε 2, we estimate the number o jumps o the sum process and the duration o one single jump rom above on a given time interval. Proposition II.16. Let { θ i Σ in t 0 : Σt n A +ϑ/2 c n A γx 2i+2 ε 2 be the growth time o Σt on the time interval t Σ0 = n A +ϑ c n A γx 2i ε 2 M Σ K 1/2+α/2}, [ τ i aa + θ iaa, τ i+ aa τ i+1 aa exist inite, positive constants, C Σ l and M, such that or all 0 i lnεk1/4 α/4 lnx II.242 ]. Then there P [ θ i Σ > C Σ l ] exp MK 4α. II.243 Proo. Let Y Σ n be deined as in the step beore and let W k be i.i.d. random variables with P[W k = 1] = C 0x 2i+2 ε 2, P[W k = 1] = 1 2 C 0x 2i+2 ε 2 and E[W 1 ] = 2C 0 x 2i+2 ε 2, II.244 which record a birth or a death event o the lower process Zn. l We choose N = 4C 0, with x 2 C = +ϑ x2 +ϑ/2 c n A, and show that P [n N] 1 exp K α C 0 Cx 2 /2. We estimate rom above the time the process Σt needs to make one jump: CK λ Σ t = ΣtK + ΣtK[D + + cσt] + N aa t C λ K λ l Σ. II.245 As beore let τ m τ m 1 the time between two jumps o Σt and let Jm Σ are i.i.d. exponential random variables with parameter [ λ l Σ. As in Step 6, by applying the exponential Chebyshev ] inequality, we get that P CK m=1 J m Σ C > 2C 0 exp C x 2 C λ 2C 0 K and hence x 2 [ n ] C P Jm Σ > 2C 0 x 2 exp MK α. II.246 C λ m=1 75

88 II. Chapter: Survival o a recessive allele in a Mendelian diploid model Part 3: For the iteration we have to ensure that once Σt hits the upper bound n A +ϑ/2 c n A γx 2i+2 ε 2 it will stay close to it or an exponential time on the given time interval. This ensures that the sum process is larger than n A ϑ+ c n A γx 2i+2 ε 2 when n aa t hits x i+1 ε and the next iteration step can start. Proposition II.17. Assume that Σ0 = n A +ϑ/2 c n A γx 2i+2 ε 2. Let { ˆτ Σ α in t > 0 : Σt n A + +ϑ/2 c n A γx 2i+2 ε 2 ϑ 2c n A γx 2i+2 ε 2}. II.247 Then, or all 0 i lnεk1/4 α/4 lnx P[ˆτ α Σ < τ i+ aa τ i+1 aa e V Kα/2 ] = ok α. II.248 Proo. The proo is similar to Step 3. Again we deine the dierence process ˆX t between ΣtK and n A K +ϑ/2 c n A γx 2i+2 ε 2 K, which is a branching process with the same rates as Σt: ˆX t = ΣtK n A K + +ϑ/2 c n A γx 2i+2 ε 2 K, II.249 ˆT 0 X in{t 0 : ˆXt = 0}, II.250 ˆT X α,m Σ {t 0 : ˆXt ϑ 2c n A γx 2i+2 ε 2 K}. II.251 Let Ŷn be the discrete process associated to ˆX t, obtained as described in Step 1. [ Lemma II.12. For n N such that ϑ n τ i aa + θ iaa, τ i+ aa τ i+1 aa e V Kα/2], there exists a constant Ĉ0 > 0 such that or all 0 i lnεk1/4 α/4 lnx P[Ŷn+1 = k 1 Ŷn = k] 1 2 Ĉ0x 2i+2 ε 2 ˆp Σ. II.252 The proo is a re-run o Step 3 by using the rates II.104. Proposition II.17 is similar to Step 3. The rest o the proo o Final Step: Calculation o the Decay Time o n aa : The ollowing proves Theorem II.2 ii. Set σ lnεk1/2 α. II.253 lnx Observe that x σ ε K 1/2+α is just the value until which we can control the decay o n aa t. Thus, to calculate the time o the controlled decay o the aa population we iterate the system, described above, until i = σ 1. Observe that σ 1 j=0 Cu x j ε = Cux 1 x K1/2 α ε 1 C lx 1 x K1/2 α 76

89 5. Proos o Theorem II.4 and the main theorems ε 1 = σ 1 C l j=0 x j ε and that the θ jaa = τ j+1 aa τ j+ aa are independent random variables. Thus, or some constant M > M > 0, we get [ Cu x P 1 x = P C ux K 1/2 α ε 1 σ 1 1 x j=0 [ Cu P xε θ 1aA C l xε,..., C u x σ ε θ σaa C l x σ ε K 1/2 α ε 1 τ σ aa τ 0+ aa C lx K 1/2 α ε 1] 1 x 1 σ exp MK 2α 1 exp MK 2α. τ j+1 aa τ j+ aa C lx 1 x ] K 1/2 α ε 1 II.254 II.255 II.256 II.257 Remark II.5. A unction t = 1 t needs a time t 1 t 0 = K 1/2 α 1 ε to decrease rom t 0 = ε to t 1 = K 1/2+α. Compared to the decay-time o n aa we see that it is o the same order. Thus the stochastic process behaves as the dynamical system. Survival until Mutation: Now we prove Theorem II.3. We already know that there is 1 no mutation beore a time o order Kµ K c. Lemma II.1 and II.23. Since we have seen that the duration o the irst step is Oln K and the time needed or the second step is bounded, the let hand side o II.15 ensures that the irst two phases are ended beore the occurrence o a new mutation. Thus we get the irst statement o II.21. To justiy the second statement o II.21, we have to calculate an upper bound on the mutation time such that there are still enough aa individuals in the population, when the next mutation to a new allele occurs. We saw that we can control the process only until the aa population has decreased to K 1/2+α. Thus we have to veriy that the mutation time is smaller than O K 1/2 α, the time the process n aa needs to decrease to K 1/2+α. The mutation rate o the whole population is the sum o the mutation rates o each subpopulation. For t [τ 0+ aa, τ σ aa ] with high probability, the new mutation occurs in the AA population, because n aa t and n aa t are very small. Let n AA t p A t = n aat II.258 n aa t + n aa t + n AA t be the relative requency o A alleles in the population at time t. The mutation rate o the AA population is given by µ AA K = µ K p A tn AA tk. II.259 For t [τ 0+ aa, τ σ aa ] we know rom the results beore that n AA is in an ε-neighbourhood o its equilibrium, n A, and n aa, n aa ε. We can estimate µ AA K : µ AA K µ K n A OεK. II.260 From II.259 we get that the time o a new mutation is smaller or equal to 1 n A OεKµ K. Thus to ensure that we still have a alleles in the population, we have to choose µ K in such a way that 1 Kµ K K 1/2 α, II

90 II. Chapter: Survival o a recessive allele in a Mendelian diploid model since we can ensure the survival o n aa t until the time C lx 1 x K 1/2 α ε 1 c. II.254. Hence, the right hand side o II.15 gives us that a new mutation occurs beore the aa population died out. This inishes the proo o Theorem II.3. 78

91 III. Chapter The recovery o a recessive allele in a Mendelian diploid model Anton Bovier, Loren Coquille and Rebecca Neukirch Abstract We study the large population limit o a stochastic individual-based model which describes the time evolution o a diploid hermaphroditic population reproducing according to Mendelian rules. In [84] it is proved that sexual reproduction allows unit alleles to survive in individuals with mixed genotype much longer than they would in populations reproducing asexually. In the present paper we prove that this indeed opens the possibility that individuals with a pure genotype can reinvade in the population ater the appearance o urther mutations. We thus expose a rigorous description o a mechanism by which a recessive allele can re-emerge in a population. This can be seen as a statement o genetic robustness exhibited by diploid populations perorming sexual reproduction. We acknowledge inancial support rom the German Research Foundation DFG through the Hausdor Center or Mathematics, the Cluster o Excellence ImmunoSensation, and the Priority Programme SPP1590 Probabilistic Structures in Evolution. L.C. has been partially supported by the LabEx PERSYVAL-Lab ANR-11-LABX through the Exploratory Project CanDyPop and by the Swiss National Science Foundation through the grant No. P300P2_ We would like to thank Pierre Collet and Vincent Beara or their help on the theory o dynamical systems and ruitul discussions. 1. Introduction In population genetics, the study o Mendelian diploid models o ixed population size began more than a century ago see e.g. [12,23,32,34,45,46,82,103,107], while their counterparts o variable population size models were studied in the context o adaptive dynamics rom 1999 onwards [59]. The approach o adaptive dynamics is to introduce competition kernels to regulate the population size instead o maintaining it constant, see [52, 72, 76]. Stochastic individual-based versions o these models appeared in the 1990s, see [14 17, 28, 36]. They assume single events o reproduction, mutation, natural death, and death by competition happen at random times to each individual in the population. An important and interesting eature o these models is that dierent limiting processes on dierent time-scales appear as the 79

92 III. Chapter: The recovery o a recessive allele in a Mendelian diploid model carrying capacity tends to ininity while mutation rates and mutation step-size tend to zero see [2,14,17,28,75]. One o the major results in this context is the convergence o a properly rescaled process to the so called Trait Substitution Sequence TSS process, which describes the evolution o a monomorphic population as a jump process between monomorphic equilibria. More generally, Champagnat and Méléard [17] obtained the convergence to a Polymorphic Evolution Sequence PES, where jumps occur between equilibria that may include populations that have multiple coexisting phenotypes. The appearance o co-existing phenotypes is, however, exceptional and happens only at so-called evolutionary singularities. From a biological point o view, this is somewhat unsatisactory, as it apparently ails to explain the biodiversity seen in real biological systems. Most o the models considered in this context assume haploid populations with a-sexual reproduction. Exceptions are the paper [18] by Collet, Méléard and Metz rom 2013 and a series o papers by Coron and co-authors [19 21] ollowing it. In [18], the Trait Substitution Sequence is derived in a Mendelian diploid model under the assumption that the itter mutant allele and the resident allele are co-dominant. The main reason why both in haploid models and in the model considered in [18] the evolution along monomorphic populations is typical is that the time scales or the ixation o a new trait and the extinction o the resident trait are the same both o order ln K unless some very special ine-tuning o parameters occurs that allows or co-existence. This precludes at least in the rare mutation scenarios considered that an initially less it trait survives long enough until ater possibly several new mutations occurred that might create a situation where this trait may become it again and recover. In a ollow-up paper to [18], two o the present authors [84], it was shown that, i instead one assumes that the resident allele is recessive, the time to extinction o this allele is dramatically increased. This will be discussed in detail in Section 1.2 and paves the way or the appearance o a richer limiting process. The general ramework in [18] and [84] is the ollowing. Each individual is characterised by a reproduction and death rate which depend on a phenotypic trait determined by its genotype, which here is determined by two alleles e.g. A and a on one single locus. The evolution o the trait distribution o the three genotypes aa, aa and AA is studied under the action o 1 heredity, which transmits traits to new osprings according to Mendelian rules, 2 mutation, which produces variations in the trait values in the population onto which selection is acting, and 3 o competition or resources between individuals. The paper [84] proves that sexual reproduction allows unit alleles to survive in individuals with mixed genotype much longer than they would in populations reproducing asexually. This opens the possibility that while this allele is still alive in the population, the appearance o new mutants alters the itness landscape in such a way that is avourable or this allele and allow it to reinvade in the population, leading to a new equilibrium with co-existing phenotypes. The goal o this paper is to rigorously prove that such a scenario indeed occurs under airly natural assumptions. Recently, Billiard and Smadi [7] considered related questions or haploid individuals perorming clonal reproduction. The authors show that a deleterious allele can reinvade ater a new mutation, but the range o parameters allowing this behaviour is though very small The stochastic model The individual-based microscopic Mendelian diploid model is a non-linear birth-and-death process. We consider a model or a population o a inite number o hermaphroditic individuals which 80

93 1. Introduction reproduce sexually. Each individual i is characterised by two alleles, u i 1 ui 2, taken rom some allele space U R. These two alleles deine the genotype o the individual i. We suppress parental eects, which means that we identiy individuals with genotype u 1 u 2 and u 2 u 1. Each individual has a Mendelian reproduction rate with possible mutations and a natural death rate. Moreover, there is an additional death rate due to ecological competition with the other individuals in the population. Let u1 u 2 R + the per capita birth rate ertility o an individual with genotype u 1 u 2, D u1 u 2 R + the per capita natural death rate o an individual with genotype u 1 u 2, K N the carrying capacity, a parameter which scales the population size, c u1 u 2,v 1 v 2 K R + the competition eect elt by an individual with genotype u 1 u 2 rom an individual o genotype v 1 v 2, R u1 u 2 v 1 v 2 {0, 1} the reproductive compatibility o the genotype v 1 v 2 with u 1 u 2, µ K R + the mutation probability per birth event. Here it is independent o the genotype, mu, dh mutation law o a mutant allelic trait u + h U, born rom an individual with allelic trait u. Scaling the competition unction c down by a actor 1/K amounts to scaling the population size to order K. We are interested in asymptotic results when K is large. We assume rare mutation, i.e. µ K 1. I a mutation occurs at a birth event, only one allele changes rom u to u + h where h is a random variable with law mu, dh. At any time t, there is a inite number, N t, o individuals, each with genotype in U 2. We denote by u 1 1 tu1 2 t,..., unt 1 tunt 2 t the genotypes o the population at time t. The population, ν t, at time t is represented by the rescaled sum o Dirac measures on U 2, ν t = 1 K N t δ u i 1 tu i t. 2 i=1 III.1 Formally, ν t takes values in the set o re-scaled point measures { } M K 1 n = δ K u i 1 u i n 0, u 1 2 1u 1 2,..., u n 1 u n 2 U 2, III.2 i=1 on U 2, equipped with the vague topology. Deine ν, g as the integral o the measurable unction g : U 2 R with respect to the measure ν M K. Then ν t, 1 = Nt K and or any u 1u 2 U 2, the positive number ν t, 1 u1 u 2 is called the density at time t o the genotype u 1 u 2. The generator o the process is deined as in [18]: irst we deine, or the genotypes u 1 u 2, v 1 v 2 and a point measure ν, the Mendelian reproduction operator: A u1 u 2,v 1 v 2 F ν [ F = 1 4 ν + δ u 1 v 1 K + F ν + δ u 1 v 2 + F ν + δ u 2 v 1 + F ν + δ ] u 2 v 2 F ν, K K K III.3 81

94 III. Chapter: The recovery o a recessive allele in a Mendelian diploid model and the Mendelian reproduction-cum-mutation operator: M u1 u 2,v 1 v 2 F ν = 1 [ F ν + δ u 1 +h,v 1 8 K R + F ν + δ u 1 +h,v 2 K mu 1, h + F ν + δ u 2 +h,v 1 + F ν + δ u 2 +h,v 2 mu 2, h K K + F ν + δ u 1,v 1 +h + F ν + δ u 2,v 1 +h mv 1, h K K + F ν + δ u 1,v 2 +h + F ν + δ ] u 2,v 2 +h mv 2, h K K dh F ν. III.4 The process ν t t 0 is then a M K -valued Markov process with generator L K, given or any bounded measurable unction F : M K R by: L K F ν = D u1 u 2 + c u1 u 2,v 1 v 2 νdv 1 v 2 F ν δ u 1 u 2 F ν Kνdu 1 u 2 U 2 U 2 K v1 v + 1 µ K 2 R u1 u 2 v 1 v 2 u1 u 2 A u1 u U 2 U 2 νr u1 u 2, 2,v 1 v 2 F ννdv 1 v 2 Kνdu 1 u 2 v1 v + µ K 2 R u1 u 2 v 1 v 2 u1 u 2 M u1 u U 2 U 2 νr u1 u 2, 2,v 1 v 2 F ννdv 1 v 2 Kνdu 1 u 2. III.5 The irst non-linear term describes the competition between individuals. The second and last non-linear terms describe the birth with and without mutation. There, v1 v 2 R u1 u 2 v 1 v 2 u1 u 2 K νr u1 u 2, is the reproduction rate o an individual with genotype u 1 u 2 with an individual with genotype v 1 v 2. Note that νr u1 u 2 is the population restricted to the pool o potential partners o an individual o genotype u 1 u 2. For all u 1 u 2, v 1 v 2 U 2, we make the ollowing Assumptions A: A1 The unctions, D and c are measurable and bounded, which means that there exists, D, c < such that 0 u1 u 2, 0 D u1 u 2 D and 0 c u1 u 2,v 1 v 2 c. III.6 A2 u1 u 2 D u1 u 2 > 0 and there exists c > 0 such that c c u1 u 2,v 1 v 2. A3 There exists a unction, m : R R +, such that mhdh < and mu, h mh or any u U and h R. For ixed K, under the Assumptions A1+A3 and assuming that E ν 0, 1 <, Fournier and Méléard [36] have shown existence and uniqueness in law o a process with ininitesimal generator L K. For K, under mild restrictive assumptions, they prove the convergence o the process ν K in the space DR +, M K o càdlàg unctions rom R + to M K, to a deterministic process, which is the solution to a non-linear integro-dierential equation. Assumption A2 ensures that the population does not tend to ininity in inite time or becomes extinct too ast. 82

95 1. Introduction 1.2. Previous works Consider the process starting with a monomorphic aa population, with one additional mutant individual o genotype aa. Assume that the phenotype dierence between the mutant and the resident population is small. The phenotype dierence is assumed to be a slightly smaller death rate compared to the resident population, namely: D aa = D, D aa = D. III.7 or some small enough > 0. The mutation probability or an individual with genotype u 1 u 2 is 1 given by µ K. Hence, the time until the next mutation in the whole population is o order Kµ K. Now assume that the demographic parameters introduced in Section 1.1 depend continuously on the phenotype. In particular, they are the same or individuals bearing the same phenotype. In [18] it is proved that i the two alleles a and A are co-dominant and i the allele A is slightly itter than the allele a, namely D aa = D, D aa = D, D AA = D 2, III.8 then in the limit o large population and rare mutations ln K 1 µ K K ev K or some V > 0, the suitably time-rescaled process converges to the TSS model o adaptive dynamics, essentially as shown in [14] in the haploid case. In particular, the genotypes containing the unit allele a decay exponentially ast ater the invasion o AA see Figure III.1. I in place o co-dominance we assume, as in [84], that the ittest phenotype A is dominant, namely D aa = D, D aa = D, D AA = D, III.9 then this has a dramatic eect on the evolution o the population and, in particular, leads to a much prolonged survival o the unit phenotype aa. Indeed, it was know or some time see e.g. [82] that in this case the unique stable ixpoint 0, 0, n AA corresponding to a monomorphic AA population is degenerate, i.e. its Jacobian matrix has zero-eigenvalue. This implies that in the deterministic system, the aa and aa populations decay in time only polynomially ast to zero, namely like 1/t 2 and 1/t, respectively. This is in contrast to the exponential decay in the co-dominant scenario see Figure III.1. In [84] it was shown that the deterministic system remains a good approximation o the stochastic system as long as the size o the aa population remains much larger than K 1/2 and thereore that the a allele survives or a time o order at least K 1/2 α, or any α > 0 1. Note that this statement is a non trivial act, since it is not a consequence o the law o large numbers, because the time window diverges as K grows. In summary, the unit recessive a allele survives in the population much longer due to the slow decay o the aa population. It is argued in [84] that i we choose the mutation time scale in such a way that there remain enough a alleles in the population when a new mutation occurs, i.e. ln K 1 µ K K K1/2 α as K, or some α > 0, III.10 and i the new mutant can coexist with the unit aa individuals, then the aa population can potentially recover. This is the starting point o the present paper. 1 In [84] only state that survival occurs up to time K 1/4 α. However, taking into account that it is really only the survival o the aa population that needs to be ensured, one can easily improve this to K 1/2 α 83

96 III. Chapter: The recovery o a recessive allele in a Mendelian diploid model Figure III.1.: Evolution o the model rom a resident aa population at equilibrium with a small amount o mutant aa, and when the alleles a and A are co-dominant let or when the mutant phenotype A is dominant right Goal o the paper The goal o this paper is to show that under reasonable hypothesis, the prolonged survival o the a allele ater the invasion o the A allele can indeed lead to a recovery o the aa-type. To do this, we assume that there will occur a new mutant allele, B, that on the one hand has a higher itness than the AA-phenotype but that or simplicity has no competition with the aa-type. The possible genotypes ater this mutation are aa, aa, AA, ab, AB, and BB, so that even or the deterministic system we have now to deal with a 6-dimensional dynamical system whose analysis i ar rom simple. Under the assumption o dominance o the ittest phenotype, and mutation rate satisying III.10, we consider the model described in Section 1.1 starting at the time o the second mutation, that is with probability converging to 1 as K the AA population being close to its equilibrium and the aa population having decreased to a size o order Kµ K, while the aa population is o the order o the square o the aa population. We assume that there just occurred a mutation to a itter and most dominant allele B: we thus start with a quantity 1 K o genotype AB. We will start with a population where AA is close to its equilibrium, the populations o aa and aa are already small o order ε 2 and ε, and by mutation a single individual o genotype AB appears. By using well known techniques [14, 17, 18], we know that the AB population behaves as a super-critical branching process and reaches the level ε with positive probability in a time o order ln K, without perturbing the 3-system aa, aa, AA. We see in numerical solutions to the deterministic system that a reduced ertility together with a reduced competition between a and B phenotypes constitutes a suicient condition or the recovery o the aa population. For simplicity and in order to prove rigorous results, we suppose that there can be no reproduction between individuals o phenotypes a and B, nor competition between them, and we reduce the number o remaining parameters as much as possible see Section 2. We study the deterministic system which corresponds to the large population limit o the stochastic counterpart, and we show that or an initial quantity ε o aa, ε 2 o aa and ε 3 o AB the system converges to a ixed point denoted by p ab consisting o the two coexisting populations aa and BB. I no urther assumptions are made, we will show that the number o individuals bearing an a allele decreases to level ε 1+ /1 where is deined in III.7 beore aa grows and stabilises at order 1. 84

97 2. Model setup α I < 1 2α, this control on the a allele is in principle suicient in order or the stochastic system to exhibit the recovery o aa with positive probability in the large population limit. Indeed, i the mutation time is o order K 1 2 α, then the initial amount o aa and aa genotypes is close to the typical luctuations o those populations. Following the heuristics o [84] although the sixdimensional stochastic process is surely much more tedious to study, the deterministic system should constitute a good approximation o the process i the typical luctuations o populations containing an a allele do not bring them to extinction. I < this ensures that the population α 1 2α containing an a allele is not alling below order K 1/2 at any time. In order to go deeper and control the speed o recovery o the aa population, we look or a parameter regime which ensures that the aa population always grows ater the invasion o B. Ensuring this lower bound on aa is not trivial at all, and the solution we ound is to introduce an additional parameter η, which lowers the competition between the aa and BB populations, compared to the one between AA and BB. Note that the competition does not depend only on the phenotype, and can be interpreted as a reinement o a phenotypic competition or resources: the strength or ability to get resources o an individual not only depends on its phenotype but also on the dominance o its genotype. We show that or η larger than some positive value o order, the aa population always grows ater the invasion o B. The time o convergence to the coexistence ixed point is thus lowered, see Figure III.5. Moreover, we point out the existence o a biurcation: or η larger than some threshold, the co-existence ixed point p ab becomes unstable and the system converges to another ixed point where all populations coexist. Our contribution is a rigorous description o a mechanism by which a recessive allele can reemerge in a population. This can be seen as a statement o genetic robustness exhibited by diploid populations perorming sexual reproduction. The structure o the paper is the ollowing. In Section 2 we describe our assumptions on the parameters o the model, and compute the large population limit; in Section 3 we present our results on the evolution o the deterministic system towards the co-existence ixed point p ab, and we give a heuristic o the proo. Section 4 is dedicated to the proo o these results. The closing Section 5 contain a heuristic considerations and numerical simulations o the model with relaxed assumption on the parameters. Notation. We write x = Θy whenever x = Oy and y = Ox as ε Model setup Let G = {aa, aa, AA, ab, AB, BB} be the genotype space. Let n i t be the number o individuals with genotype i G in the population at time t and set n K i t 1 K n it. Deinition III.1. The equilibrium size o a monomorphic uu population, u {a, A, B}, is the ixed point o a 1-dimensional Lotka-Volterra equation and is given by Deinition III.2. For u, v {a, A, B}, we call n u = uu D uu c uu,uu. III.11 S uv,uu = uv D uv c uv,uu n u, III.12 the invasion itness o a mutant uv in a resident uu population. 85

98 III. Chapter: The recovery o a recessive allele in a Mendelian diploid model Figure III.2.: Simulation o the stochastic system or = 6, D = 0.7, = 0.1, c = 1, η = 0.02, ε = and K = We take the phenotypic viewpoint and assume that the B allele is the most dominant one. That means the ascending order o dominance in the Mendelian sense is given by a < A < B, i.e. 1. phenotype a consists o the genotype aa, 2. phenotype A consists o the genotypes aa, AA, 3. phenotype B consists o the genotypes ab, AB, BB. For simplicity, we assume that the ertilities are the same or all genotypes, and that natural death rates are the same within the three dierent phenotypes. Moreover, we assume that there can be no reproduction between a and B phenotypes. To sumarize, we make the ollowing Assumptions B on the rates: B1 Fertilities. For all i G, and some > 0 i, III.13 B2 Natural death rates. The dierence in itness o the three phenotypes is realised by choosing a slightly higher natural death-rate o the a-phenotype and a slightly lower death-rate or the B-phenotype. For some 0 < < D, Daa = D +, DAA DaA = D, DaB DAB DBB = D. 86 III.14 III.15 III.16

99 2. Model setup B3 Competition rates. We require that phenotypes a and B do not compete with each other. Moreover, we introduce a parameter η 0 which lowers the competition between BB and aa. For some 0 η < c, c i,j {i,j} G G =: aa aa AA ab AB BB aa c c c aa c c c c c c η AA c c c c c c ab 0 c c c c c AB 0 c c c c c BB 0 c η c c c c A biological interpretation or this kind o competition could be that it is coded in the alleles which ood an individual with a given genotype preers. Since an AB individual shares one B allele with a BB individual, they compete stronger or the same ood than AA with BB since those have completely dierent alleles. B4 Reproductive compatibility. We require that phenotypes a and B do not reproduce with each other, aa aa AA ab AB BB aa aa R i j {i,j} G G AA ab AB BB Observe that, under Assumptions B, S AB,AA = D c n AA = D + c D c S aa,bb = D. =, III.17 III.18 Thereore, the mutant AB has a positive invasion itness in the population AA, as well as aa in the BB population due to the absence o competition between them Birth rates We assume that there is no recombination between phenotypes a and B. Thus, 1. the pool o possible partners or the phenotype a consists o phenotypes a and A; the total population o this pool is denoted by Σ 3 := n aa + n aa + n AA, III the pool o possible partners or the phenotype A consists o the three phenotypes a, A, and B; the total population o this pool is denoted by Σ 6 := n aa + n aa + n AA + n ab + n AB + n BB, III.20 87

100 III. Chapter: The recovery o a recessive allele in a Mendelian diploid model 3. the pool o possible partners or the phenotype B consists o phenotypes A and B; the total population o this pool is denoted by Σ 5 := n aa + n AA + n ab + n AB + n BB. III.21 Computing the reproduction rates with the Mendelian rules as described in III.5 leads to the ollowing time-dependant birth-rates b i = b i nt: b aa = n aa naa n aa + n aa + n aa + n AA 1 2 n 1 ab 2 n aa n ab n aa + n AA + n ab + n AB + n BB n aa naa n aa n ab, n aa + n aa + n AA + n ab + n AB + n BB b aa = n 1 aa 2 n aa + n AA n aa + n aa + n AA n 1 aa 2 n ab n AB n ab n AA + n AB n aa + n AA + n ab + n AB + n BB III n aa + n AA naa + n aa n ab n aan AB, III.23 n aa + n aa + n AA + n ab + n AB + n BB 1 2 b AA = n 1 AB 2 n aa + n AA n 1 AB 2 + n aa + n 1 AA 2 n aa + n AA n AB, n aa + n AA + n ab + n AB + n BB n aa + n aa + n AA + n ab + n AB + n BB III b ab = n aa + n 1 ab 2 n ab n 1 AB + n BB 2 + n 1 aa 2 n ab n AB + n BB, n aa + n AA + n ab + n AB + n BB n aa + n aa + n AA + n ab + n AB + n BB III.25 b AB = 1 2 n aa + n AA + n AB 1 2 n ab n AB + n BB n aa + n AA + n ab + n AB + n BB 1 2 n aa + n 1 AA 2 n ab n AB + n BB 1 4 b BB = n ab + n AB + 2n BB 2. n aa + n AA + n ab + n AB + n BB +, n aa + n aa + n AA + n ab + n AB + n BB III.26 III Death rates The death rates are the sum o the natural death and the competition: d aa = n aa D + + cn aa + n aa + n AA, d aa = n aa D + cn aa + n aa + n AA + n ab + n AB + c ηn BB, d AA = n AA D + cn aa + n aa + n AA + n ab + n AB + n BB, d ab = n ab D + cn aa + n AA + n ab + n AB + n BB, d AB = n AB D + cn aa + n AA + n ab + n AB + n BB, III.28 III.29 III.30 III.31 III.32 88

101 2. Model setup d BB = n BB D + c ηn aa + cn AA + n ab + n AB + n BB. III Large population limit By [36] or [18], or large populations, the behaviour o the stochastic process is close to the solution o a deterministic equation. Proposition III.1 Generalisation o Proposition 3.2 in [18]. Let T > 0 and C R 6 + be a compact set. Assume that the initial condition n K 0 = 1 K n aa0, n aa 0, n AA 0, n ab 0, n AB 0, n BB 0 converges almost surely to a deterministic vector x 0 = x 0 1, x0 2, x0 3, x0 4, x0 5, x0 6 C, as K. Let ñt, x 0 denote the solution to rm i.e. ṅt = bnt dnt F nt, III.34 ṅ i t = b i nt D i + c i,j n j t n i t, or all i G, III.35 j G with initial condition x 0, where b i i G and d i i G are given in III.22-III.27 and III.28- III.33. Then, or all T > 0, or all i G. lim sup K t [0,T ] 2.4. Initial condition n K i t ñ i t, x 0 = 0, a.s., III.36 Fix ε > 0 suiciently small. For the results below, we will consider the dynamical system III.34 starting with the initial condition: n A n AA 0 n A Θε, n aa 0 = ε, n aa 0 = Θε 2, n AB 0 = ε 3, n BB 0 = 0, n ab 0 = 0. III.37 III.38 III.39 III.40 III.41 III.42 Remark III.1. In all the igures below, the choice o parameters is the ollowing: = 6, D = 0.7, = 0.1, c = 1, ε = 0.01, and the parameter η is speciied on each picture. 89

102 III. Chapter: The recovery o a recessive allele in a Mendelian diploid model Figure III.3.: General qualitative behaviour o {n i t, i G} and projection o the dynamical system on the coordinates aa, AA and BB. The re-invasion o the aa population happens sooner and sooner as η grows η = 0.02 or both pictures. 3. Results We are working with a 6-dimensional dynamical system, and computing all the ixed points analytically is impossible or a general choice o the parameters. We can, however, compute those which are relevant or our study. We will call p A resp. p B the ixed points corresponding to the monomorphic AA resp. BB population at equilibrium, and p ab the ixed point corresponding to the coexisting aa and BB populations. Setting the relevant populations to 0 and solving ṅt = 0, we get: p A = 0, 0, n A, 0, 0, 0, p B = 0, 0, 0, 0, 0, n B, p ab = n a, 0, 0, 0, 0, n B, III.43 III.44 III.45 where n a = D c, n A = D c, and n B = D+ c. Note that the BB equilibrium population is the same in p B and p ab. This is due to the non-interaction between phenotypes a and B. Our general result is that starting with initial conditions III.37-III.42, that is close to p A with small coordinates in directions aa, aa and AB, and under minimal assumptions on the parameters, the system gets very close to p B beore inally converging to p ab, see Figure III.3. Theorem III.1. Consider the dynamical system III.34 started with initial conditions III.37-III.42. Suppose the ollowing Assumptions C on the parameters hold: C1 suiciently small, C2 suiciently large, C3 0 η < c/2. 90