The influence of genetic drift on the formation and stability of polymorphisms arising from negative frequency-dependent selection

We consider the simplest form of negative frequency-dependent selection in a biallelic haploid population, where the selection coefficient of a mutant allele is a linear function of the allele's frequency, and changes from positive to negative as the frequency is increased. In an effectively infinite population this behaviour leads to a stable polymorphism. We present a theoretical investigation of what occurs in a finite population, where a long-lived polymorphism may be formed, but which fluctuates and ultimately disappears due to random genetic drift. We model the dynamics as a branching process and explicitly take into account differences between the census population size and the effective population size, which play different roles in the dynamics. We characterise the behaviour of the population in terms of three distinct timescales associated with: (i) early loss of mutant alleles, (ii) achievement of the long-lived polymorphism, (iii) disappearance of the polymorphism. Timescales (i) and (iii) depend on the effective population size and are, as a consequence, affected by random genetic drift, while timescale (ii) depends primarily on the census size and is relatively insensitive to genetic drift. Analysis and simulations of the branching process clarify the different influences of the census and effective population sizes. One substantial quantitative difference, between populations where the effective and census population sizes coincide and where they differ, lies in the number of mutant alleles in the long-lived polymorphism. This number is approximately proportional to the census size. Thus assuming the census size equals a much smaller effective population size predicts a much smaller number of mutants in the long-lived polymorphism.