Fixation Probabilities of Mutant Alleles in an Ecological Context

For decades, population geneticists have relied upon a formula derived by Malécot and by Kimura to estimate the fixation probability of mutant alleles $\left(\varphi \right)$ . Among other things, this formula leads to the conclusion that in sufficiently large populations $\varphi$ asymptotically approaches $2s\left(N_e/N\right)$ (for small $s$ ), where $s$ is the relative selective advantage of the mutant allele, and $N_e$ and $N$ respectively denote the effective and absolute (haploid) population sizes. In contrast, in sufficiently small populations, $\varphi =1/N$ , with the two domains of behavior being separated at the point where $s\simeq 1/\left(2N_e\right)$ . These results hold when $s,N$ and $N_e$ remain constant during the fixation process, but require modification when populations are changing in size. Here, we show that if there are ecological effects associated with competing alleles, such that the genetic composition of the population influences the total population size, the fixation probability of a beneficial allele can substantially deviate from the levels suggested above, even in the domain of effective neutrality (i.e., $\left|s\right|\lesssim 1/\left(2N_e\right)$ ). We obtain analytical results for a variety of frequency-dependent functions for the overall population size, and show that the overall effect is largely a function of the response at low mutant-allele frequency. We also derive expressions for times to fixation and for fixation probabilities of deleterious alleles, and show how our results relate to prior work for the situation in which population sizes change in frequency-independent manners.