Long-term stability from fixation probabilities in finite populations: new perspectives for ESS theory

For mixed strategies in finite populations, long-term stability is defined with respect to the probability of fixation of a mutant. Under weak selection, necessary and sufficient conditions are obtained using a diffusion approximation of the Wright-Fisher model or exact solutions for the Moran model. These differ from the usual ESS conditions if the strategies affect fertility instead of viability, leading to a game matrix depending on the population size, or if the mutant mixed strategy uses a new pure strategy. In this case, the mutant deviation must not exceed some threshold value depending on the population size. In a diploid population, long-term stability may not occur unless there is partial dominance. In the case of sex allocation, continuous stability of an even sex ratio is ascertained. If sex allocation is random, an evolutionary decrease of the variance is predicted.