The impact of random frequency-dependent mutations on the average population fitness

In addition to selection, the process of evolution is accompanied by stochastic effects, such as changing environmental conditions, genetic drift and mutations. Commonly it is believed that without genetic drift, advantageous mutations quickly fixate in a halpoid population due to strong selection and lead to a continuous increase of the average fitness. This conclusion is based on the assumption of constant fitness. However, for frequency dependent fitness, where the fitness of an individual depends on the interactions with other individuals in the population, this does not hold.

We propose a mathematical model that allows to understand the consequences of random frequency dependent mutations on the dynamics of an infinite large population. The frequencies of different types change according to the replicator equations and the fitness of a mutant is random and frequency dependent. To capture the interactions of different types, we employ a payoff matrix of variable size and thus are able to accommodate an arbitrary number of mutations. We assume that at most one mutant type arises at a time. The payoff entries to describe the mutant type are random variables obeying a probability distribution which is related to the fitness of the parent type.

We show that a random mutant can decrease the average fitness under frequency dependent selection, based on analytical results for two types and simulations for n types. Interestingly, in the case of at most two types the probabilities to increase or decrease the average fitness are independent of the concrete probability density function. Instead, they only depend on the probability that the payoff entries of the mutant are larger than the payoff entries of the parent type.