Evolution of mating types in finite populations: The precarious advantage of being rare

Sexually reproducing populations with self-incompatibility bear the cost of limiting potential mates to individuals of a different type. Rare mating types escape this cost since they are unlikely to encounter incompatible partners, leading to the deterministic prediction of continuous invasion by new mutants and an ever-increasing number of types. However, rare types are also at an increased risk of being lost by random drift. Calculating the number of mating types that a population can maintain requires consideration of both the deterministic advantages and the stochastic risks. By comparing the relative importance of selection and drift, we show that a population of size N can maintain a maximum of approximately N^1/3 mating types for intermediate population sizes, whereas for large N, we derive a formal estimate. Although the number of mating types in a population is quite stable, the rare-type advantage promotes turnover of types. We derive explicit formulas for both the invasion and turnover probabilities in finite populations.