Multimodal pattern formation in phenotype distributions of sexual populations

During bouts of evolutionary diversification, such as adaptive radiations, the emerging species cluster around different locations in phenotype space. How such multimodal patterns in phenotype space can emerge from a single ancestral species is a fundamental question in biology. Frequency-dependent competition is one potential mechanism for such pattern formation, as has previously been shown in models based on the theory of adaptive dynamics. Here, we demonstrate that also in models similar to those used in quantitative genetics, phenotype distributions can split into multiple modes under the force of frequency-dependent competition. In sexual populations, this requires assortative mating, and we show that the multimodal splitting of initially unimodal distributions occurs over a range of assortment parameters. In addition, assortative mating can be favoured evolutionarily even if it incurs costs, because it provides a means of alleviating the effects of frequency dependence. Our results reveal that models at both ends of the spectrum between essentially monomorphic (adaptive dynamics) and fully polymorphic (quantitative genetics) yield similar results. This underscores that frequency-dependent selection is a strong agent of pattern formation in phenotype distributions, potentially resulting in adaptive speciation.