The three-species problem: Incorporating competitive asymmetry and intransitivity in modern coexistence theory

While natural communities can contain hundreds of species, modern coexistence theory focuses primarily on species pairs. Alternatively, the structural stability approach considers the feasibility of equilibria, gaining scalability to larger communities but sacrificing information about dynamic stability. Three-species competitive communities are a bridge to more-diverse communities. They display novel phenomena while remaining amenable to mathematical analysis, but remain incompletely understood. Here, we combine these approaches to identify the key quantities that determine three-species competition outcomes. We show that pairwise niche overlap and fitness differences are insufficient to completely characterize competitive outcomes, which requires a strictly triplet-wise quantity: cyclic asymmetry, which underlies intransitivity. Low pairwise niche overlap stabilizes the triplet, while high fitness differences promote competitive exclusion. The effect of cyclic asymmetry on stability is complex and depends on pairwise niche overlap. In summary, we elucidate how pairwise niche overlap, fitness differences and cyclic asymmetry determine three-species competition outcomes.