From Individual to Group Territoriality: Competitive Environments Promote the Evolution of Sociality

In many animal species individuals compete for the sole possession of a breeding territory, whereas in other species communal territories are shared among same-sex conspecifics. Under what conditions does natural selection favor the evolution of individual territoriality, and under what conditions does it favor the evolution of sociality? We develop a self-consistent game-theoretic model that allows for feedbacks between evolutionary and population dynamics. In this model, nonresident floaters can chose between three strategies: they can wait for a territory vacancy to arise, they can try to forcefully take over an already-occupied territory, or they can share a territory with an established resident. We show that competitive environments initially favor the evolution of an aggressive (territorial) strategy. Yet as competition increases further, a shift occurs from aggressive to social strategies. Moreover, territory owners (residents) respond to the behavior of floaters, such that a feedback occurs in which residents adjust their degree of tolerance to the level of floater aggression and vice versa. This feedback resembles the dynamics of a biological market and eventually leads to the coexistence of aggressive and social floater strategies in the population. Such mixed equilibria commonly occur in nature.