Integrating resource defence theory with a neural nonapeptide pathway to explain territory-based mating systems

The ultimate-level factors that drive the evolution of mating systems have been well studied, but an evolutionarily conserved neural mechanism involved in shaping behaviour and social organization across species has remained elusive. Here, we review studies that have investigated the role of neural arginine vasopressin (AVP), vasotocin (AVT), and their receptor V1a in mediating variation in territorial behaviour. First, we discuss how aggression and territoriality are a function of population density in an inverted-U relationship according to resource defence theory, and how territoriality influences some mating systems. Next, we find that neural AVP, AVT, and V1a expression, especially in one particular neural circuit involving the lateral septum of the forebrain, are associated with territorial behaviour in males of diverse species, most likely due to their role in enhancing social cognition. Then we review studies that examined multiple species and find that neural AVP, AVT, and V1a expression is associated with territory size in mammals and fishes. Because territoriality plays an important role in shaping mating systems in many species, we present the idea that neural AVP, AVT, and V1a expression that is selected to mediate territory size may also influence the evolution of different mating systems. Future research that interprets proximate-level neuro-molecular mechanisms in the context of ultimate-level ecological theory may provide deep insight into the brain-behaviour relationships that underlie the diversity of social organization and mating systems seen across the animal kingdom.