Oscillatory evolutionarily stable state and limit cycle in replicator dynamics

The idea of the evolutionarily stable state (ESS) of a population is a cornerstone of evolutionary game theory; moreover, it coincides with the game-theoretic concept of Nash equilibrium. Such a state corresponds to a strategy adopted by the population such that a rare mutant strategy cannot invade the population. In parallel, the dynamical formulation of evolutionary game theory-particularly through replicator dynamics embodying the tenet of survival of the fittest-provides a framework for modeling frequency-dependent selection over time. While it is well known that an ESS corresponds to the stable fixed point in replicator dynamics, the evolutionary game-theoretic characterization of limit cycles is unknown. Here we fill this lacuna by defining an oscillatory ESS (OESS), which we prove to be a stable limit cycle. We also determine the conditions for the existence and the phase-space locations of OESSs.