Transmissible cancers and the evolution of sex under the Red Queen hypothesis

The predominance of sexual reproduction in eukaryotes remains paradoxical in evolutionary theory. Of the hypotheses proposed to resolve this paradox, the 'Red Queen hypothesis' emphasises the potential of antagonistic interactions to cause fluctuating selection, which favours the evolution and maintenance of sex. Whereas empirical and theoretical developments have focused on host-parasite interactions, the premises of the Red Queen theory apply equally well to any type of antagonistic interactions. Recently, it has been suggested that early multicellular organisms with basic anticancer defences were presumably plagued by antagonistic interactions with transmissible cancers and that this could have played a pivotal role in the evolution of sex. Here, we dissect this argument using a population genetic model. One fundamental aspect distinguishing transmissible cancers from other parasites is the continual production of cancerous cell lines from hosts' own tissues. We show that this influx dampens fluctuating selection and therefore makes the evolution of sex more difficult than in standard Red Queen models. Although coevolutionary cycling can remain sufficient to select for sex under some parameter regions of our model, we show that the size of those regions shrinks once we account for epidemiological constraints. Altogether, our results suggest that horizontal transmission of cancerous cells is unlikely to cause fluctuating selection favouring sexual reproduction. Nonetheless, we confirm that vertical transmission of cancerous cells can promote the evolution of sex through a separate mechanism, known as similarity selection, that does not depend on coevolutionary fluctuations.