Transitions between male and female heterogamety caused by sex-antagonistic selection

Many animal taxa show frequent and rapid transitions between male heterogamety (XY) and female heterogamety (ZW). We develop a model showing how these transitions can be driven by sex-antagonistic selection. Sex-antagonistic selection acting on loci linked to a new sex-determination mutation can cause it to invade, but when acting on loci linked to the ancestral sex-determination gene will inhibit an invasion. The strengths of the consequent indirect selection on the old and new sex-determination loci are mediated by the strengths of sex-antagonistic selection, linkage between the sex-antagonistic and sex-determination genes, and the amount of genetic variation. Sex-antagonistic loci that are tightly linked to a sex-determining gene have a vastly stronger influence on the balance of selection than more distant loci. As a result, changes in linkage, caused, for example, by an inversion that captures a sex-determination mutation and a gene under sex-antagonistic selection, can trigger transitions between XY and ZW systems. Sex-antagonistic alleles can become more strongly associated with pleiotropically dominant sex-determining factors, which may help to explain biases in the rates of transitions between male and female heterogamety. Deleterious recessive mutations completely linked to the ancestral Y chromosome can prevent invasion of a neo-W chromosome or result in a stable equilibrium at which XY and ZW systems segregate simultaneously at two linkage groups.