Transgressive segregation in mating traits drives hybrid speciation

Hybridization can rapidly generate novel genetic variation, which can promote ecological speciation by creating novel adaptive phenotypes. However, it remains unclear how hybridization, creating novel mating phenotypes (e.g., mating season, genitalia shapes, sexual displays, mate preferences), affects speciation especially when the phenotypes do not confer adaptive advantages. Here, based on individual-based evolutionary simulations, we propose that transgressive segregation of mating traits can drive incipient hybrid speciation. Simulations demonstrated that incipient hybrid speciation occurred most frequently when the hybrid population received moderate continued immigration from parental lineages causing recurrent episodes of hybridization. Recurrent hybridization constantly generated genetic variation, which promoted the rapid stochastic evolution of mating phenotypes in a hybrid population. The stochastic evolution continued until a novel mating phenotype came to dominate the hybrid population, which reproductively isolates the hybrid population from parental lineages. However, too frequent hybridization rather hindered the evolution of reproductive isolation by inflating the variation of mating phenotypes to produce phenotypes allowing mating with parental lineages. Simulations also revealed conditions for the long-term persistence of hybrid species after their incipient emergence. Our results suggest that recurrent transgressive segregation of mating phenotypes can offer a plausible explanation for hybrid speciation and radiations that involved little adaptive ecological divergence.