Species coexist more easily if reinforcement is based on habitat preferences than on species recognition

Maladaptive hybridization selects for prezygotic isolation, a process known as reinforcement. Reinforcement reduces gene flow and contributes to the final stage of speciation. Ecologically, however, coexistence of the incipient species is difficult if they initially use identical resources. Habitat segregation offers an alternative to species discrimination as a way to reduce gene flow: production of unfit hybrids is reduced if mate encounters become rare due to differing habitat choice. Using a modelling approach, we show that hybridization avoidance alone can select for habitat specialization, even if neither of the species is intrinsically better at using a specific niche. While habitat segregation and species discrimination both reduce the risk of producing unfit hybrids, these two isolation mechanisms differ from each other with respect to their effects on resource competition. Our model shows that, as a consequence of such differences, reinforcement evolves much more easily if hybridization is avoided based on habitat segregation than if the mechanism involves species recognition (mate choice traits). We also examine the outcomes when both isolation mechanisms evolve jointly. The establishment of one isolation mechanism a priori weakens selection for the other. However, an asymmetry persists here too. The net effect of habitat segregation on species discrimination was typically facilitative, but not vice versa. This asymmetry arises because habitat segregation, by enhancing coexistence, secures time for the subsequent evolution of species discrimination in a mate choice context (still relevant if habitat use is not perfectly segregated). Species discrimination does not have such a stabilizing effect on coexistence. Our results emphasize the importance of habitat segregation in reinforcement and offer a way to interpret findings where closely related taxa show similar performance on different resources or in different habitats. Studies of ecological generalization and specialization should therefore take into account that niche differences can be initiated and/or maintained by hybridization avoidance.