Epistasis can facilitate the evolution of reproductive isolation by peak shifts: a two-locus two-allele model

The influence of epistasis on the evolution of reproductive isolation by peak shifts is studied in a two-locus two-allele model of a quantitative genetic character under stabilizing selection. Epistasis is introduced by a simple multiplicative term in the function that maps gene effects onto genotypic values. In the model with only additive effects on the trait, the probability of a peak shift and the amount of reproductive isolation are always inversely related, i.e., the higher the peak shift rate, the lower the amount of reproductive isolation caused by the peak shift. With epistatic characters there is no consistent relationship between these two values. Interestingly, there are cases where both transition rates as well as the amount of reproductive isolation are increased relative to the additive model. This effect has two main causes: a shift in the location of the transition point, and the hybrids between the two alternative optimal genotypes have lower average fitness in the epistatic case. A review of the empirical literature shows that the fitness relations resulting in higher peak shift rates and more reproductive isolation are qualitatively the same as those observed for genes causing hybrid inferiority.