The evolution of hybrid incompatibilities along a phylogeny

The Dobzhansky-Muller model of speciation posits that defects in hybrids between species are the result of negative epistatic interactions between alleles that arose in independent genetic backgrounds. Tests of one important prediction from this model, that incompatibilities "snowball," have relied on comparisons of the number of incompatibilities between closely related pairs of species separated by different divergence times. How incompatibilities accumulate along phylogenies, however, remains poorly understood. We extend the Dobzhansky-Muller model to multispecies clades to describe the mathematical relationship between tree topology and the number of shared incompatibilities among related pairs of species. We use these results to develop a statistical test that distinguishes between the snowball and alternative incompatibility accumulation models, including nonepistatic and multilocus incompatibility models, in a phylogenetic context. We further demonstrate that patterns of incompatibility sharing across species pairs can be used to estimate the relative frequencies of different types of incompatibilities, including derived-derived versus derived-ancestral incompatibilities. Our results and statistical methods should motivate comparative genetic mapping of hybrid incompatibilities to evaluate competing models of speciation.