How population subsampling to concentrate selection effects can help to find epistatic interactions of transmission...

Gene-gene incompatibilities-caused by deleterious allele combinations at different loci - can impair gametophyte or zygote viability, reducing fertility or overall viability in hybrids or hybrid-derived populations. These incompatibilities can be detected through transmission ratio distortion (TRD), which reflects deviations from Mendelian inheritance due to selection against incompatible allele combinations. However, identifying epistatic TRD locus pairs remains challenging, as standard multiple-testing approaches are underpowered. To address this limitation, a stepwise workflow is introduced that achieves higher sensitivity than standard approaches by using targeted subsampling of the population. This strategy effectively concentrates TRD signals shared between two interacting loci into a single locus, thereby strengthening the signal, simplifying the analysis, and eliminating the need for multiple-testing correction. In addition, incompatibility models were developed to predict TRD patterns under various selection mechanisms (gametophytic or zygotic) and inheritance modes (symmetric or asymmetric) across common mapping populations (F2, BC1, RIL). Applied to 17 F2 and 2 RIL intraspecific Arabidopsis thaliana populations, the stepwise workflow identified seven novel and two known TRD locus pairs. In most cases, the observed TRD patterns offered insights into potential selection mechanisms. The demonstrated sensitivity and robustness of the proposed method highlight its potential to uncover numerous additional incompatibility locus pairs across diverse species. Systematic application of the method could elevate the discovery of hybrid incompatibilities beyond current incidental discoveries and open new avenues for understanding the genetic architecture of speciation.