Population Size Differences Can Lead to Biases in Phylogenetic Inference and Introgression Detection in the Presence...

Assumptions about the probability distribution of gene tree topologies provide a basis for phylogenetic reconstruction and introgression detection. Initial evidence has suggested that in the presence of purifying selection, population size differences can affect the probability distribution of gene tree topologies. Nevertheless, the impact of this phenomenon on phylogenetic reconstruction and introgression detection remains to be explored. Additionally, a theoretical understanding of this phenomenon remains absent. Here, using the population genetic simulation software SLiM, we provide evidence that in the presence of purifying selection, population size differences can cause biases in phylogenetic inference. We also provide evidence that in the presence of purifying selection, population size differences can cause statistics used for introgression detection to exhibit patterns resembling those caused by introgression. Additionally, we present a theoretical analysis showing that using a random gene tree labeling process to model the gene tree formation process in the occurrence of deep coalescence lacks a biological basis under purifying selection, and gene tree distributions under purifying selection are expected to exhibit population size dependence. This work underscores the importance of considering the potential confounding impact of purifying selection on phylogenetic inference and introgression detection.