Inconsistency of Triplet-Based and Quartet-Based Species Tree Estimation under Intralocus Recombination

We consider species tree estimation from multiple loci subject to intralocus recombination. We focus on $$, a summary coalescent-based method using rooted triplets, as well as a related quartet-based inference method. We demonstrate analytically that in both cases, intralocus recombination gives rise to an inconsistency zone, in which correct inference is not assured even in the limit of infinite amount of data. In addition, we validate and characterize this inconsistency zone through a simulation study, which suggests that differential rates of recombination between closely related taxa can amplify the effect of incomplete lineage sorting and contribute to inconsistency.