Ancient structure in Africa unlikely to explain Neanderthal and non-African genetic similarity

Neanderthals have been shown to share more genetic variants with present-day non-Africans than Africans. Recent admixture between Neanderthals and modern humans outside of Africa was proposed as the most parsimonious explanation for this observation. However, the hypothesis of ancient population structure within Africa could not be ruled out as an alternative explanation. We use simulations to test whether the site frequency spectrum, conditioned on a derived Neanderthal and an ancestral Yoruba (African) nucleotide (the doubly conditioned site frequency spectrum [dcfs]), can distinguish between models that assume recent admixture or ancient population structure. We compare the simulations to the dcfs calculated from data taken from populations of European, Chinese, and Japanese descent in the Complete Genomics Diversity Panel. Simulations under a variety of plausible demographic parameters were used to examine the shape of the dcfs for both models. The observed shape of the dcfs cannot be explained by any set of parameter values used in the simulations of the ancient structure model. The dcfs simulations for the recent admixture model provide a good fit to the observed dcfs for non-Africans, thereby supporting the hypothesis that recent admixture with Neanderthals accounts for the greater similarity of Neanderthals to non-Africans than Africans.