Error rates in QST-FST comparisons depend on genetic architecture and estimation procedures

Genetic and phenotypic variation among populations is one of the fundamental subjects of evolutionary genetics. One question that arises often in data on natural populations is whether differentiation among populations on a particular trait might be caused in part by natural selection. For the past several decades, researchers have used QST-FST approaches to compare the amount of trait differentiation among populations on one or more traits (measured by the statistic QST) with differentiation on genome-wide genetic variants (measured by FST). Theory says that under neutrality, FST and QST should be approximately equal in expectation, so QST values much larger than FST are consistent with local adaptation driving subpopulations' trait values apart, and QST values much smaller than FST are consistent with stabilizing selection on similar optima. At the same time, investigators have differed in their definitions of genome-wide FST (such as "ratio of averages" vs. "average of ratios" versions of FST) and in their definitions of the variance components in QST. Here, we show that these details matter. Different versions of FST and QST have different interpretations in terms of coalescence time, and comparing incompatible statistics can lead to elevated type I error rates, with some choices leading to type I error rates near one when the nominal rate is 5%. We conduct simulations under varying genetic architectures and forms of population structure and show how they affect the distribution of QST. When many loci influence the trait, our simulations support procedures grounded in a coalescent-based framework for neutral phenotypic differentiation.