The effect of neutral nonadditive gene action on the quantitative index of population divergence

For neutral additive genes, the quantitative index of population divergence (Q(ST)) is equivalent to Wright's fixation index (F(ST)). Thus, divergent or convergent selection is usually invoked, respectively, as a cause of the observed increase (Q(ST) > F(ST)) or decrease (Q(ST) < F(ST)) of Q(ST) from its neutral expectation (Q(ST) = F(ST)). However, neutral nonadditive gene action can mimic the additive expectations under selection. We have studied theoretically the effect of consecutive population bottlenecks on the difference F(ST) - Q(ST) for two neutral biallelic epistatic loci, covering all types of marginal gene action. With simple dominance, Q(ST) < F(ST) for only low to moderate frequencies of the recessive alleles; otherwise, Q(ST) > F(ST). Additional epistasis extends the condition Q(ST) < F(ST) to a broader range of frequencies. Irrespective of the type of nonadditive action, Q(ST) < F(ST) generally implies an increase of both the within-line additive variance after bottlenecks over its ancestral value (V(A)) and the between-line variance over its additive expectation (2F(ST)V(A)). Thus, both the redistribution of the genetic variance after bottlenecks and the F(ST) - Q(ST) value are governed largely by the marginal properties of single loci. The results indicate that the use of the F(ST) - Q(ST) criterion to investigate the relative importance of drift and selection in population differentiation should be restricted to pure additive traits.