Predicting response to selection on a quantitative trait: a comparison between models for mixed-mating populations

Two different theoretical frameworks have been developed to predict response to selection in a mixed mating population (in which reproduction occurs by a mixture of outcrossing and self-fertilization). The genotypic covariance model (GCM) and the structured linear model (SLM) rely on the same assumptions regarding quantitative trait inheritance, but use different genetic summary statistics. Here, we demonstrate the algebraic relationships between the various genetic metrics used in each theory. This is accomplished by reformulating the GCM in terms of the Wright-Kempthorne equation. We use stochastic simulations to investigate the relative accuracy of each theory for a range of selfing rates. The SLM is generally more accurate than the GCM, the most pronounced differences emerging in simulations with inbreeding depression for fitness. In fact, with strong inbreeding depression and high selfing rates, evolution can occur opposite the direction predicted by the GCM. The simulations also indicate that direct application of random mating models to partially selfing populations can produce very inaccurate predictions if quantitative trait loci exhibit dominance.