Epistasis in polygenic traits and the evolution of genetic architecture under stabilizing selection

We consider the effects of epistasis in a polygenic trait in the balance of mutation and stabilizing selection. The main issues are the genetic variation maintained in equilibrium and the evolution of the mutational effect distribution. The model assumes symmetric mutation and a continuum of alleles at all loci. Epistasis is modeled proportional to pairwise products of the single-locus effects. A general analytical formalism is developed. Assuming linkage equilibrium, we derive results for the equilibrium mutation load and the genetic and mutational variance in the house of cards and the Gaussian approximation. The additive genetic variation maintained in mutation-selection balance is reduced by any pattern of the epistatic interactions. The mutational variance, in contrast, is often increased. Large differences in mutational effects among loci emerge, and a negative correlation among (standard mean) locus mutation effects and mutation rates is predicted. Contrary to the common view since Waddington, we find that stabilizing selection in general does not lead to canalization of the trait. We propose that canalization as a target of selection instead occurs at the genic level. Here, primarily genes with a high mutation rate are buffered, often at the cost of decanalization of other genes. An intuitive interpretation of this view is given in the discussion.