Modifiers of mutation rate: a general reduction principle

A deterministic two-locus population genetic model with random mating is studied. The first locus, with two alleles, is subject to mutation and arbitrary viability selection. The second locus, with an arbitrary number of alleles, controls the mutation at the first locus. A class of viability-analogous Hardy-Weinberg equilibria is analyzed in which the selected gene and the modifier locus are in linkage equilibrium. It is shown that at these equilibria a reduction principle for the success of new mutation-modifying alleles is valid. A new allele at the modifier locus succeeds if its marginal average mutation rate is less than the mean mutation rate of the resident modifier allele evaluated at the equilibrium. Internal stability properties of these equilibria are also described.