Coevolution of self-fertilization and inbreeding depression. I. Mutation-selection balance at one and two loci

Simple theories for the evolution of breeding systems suggest that the fate of an allele that modifies the rate of self-fertilization hinges only on the degree to which selfing reduces opportunities for outcrossing ("pollen discounting") and the extent of inbreeding depression. These theories predict that outcrossing evolves whenever deleterious mutations have a more severe effect in combination than expected from their individual effects. We study the evolutionary dynamics of a modifier of the rate of self-fertilization in populations subject to complete pollen discounting and recurrent mutations which impair viability at a single locus in diploids and at two loci in haploids. Our analysis indicates that genetic associations arising immediately upon the introduction of a rare modifier allele generate substantial quantitative and qualitative departures from expectation. Higher rates of segregation under selfing in our one-locus diploid model generate positive associations between enhancers of selfing and wild-type viability alleles, which in turn favor the evolution of selfing under a wider range of conditions than expected. Greater opportunities for recombination under outcrossing in our two-locus haploid model generate positive associations between enhancers of outcrossing and wild-type viability alleles. These associations favor the evolution of outcrossing under a wider range of conditions, and introduce the possibility of stable mixed mating systems involving both selfing and outcrossing. Our explicit analysis of genetic associations between loci affecting viability and the rate of self-fertilization indicates that modifiers that enhance the production of offspring with very high (and very low) viability by promoting segregation or recombination develop positive associations with high viability. This advantage of producing extremes can compensate for an initial disadvantage in offspring number.