Evolution under Stochastic Transmission: Mutation-rate Modifiers

Evolutionary analyses of large populations commonly incorporate stochasticity through temporal variation in selection while treating genetic transmission parameters as fixed. Much less attention has been given to stochasticity in transmission itself. We study a selected locus with alleles A and a under constant selection, linked to a neutral modifier locus whose alleles M₁ and M₂ control the mutation rate from A to a. Under constant transmission, the Reduction Principle applies: near a mutation-selection balance where M₁ is fixed with mutation rate u₁, a rare allele M₂ invades if its associated rate u₂1 and fails to invade if u₂>u₁. This result holds for both haploid and diploid populations and is independent of other parameters such as recombination, which affects only the rate, not the direction, of change. We extend this framework by allowing the mutation rate associated with the invading modifier allele M₂ to fluctuate randomly across generations. In this stochastic setting, invasion is no longer determined by mean mutation rates alone. Instead, it depends on the temporal distribution of mutation rates, the strength of selection at the focal locus, and the recombination rate between modifier and target loci. Stochastic transmission therefore does not merely rescale deterministic predictions based on the Reduction Principle; it can alter the direction of selection acting on modifier alleles.