Competition between recombination and epistasis can cause a transition from allele to genotype selection

Biochemical and regulatory interactions central to biological networks are expected to cause extensive genetic interactions or epistasis affecting the heritability of complex traits and the distribution of genotypes in populations. However, the inference of epistasis from the observed phenotype-genotype correlation is impeded by statistical difficulties, while the theoretical understanding of the effects of epistasis remains limited, in turn limiting our ability to interpret data. Of particular interest is the biologically relevant situation of numerous interacting genetic loci with small individual contributions to fitness. Here, we present a computational model of selection dynamics involving many epistatic loci in a recombining population. We demonstrate that a large number of polymorphic interacting loci can, despite frequent recombination, exhibit cooperative behavior that locks alleles into favorable genotypes leading to a population consisting of a set of competing clones. When the recombination rate exceeds a certain critical value that depends on the strength of epistasis, this "genotype selection" regime disappears in an abrupt transition, giving way to "allele selection"--the regime where different loci are only weakly correlated as expected in sexually reproducing populations. We show that large populations attain highest fitness at a recombination rate just below critical. Clustering of interacting sets of genes on a chromosome leads to the emergence of an intermediate regime, where blocks of cooperating alleles lock into genetic modules. These haplotype blocks disappear in a second transition to pure allele selection. Our results demonstrate that the collective effect of many weak epistatic interactions can have dramatic effects on the population structure.