Group selection on the boundary of a stable population

A model of group selection is constructed for the case of differential extinction acting on small boundary populations of a large, fixed population. Consideration is restricted to extinction operators acting at or near to carrying capacity. Under the assumption that the extinction rate is large relative to individual genetic parameters affecting gene frequencies in boundary populations, we discuss the conditions under which differential extinction is most likely to produce a significant effect. In particular, a condition for bimodality in the distribution of gene frequencies in boundary populations (population polymorphism) is that there be some critical allele frequency at which the extinction rate jumps from high to low. An extinction operator linear in allele frequency produces no qualitative effect of this kind. In consequence, we are able to make precise rather limited circumstances under which group selection is likely to have a significant effect.